Thursday, April 21, 2016

Using the DosBox Debugger

It's up to the individual if you want to use a debugger when reverse engineering a program. Some prefer a more cerebral challenge of only figuring out code execution using a decompiler tool, whereas others may find using a debugger useful for figuring out what values are passed to functions. I would recommend using a debugger particularly when reversing a game for the purpose of adding ScummVM support. When you start implementing code to implement game functionality, once you've got portions of the game disassembled, it can be immensely useful for tracking down bugs. Particularly if you initially write your code with names that closely match the names you give the methods in the disassembly.
For debugging purposes, if the game is a DOS game, the DosBox Debugger is the best tool I've found for executing and debugging DOS programs. The default distribution of DosBox doesn't have it enabled, but you can either compile DosBox with it enabled, or download a previously compiled executable. See the DosBox Debugger Thread for more information.
One of the biggest initial steps when using the DosBox debugger is matching addresses in executable at run-time with your disassembly in IDA. This can be done either from the debugger to IDA, or from IDA to the debugger: