If you’re an experienced developer, you may have encountered a strange phenomenon: a program can run perfectly fine under a debugger, yet still crash when executed standalone, i.e. outside a debugger session. The mere act of observing the program under a debugger yields to different results than running the program as-is.
Even very simple programs can be affected by this observer effect, which one would usually suspect in the field of physics (i.e. Heisenberg Uncertainty Principle) rather than that of our deterministic machines and computers!
In this post, I’ll illustrate the observer effect with a very simple assembly program, written both for FreeBSD/amd64 and FreeBSD/i386 platforms.
A decreasing counter
We start by writing a simple program in assembly that displays a decreasing counter. Let’s say, we want it to display the digits 5, 4, 3, 2, 1, 0, one per line, and then exit.
There are many ways to write such a program: the counter can be stored in a register, on the stack, or in some other memory location. Of course, register-based counters are the fastest, and stack-based counters are the second-easiest to set up because we already have stack memory at our disposal. However, to demonstrate the observer effect, we start by putting the counter in the
.data section of our program.
The FreeBSD/amd64 version
Without much ado, here’s the program for FreeBSD/amd64. We use macros for the syscalls, as we did in the previous episode.
We define the WRITE and EXIT syscalls as macros write and exit. The counter counter is stored in the
.data section, as well as an output buffer.
After initializing the counter to 6 at init_counter, we simply decrease it in loop_counter, until it reaches 0. Each time we decrease the counter, we call the assembly function write_counter to display it. Finally, we exit cleanly.
The write_counter function converts the counter to ASCII by adding 0x30 to it. Note that we assume that the counter is always less than 10, so it can be rendered in a single byte! We stuff the ASCII byte into the zeroth byte of the output buffer, and call write to display both the ASCII code, and the next byte of that buffer (which was already initialized to newline
\n). This way, we display one digit per line.
Assembling, linking and calling the program is trivial:
If you want, try running the same program under the debugger gdb(1). The output is the same.
The FreeBSD/i386 version
The FreeBSD/i386 version parallels the above program. We simply adapt to the different kernel call ABI and use 32-bit registers:
The program logic is exactly the same as above: the counter is located in the
.data, alongside the output buffer. Here too, assembling and linking, then running that program doesn’t yield any surprises:
No surprises, which is good.
A self-modifying decreasing counter
To trigger the dreaded observer effect, we modify both selfmod0_amd64.S and selfmod0_i386.S in such a way, that counter is in
.text instead of
.data. Note that buffer remains in
.data in this case.
The FreeBSD/amd64 version
The program is nearly identical to the previous one: only the location of counter has changed:
Let’s assemble and link it:
And now, prepare to witness the observer effect (drum roll)!
What do we see here? The program crashes when it executes standalone, but it doesn’t crash when executed under the debugger gdb(1)!
We can try to trace it with ktrace(1) and kdump(1):
There’s not much that can be seen here, except that the process got a SIGBUS signal very early on, that was handled with the SIG_DFL handler. A SIGBUS signal is usually a hint that the process tried to access a location in memory that was either not available or protected. In other words: the operating system kernel terminated our process, because we tried to access memory in a certain way that was forbidden or impossible.
The sneaky (and nasty) part here is that even if you ran the program under the debugger, there’s no way to hit the location that would cause the core dump! Remember: under the debugger, the program runs just fine!
The mere act of observing selfmod1_amd64.S under the debugger causes it to behave differently then when ran standalone. This is a typical instance of the observer effect in information theory.
The FreeBSD/i386 version
Let’s repeat the experiment with FreeBSD/i386:
The program is the same as before, and only counter migrated to the
.text section. Assembling and linking it as usual:
Will there be an observer effect here too?
Yes, there is (applause)! Tracing with ktrace(1) and kdump(1):
We see again that the operating system kernel terminated our process with a SIGBUS signal, i.e. we accessed memory in a wrong way.
What causes this Heisenbug?
You may now be asking yourself what caused this strange observer effect (a Heisenbug), right? In order not to spoil the fun, I’ll postpone the discussion of this aspect to a (probable) subsequent post.
Please stop reading now if you want to do some serious thinking on your own and if you love challenges!
Alright, you’re still totally at loss? Here are a few hints that may help setting you on the right track:
- The only difference between the working and non-working examples was that counter moved from
- Memory in
.datacan be modified freely, but memory in
.textis (supposed to be) read-only.
- Debuggers must have a way to insert opcodes in the code they debug, so that they can set breakpoints, or even single step through the code. They obviously can’t do that in a read-only section of memory!
- Operating systems like FreeBSD provide a system call that can modify the protection attributes of memory regions in a certain way. Debuggers like gdb(1) make extensive use of those system calls. Hint: ptrace(2).
- On FreeBSD, you may want to investigate the mprotect(2) C-Function (and associated system call), and use that to deprotect the
With all those hints, you should be able to extend selfmod1_amd64.S and selfmod1_i386.S in such a way, that they run both outside and inside a debugger session, while still keeping counter where it is, i.e. inside the
.text section. In the following post, I provide a solution to this little exercise.
When the mere act of observing a running program (e.g. in a debugger session) changes its behavior, we have an instance of the observer effect. While this effect can be annoying for developers seeking to find obscure bugs, it can be extremely useful for developers who would like to programatically detect if their programs are actually being reverse-engineered in a debugger session.
As we’ve seen in this post, the read-only property of the
.text section disappears when ran under a debugger. Programs that would like to detect debugger sessions (e.g. DRM, or tamper-proofing) can intentionally test for this property, and refuse to execute properly when
.text is writable.
Of course, this trick won’t prevent truly determined reverse-engineering efforts, and we’ve got to be thankful for that. But there are more advanced anti-tampering techniques: imagine a code that creates op-codes on the fly. With a judicious choice of interdependent op-codes and relative addressing, the modifications that a debugger inserts into that code could effectively break it, and you may have to run the program in a complete virtual machine environment in order to debug it. A real emulator in software like Bochs, while extremely slow, is preferable to hybrid emulators like VirtualBox for this kind of reverse-engineering.