The harder targets so far were hard by accident. Nobody strips a binary or links it statically to annoy reverse engineers. Those are ordinary build choices that happen to remove information. This lesson is about binaries made difficult on purpose, with tools and tricks whose only job is to defeat the kind of reading this course teaches.
It is a short tour of three of them: packing, control-flow flattening, and tricks aimed at disassemblers. For each, the goal is to recognize it quickly, because part of the answer in every case is knowing when reading stops being the right approach.
echo 1 | ./challCode: Locked.It asks for a code and has two messages, like half the programs in this course. This one will not show them to us.
The usual first steps go wrong immediately:
strings chall | grep -c Unlocked0The program can print Unlocked.. We wrote it. Yet the string is not in the file. Ghidra does no better. After analysis the Symbol Tree lists three functions in total, and the only one with any size is a dense loop of shifts and byte copies that has nothing to do with checking a code.
The file has been packed. A packer takes a finished program, compresses or encrypts it, and wraps it in a small piece of code called a stub. When the packed file runs, the stub unpacks the original program into memory and jumps to it. On disk there is only the stub and a block of compressed data. The real code and strings exist only while the program is running, which puts them out of reach of static analysis.
Packers have honest uses, mainly making files smaller, but they are a favorite of malware authors for exactly this reason.
Several things give a packed file away, and you have seen most of them already:
file challchall: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), BuildID[sha1]=7fa1793526889e1aa4d1754fea6ac5683f787377, for GNU/Linux 3.2.0, statically linked, no section headerreadelf -S challThere are no sections in this file.file says no section header, and readelf confirms there are no sections at all. A normal binary has .text, .data, .rodata and the rest. The stub does not need them, so the packer throws them away.strings does print is mostly short runs of noise. Compressed data looks random, and random bytes rarely form readable text.A large file with very little recognizable code, few sensible strings and no imports is packed until proven otherwise. Tools can measure the randomness directly as entropy. Ordinary code and data score in the middle of the scale, and compressed or encrypted data scores near the top.
Often the packer announces itself:
strings chall | grep UPXYUPX!\$Info: This file is packed with the UPX executable packer http://upx.sf.net $$Id: UPX 5.21 Copyright (C) 1996-2026 the UPX Team. All Rights Reserved. $UPX!UPX!UPX is the most common packer there is, free and open source, and it leaves its name in the file. It also unpacks its own work:
upx -d -o unpacked chall Ultimate Packer for eXecutables Copyright (C) 1996 - 2026UPX 5.2.1 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 27th 2026 File size Ratio Format Name -------------------- ------ ----------- ----------- 726848 <- 311660 42.88% linux/amd64 unpackedUnpacked 1 file.strings unpacked | grep -c Unlocked1unpacked is the original program, byte for byte, and everything in this course works on it again. When you identify a known packer, look for its unpacker before doing anything else.
UPX is the easy case. Malware often uses a modified UPX with the markers removed so that upx -d refuses the file, or a custom packer with no public unpacker. Then the dependable method is to let the program unpack itself: run it under a debugger in a safe environment, stop once the stub has finished, and save the unpacked program out of memory. That is dynamic analysis, and it is the first of several places in this lesson where static analysis hands over.
With unpacked loaded in Ghidra, we can find main from the entry point as in Statically-Linked Binaries, and from main the function that checks the code:
1undefined8 FUN_00401c05(uint param_1)
2{
3 uint local_10;
4 uint local_c;
Thirty lines, a loop, and comparisons against numbers that mean nothing. The logic they hide is two lines long:
1unsigned int mixed = (code ^ 0x5555) + 0x1234;
2return mixed == 0x7777;This is control-flow flattening. The original function was a straight line: do this, then this, then test. The flattened version cuts it into pieces, puts every piece inside one loop, and adds a variable that says which piece runs next. Each piece does its small job and then sets that variable to the number of the next piece. The order of execution, which used to be visible in the layout of the code, is now hidden in data.
We flattened this function by hand so that it would be small enough to show. Obfuscating compilers, of which OLLVM is the best known, do it automatically to every function in a program, and the result in a real target is a loop around a switch with hundreds of cases.
To recognize it, look for these together:
local_c. This is the state variable.To read it, follow the state variable by hand. Start from its first value and write down what each state does and where it goes next:
| State | Does | Next state |
|---|---|---|
0x3a | local_10 = param_1 ^ 0x5555 | 0x91 |
0x91 | local_10 = local_10 + 0x1234 | 0x17 |
0x17 | Tests local_10 == 0x7777 | 0xe4 if equal, else 0x60 |
0xe4 | Returns 1 | |
0x60 | Returns 0 |
Read down the middle column and the original function is back: XOR with 0x5555, add 0x1234, compare with 0x7777. Undo it with the method from Obfuscation, last step first:
echo $(( (0x7777 - 0x1234) ^ 0x5555 ))12310echo 12310 | ./challCode: Unlocked.A table works for five states. For five hundred it does not, and people use scripts and dedicated deobfuscation tools to rebuild the original flow. It is also a case where watching the program run, and recording which blocks execute in what order, can give the answer faster than reading.
The third kind of defense targets the tools themselves. A disassembler has to decide which bytes are instructions, and there are two ways to do that.
objdump uses the simple one, called linear sweep: start at the top of the code, decode an instruction, move to the byte after it, decode the next, and so on to the end. It assumes everything in the code section is an instruction and that they sit end to end.
That assumption can be broken with one byte. Here is a piece of main in the unpacked binary:
objdump -d -M intel --start-address=0x401cd6 --stop-address=0x401ce2 unpacked0000000000401cd6 <.text+0x1956>: 401cd6: eb 01 jmp 0x401cd9 401cd8: e8 48 8d 45 e0 call 0xffffffffe085aa25 401cdd: ba 0a 00 00 00 mov edx,0xaThe first instruction, eb 01, is a jump forward by one byte. It lands on 0x401cd9. The disassembler does not follow jumps. It carries on to the next byte, 0x401cd8, finds e8, which is the first byte of a call instruction, and decodes it along with the four bytes that follow as a call to an absurd address.
That e8 byte is junk. It is never executed, because the jump goes over it. The four bytes after it, 48 8d 45 e0, are a real instruction, a lea that loads the address of the input buffer, and objdump has swallowed it into the fake call. The listing shows a call that does not exist and hides an instruction that does.
Ghidra is not fooled by this one. It uses the other method, recursive descent: start at a known entry point and follow the program's own control flow, decoding only the bytes that execution can reach. It follows the jump, skips the junk byte and decodes the lea correctly. That is one reason to prefer it over objdump for anything not written by a cooperative author.
Recursive descent can be fooled as well, with more effort. A common method is a conditional jump whose condition is always true, built so that the tool cannot tell. The tool dutifully decodes the path that is never taken, which is junk, and the junk overlaps and corrupts the real instructions after it. Signs of this in Ghidra are red bad-instruction markers in the Listing, decompiler warnings about overlapping instructions, and functions that end abruptly in nonsense. When you see them, the fix is manual: clear the bad bytes with C, mark the junk as data, and disassemble again from the right address with D.
There is a whole family of tricks beyond these, including code that rewrites itself as it runs and checks that detect a debugger and change behavior. What they share is that they attack the gap between what the file contains and what the processor will do with it.
Static analysis reads the file. Each technique in this lesson works by making the file an unreliable description of the program. A packer hides the code until run time. Flattening hides the order of execution in a variable. Junk bytes make the disassembly lie.
The processor is never fooled. Whatever the file looks like, when the program runs, the real instructions execute in the real order on the real data. That is the case for dynamic analysis: running the program under observation, in an environment where it can do no harm.
Signs that it is time to switch:
Switching is not giving up on static analysis. The two alternate. Static analysis found the stub and told us the file was packed. It told us which function was the check and what to break on. After a debugger has produced an unpacked program or a decrypted string, static analysis takes over again on better material. Good reverse engineers move between them constantly, using each to answer the questions the other makes expensive.
This course has stayed on the static side so that the habit of reading comes first. The Dynamic Analysis course picks up from here, and the GDB Guide is the place to start if you want to try it now.
objdump. Ghidra follows control flow and resists the simple versions, and its bad-instruction markers flag the cleverer ones.That is the end of the lessons. What remains is practice: the Challenges are four crackmes that use what the course has covered, starting gently.