Table of Contents
IronForgeCyber
Courses
Resources
Blog
Log In


IronForgeCyber

AboutPrivacyTerms

© 2026 IronForgeCyber. All rights reserved.

  1. courses
  2. re-static-analysis
  3. anti-analysis
13 min read

Anti-Analysis

The harder targets so far were hard by accident. Nobody strips a binary or links it statically to annoy reverse engineers. Those are ordinary build choices that happen to remove information. This lesson is about binaries made difficult on purpose, with tools and tricks whose only job is to defeat the kind of reading this course teaches.

It is a short tour of three of them: packing, control-flow flattening, and tricks aimed at disassemblers. For each, the goal is to recognize it quickly, because part of the answer in every case is knowing when reading stops being the right approach.

The Program

chall
$ echo 1 | ./challCode: Locked.

It asks for a code and has two messages, like half the programs in this course. This one will not show them to us.

Packing

The usual first steps go wrong immediately:

$ strings chall | grep -c Unlocked0

The program can print Unlocked.. We wrote it. Yet the string is not in the file. Ghidra does no better. After analysis the Symbol Tree lists three functions in total, and the only one with any size is a dense loop of shifts and byte copies that has nothing to do with checking a code.

The file has been packed. A packer takes a finished program, compresses or encrypts it, and wraps it in a small piece of code called a stub. When the packed file runs, the stub unpacks the original program into memory and jumps to it. On disk there is only the stub and a block of compressed data. The real code and strings exist only while the program is running, which puts them out of reach of static analysis.

Packers have honest uses, mainly making files smaller, but they are a favorite of malware authors for exactly this reason.

Recognizing a Packed File

Several things give a packed file away, and you have seen most of them already:

$ file challchall: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), BuildID[sha1]=7fa1793526889e1aa4d1754fea6ac5683f787377, for GNU/Linux 3.2.0, statically linked, no section header$ readelf -S challThere are no sections in this file.
  • file says no section header, and readelf confirms there are no sections at all. A normal binary has .text, .data, .rodata and the rest. The stub does not need them, so the packer throws them away.
  • There are almost no functions. Ghidra found three in a file of 300 KB.
  • The strings you expect are missing, and what strings does print is mostly short runs of noise. Compressed data looks random, and random bytes rarely form readable text.
  • There are no imports to speak of.

A large file with very little recognizable code, few sensible strings and no imports is packed until proven otherwise. Tools can measure the randomness directly as entropy. Ordinary code and data score in the middle of the scale, and compressed or encrypted data scores near the top.

UPX

Often the packer announces itself:

$ strings chall | grep UPXYUPX!\$Info: This file is packed with the UPX executable packer http://upx.sf.net $$Id: UPX 5.21 Copyright (C) 1996-2026 the UPX Team. All Rights Reserved. $UPX!UPX!

UPX is the most common packer there is, free and open source, and it leaves its name in the file. It also unpacks its own work:

upx
$ upx -d -o unpacked chall                       Ultimate Packer for eXecutables                          Copyright (C) 1996 - 2026UPX 5.2.1       Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 27th 2026        File size         Ratio      Format      Name   --------------------   ------   -----------   -----------    726848 <-    311660   42.88%   linux/amd64   unpackedUnpacked 1 file.$ strings unpacked | grep -c Unlocked1

unpacked is the original program, byte for byte, and everything in this course works on it again. When you identify a known packer, look for its unpacker before doing anything else.

UPX is the easy case. Malware often uses a modified UPX with the markers removed so that upx -d refuses the file, or a custom packer with no public unpacker. Then the dependable method is to let the program unpack itself: run it under a debugger in a safe environment, stop once the stub has finished, and save the unpacked program out of memory. That is dynamic analysis, and it is the first of several places in this lesson where static analysis hands over.

Control-Flow Flattening

With unpacked loaded in Ghidra, we can find main from the entry point as in Statically-Linked Binaries, and from main the function that checks the code:

Ghidra
1undefined8 FUN_00401c05(uint param_1)
2{
3  uint local_10;
4  uint local_c;

Thirty lines, a loop, and comparisons against numbers that mean nothing. The logic they hide is two lines long:

source.c
1unsigned int mixed = (code ^ 0x5555) + 0x1234;
2return mixed == 0x7777;

This is control-flow flattening. The original function was a straight line: do this, then this, then test. The flattened version cuts it into pieces, puts every piece inside one loop, and adds a variable that says which piece runs next. Each piece does its small job and then sets that variable to the number of the next piece. The order of execution, which used to be visible in the layout of the code, is now hidden in data.

We flattened this function by hand so that it would be small enough to show. Obfuscating compilers, of which OLLVM is the best known, do it automatically to every function in a program, and the result in a real target is a loop around a switch with hundreds of cases.

To recognize it, look for these together:

  • One loop that contains the whole function.
  • A variable that is compared against many unrelated constants and is never used in real arithmetic. Here it is local_c. This is the state variable.
  • Blocks that end by assigning a constant to that variable.
  • In the Function Graph, a wide, flat shape: one block at the top that every other block returns to, where a normal function would be tall and branching.

To read it, follow the state variable by hand. Start from its first value and write down what each state does and where it goes next:

StateDoesNext state
0x3alocal_10 = param_1 ^ 0x55550x91
0x91local_10 = local_10 + 0x12340x17
0x17Tests local_10 == 0x77770xe4 if equal, else 0x60
0xe4Returns 1
0x60Returns 0

Read down the middle column and the original function is back: XOR with 0x5555, add 0x1234, compare with 0x7777. Undo it with the method from Obfuscation, last step first:

$ echo $(( (0x7777 - 0x1234) ^ 0x5555 ))12310$ echo 12310 | ./challCode: Unlocked.

A table works for five states. For five hundred it does not, and people use scripts and dedicated deobfuscation tools to rebuild the original flow. It is also a case where watching the program run, and recording which blocks execute in what order, can give the answer faster than reading.

Tricks Against Disassemblers

The third kind of defense targets the tools themselves. A disassembler has to decide which bytes are instructions, and there are two ways to do that.

objdump uses the simple one, called linear sweep: start at the top of the code, decode an instruction, move to the byte after it, decode the next, and so on to the end. It assumes everything in the code section is an instruction and that they sit end to end.

That assumption can be broken with one byte. Here is a piece of main in the unpacked binary:

objdump
$ objdump -d -M intel --start-address=0x401cd6 --stop-address=0x401ce2 unpacked0000000000401cd6 <.text+0x1956>:  401cd6:       eb 01                   jmp    0x401cd9  401cd8:       e8 48 8d 45 e0          call   0xffffffffe085aa25  401cdd:       ba 0a 00 00 00          mov    edx,0xa

The first instruction, eb 01, is a jump forward by one byte. It lands on 0x401cd9. The disassembler does not follow jumps. It carries on to the next byte, 0x401cd8, finds e8, which is the first byte of a call instruction, and decodes it along with the four bytes that follow as a call to an absurd address.

That e8 byte is junk. It is never executed, because the jump goes over it. The four bytes after it, 48 8d 45 e0, are a real instruction, a lea that loads the address of the input buffer, and objdump has swallowed it into the fake call. The listing shows a call that does not exist and hides an instruction that does.

Ghidra is not fooled by this one. It uses the other method, recursive descent: start at a known entry point and follow the program's own control flow, decoding only the bytes that execution can reach. It follows the jump, skips the junk byte and decodes the lea correctly. That is one reason to prefer it over objdump for anything not written by a cooperative author.

Recursive descent can be fooled as well, with more effort. A common method is a conditional jump whose condition is always true, built so that the tool cannot tell. The tool dutifully decodes the path that is never taken, which is junk, and the junk overlaps and corrupts the real instructions after it. Signs of this in Ghidra are red bad-instruction markers in the Listing, decompiler warnings about overlapping instructions, and functions that end abruptly in nonsense. When you see them, the fix is manual: clear the bad bytes with C, mark the junk as data, and disassemble again from the right address with D.

There is a whole family of tricks beyond these, including code that rewrites itself as it runs and checks that detect a debugger and change behavior. What they share is that they attack the gap between what the file contains and what the processor will do with it.

When to Stop Reading

Static analysis reads the file. Each technique in this lesson works by making the file an unreliable description of the program. A packer hides the code until run time. Flattening hides the order of execution in a variable. Junk bytes make the disassembly lie.

The processor is never fooled. Whatever the file looks like, when the program runs, the real instructions execute in the real order on the real data. That is the case for dynamic analysis: running the program under observation, in an environment where it can do no harm.

Signs that it is time to switch:

  • The code you need is not in the file, because it is unpacked, decrypted or downloaded at run time.
  • A value you need is computed by something too tangled to follow, and a debugger could simply show it to you.
  • You have spent an hour on a function and cannot say what it does.
  • The disassembly contradicts itself.

Switching is not giving up on static analysis. The two alternate. Static analysis found the stub and told us the file was packed. It told us which function was the check and what to break on. After a debugger has produced an unpacked program or a decrypted string, static analysis takes over again on better material. Good reverse engineers move between them constantly, using each to answer the questions the other makes expensive.

This course has stayed on the static side so that the habit of reading comes first. The Dynamic Analysis course picks up from here, and the GDB Guide is the place to start if you want to try it now.

Summary

  • A packed file contains a stub and compressed data. It has few functions, no sections, no useful strings, and high entropy. Identify the packer and unpack it, with its own tool if one exists or by letting it run.
  • Control-flow flattening turns a function into one loop driven by a state variable. Recognize the shape, then trace the states in order to recover the logic.
  • Junk bytes desynchronize linear disassemblers like objdump. Ghidra follows control flow and resists the simple versions, and its bad-instruction markers flag the cleverer ones.
  • All three work by making the file misleading. When the file cannot be trusted, run the program and observe it.

That is the end of the lessons. What remains is practice: the Challenges are four crackmes that use what the course has covered, starting gently.

5
6 local_c = 0x3a;
7 local_10 = 0;
8 while( true ) {
9 if (local_c == 0xe4) {
10 return 1;
11 }
12 if (0xe4 < local_c) break;
13 if (local_c == 0x91) {
14 local_10 = local_10 + 0x1234;
15 local_c = 0x17;
16 }
17 else {
18 if (0x91 < local_c) {
19 return 0;
20 }
21 if (local_c == 0x17) {
22 if (local_10 == 0x7777) {
23 local_c = 0xe4;
24 }
25 else {
26 local_c = 0x60;
27 }
28 }
29 else {
30 if (local_c != 0x3a) {
31 return 0;
32 }
33 local_10 = param_1 ^ 0x5555;
34 local_c = 0x91;
35 }
36 }
37 }
38 return 0;
39}