Table of Contents
IronForgeCyber
Courses
Resources
Blog
Log In


IronForgeCyber

AboutPrivacyTerms

© 2026 IronForgeCyber. All rights reserved.

  1. courses
  2. re-static-analysis
  3. obfuscation
11 min read

Obfuscation

Every check so far has compared our input with something we could read: a number in an instruction, a PIN in .data, a string in .rodata. Authors who do not want their check read that easily take one more step. They scramble the input first and compare the scrambled result against scrambled data. The answer is no longer in the file. What is in the file is the answer after a transformation, and the code that performs it.

That turns out to be enough. If we can read the transformation, we can run it backwards. This lesson reverses one such check from start to finish and ends with a script that prints the password.

The Program

chall
$ echo hunter2 | ./challPassword: Wrong length.

It tells us the length is wrong, which is already more than it should. The download holds chall and its source. As before, we work from the binary and keep the source for checking afterwards.

Spotting an Encoded Check

strings finds the messages and nothing that looks like a password, so we go to main in Ghidra:

Ghidra
1undefined8 main(void)
2{
3  int iVar1;
4  char

Most of this is familiar. The input is read and trimmed, and its length must be 0x13, which is 19. Then come the two lines that matter:

Ghidra
1      scramble(local_48,0x13);
2      iVar1 = memcmp(local_48,target,0x13);

Our input is compared with 19 bytes at target, but only after a function has been run over it. That ordering is the signature of an encoded check: the input is modified, and then compared. Whatever target holds, it is not the password. It is what the password turns into.

You will not always get a function named scramble. The thing to look for is a loop that rewrites the input buffer, sitting between the read and the compare.

Reading the Transformation

Ghidra
1void scramble(long param_1,ulong param_2)
2{
3  char cVar1;
4  ulong local_10;

It is dense, so take it apart. param_1 is the buffer and param_2 its length, and local_10 counts from 0 up to the length, so call it i. Inside the loop there are three steps, and it helps to write each as its own line:

  1. *(byte *)(local_10 + param_1) is buf[i], and *(byte *)(&key + (i & 3)) is key[i & 3]. The two are combined with ^, which is XOR. i & 3 keeps the lowest two bits of i, which is the same as i % 4: it goes 0, 1, 2, 3, 0, 1, 2, 3. So each byte is XORed with one of four key bytes, in rotation.
  2. The result goes through rotate_left with an argument of 3.
  3. i is added, and the sum is stored back into buf[i].

In short: buf[i] = rotate_left(buf[i] ^ key[i % 4], 3) + i.

The helper is one line:

Ghidra
1uint rotate_left(byte param_1,byte param_2)
2{
3  return (int)(uint)param_1 >> (8 

Two shifts joined with OR, where one shifts left by n and the other shifts right by 8 - n. This pattern is worth memorizing, because C has no rotate operator and this is how everyone writes one. A plain left shift pushes the top bits off the end and loses them. A rotation brings them back in at the bottom, so nothing is lost. Rotating 10110001 left by 3 gives 10001101. The name in this binary gave it away, but in a stripped binary the two-shifts-and-an-OR shape is how you would recognize it.

Common Transformations

Before undoing this one, it is worth knowing the usual building blocks, because obfuscated checks are nearly always a few of these stacked up:

TransformationLooks likeUndone by
XOR with a keyx ^ kXOR with the same key
Addition or subtractionx + k, x - kThe opposite operation
RotationTwo shifts and an ORRotating the other way by the same amount
Substitutiontable[x], with a 256-byte tableA table that maps each value back
Reorderingout[i] = in[perm[i]]Putting each byte back where it came from
Multiplication by an odd numberx * kMultiplying by the modular inverse

Every row can be reversed exactly, because no information is lost: each possible input gives a different output. That is the property to check for. A plain shift, a division, or an AND with a mask throws bits away and cannot be undone, and neither can a hash like the CRC-32 in Strings & Constants. When a check uses one of those, you need a different approach, such as trying candidates until one matches.

XOR deserves a special mention, because it is the most common of all. XOR with a value is its own inverse: (x ^ k) ^ k is x again. If a program XORs data with a key, running the same XOR over the result gives the original back.

Inverting It

To reverse a sequence of steps, undo them in the opposite order, last step first. Getting dressed, you put on socks and then shoes; getting undressed, the shoes come off first.

The forward direction, for the byte at position i:

  1. XOR with key[i % 4]
  2. Rotate left by 3
  3. Add i

So the reverse is:

  1. Subtract i
  2. Rotate right by 3
  3. XOR with key[i % 4]

One detail matters throughout. These are single bytes, so every result wraps around at 256. Adding i to 250 does not give 260. In C that happens automatically because the buffer holds char values. In a script we have to do it ourselves, by applying & 0xff after each step.

We need the two pieces of data, key and target. Both are constants, so they are in .rodata:

objdump
$ objdump -s -j .rodata chall | head -n 8Contents of section .rodata: 4012d0 01000200 00000000 00000000 00000000  ................ 4012e0 1337c0de 00000000 00000000 00000000  .7.............. 4012f0 d28c1e30 37cf2b7e 6a2917f8 e750ab64  ...07.+~j)...P.d 401300 114bff50 61737377 6f72643a 20000a00  .K.Password: ...

Ghidra shows key at 0x4012e0 and target at 0x4012f0. So the key is the four bytes 13 37 c0 de, and the target is the 19 bytes that start d2 8c 1e 30. In Ghidra you can also select the bytes in the Listing, right-click, and use Copy Special to copy them in a format ready to paste into a script.

Try the first byte by hand before writing any code. target[0] is 0xd2, and i is 0:

  1. Subtract 0: still 0xd2, which is 11010010 in binary.
  2. Rotate right by 3: the lowest three bits, 010, move to the top, giving 01011010, which is 0x5a.
  3. XOR with key[0], 0x13: 0x5a ^ 0x13 is 0x49.

0x49 is the letter I. That is a plausible first letter for a flag, which is good evidence that we read the function correctly. Checking one value by hand like this catches most mistakes before they cost time.

A Solve Script

Now the same thing for all 19 bytes:

solve.py
1key = bytes.fromhex("1337c0de")
2target = bytes.fromhex("d28c1e3037cf2b7e6a2917f8e750ab64114bff")
3
4

The loop body is the three reverse steps, in order, and rotate_right is the same two-shifts-and-an-OR idiom pointing the other way.

$ python3 solve.pyIFC{und0_3ach_st3p}$ python3 solve.py | ./challPassword: Correct.

We never guessed a password and never ran the program with one until we had the answer. All the information came from reading.

When You Would Rather Not Invert

There is a shortcut that works on many checks like this one. Look at what each output byte depends on: only the input byte at the same position, and the position itself. No byte affects any other. That means each position can be solved alone, and a single byte has only 256 possible values.

So you can skip the inversion: copy the forward transformation into a script exactly as the decompiler shows it, and for each position try all 256 values until the output matches target[i]. That is at most 19 times 256 attempts, which takes no time at all, and it needs no understanding of how to reverse a rotation.

This works whenever bytes are transformed independently, and it is often the fastest route in a timed competition. It stops working when the transformation mixes positions together, for example when each byte is combined with the previous output. Then the reverse-order method above is the one to use.

Summary

  • An encoded check modifies the input and then compares it. The stored data is the transformed answer, not the answer.
  • Read the transformation one step at a time and write it as a formula. Recognize the idioms: i & 3 is i % 4, and two shifts joined by an OR are a rotation.
  • XOR, addition, rotation, substitution and reordering can all be reversed exactly. Shifts, masks and hashes cannot.
  • Undo the steps in the opposite order, keep every value within a byte, and check one value by hand before scripting the rest.
  • If each byte is transformed independently, trying all 256 values per position avoids inverting anything.

Next is Other Languages: what changes when the program was not written in C.

*
pcVar2
;
5 undefined8 uVar3;
6 size_t sVar4;
7 char local_48 [64];
8
9 printf("Password: ");
10 pcVar2 = fgets(local_48,0x40,stdin);
11 if (pcVar2 == (char *)0x0) {
12 uVar3 = 1;
13 }
14 else {
15 sVar4 = strcspn(local_48,"\n");
16 local_48[sVar4] = '\0';
17 sVar4 = strlen(local_48);
18 if (sVar4 == 0x13) {
19 scramble(local_48,0x13);
20 iVar1 = memcmp(local_48,target,0x13);
21 if (iVar1 == 0) {
22 puts("Correct.");
23 }
24 else {
25 puts("Wrong.");
26 }
27 uVar3 = 0;
28 }
29 else {
30 puts("Wrong length.");
31 uVar3 = 1;
32 }
33 }
34 return uVar3;
35}
5
6 for (local_10 = 0; local_10 < param_2; local_10 = local_10 + 1) {
7 cVar1 = rotate_left(*(byte *)((long)&key + (ulong)((uint)local_10 & 3)) ^
8 *(byte *)(local_10 + param_1),3);
9 *(char *)(local_10 + param_1) = cVar1 + (char)local_10;
10 }
11 return;
12}
-
param_2
&
0x1f
)
|
(
uint
)
param_1
<<
(
param_2
&
0x1f
)
;
4}
5def rotate_right(value, bits):
6 return ((value >> bits) | (value << (8 - bits))) & 0xFF
7
8
9password = ""
10for i, byte in enumerate(target):
11 byte = (byte - i) & 0xFF
12 byte = rotate_right(byte, 3)
13 byte ^= key[i % 4]
14 password += chr(byte)
15
16print(password)