Table of Contents

IronForgeCyber

Courses

Resources

Blog

Log In



IronForgeCyber

AboutPrivacyTerms

© 2026 IronForgeCyber. All rights reserved.

  1. courses
  2. reverse-engineering
  3. undo
12 min read

Undoing Binary Obfuscation

This binary encodes our input data and is compared against a buffer. We can use Ghidra to determine how the flag is encrypted and reverse the encryption to get the flag.

This will be a good introduction to reversing obfuscated functions in Ghidra.

Running checksec on the binary, we notice all security features are enabled:

$ checksec chall[*] './chall'    Arch:       i386-32-little    RELRO:      Partial RELRO    Stack:      Canary found    NX:         NX enabled    PIE:        PIE enabled    Stripped:   No

Static Analysis

Let's check the raw decompilation of main():

c
1undefined4 main(void)
2{
3  undefined4 uVar1;
4  

This code is a mess. There is a label in the middle of the function, a do-while loop, and many wonky variables. Decompiled output rarely matches the original source — the compiler reshapes code on the way down, as covered in reading optimized code. Let's discuss how we can clean this up:

  • First, we'll rename variables to something more meaningful. This immediately makes the code easier to read.
  • Second, we'll remove the label by moving the code after the label to the jump.
  • Third, we can remove the canary check. We know it's there, and it doesn't affect control flow.
  • We notice that param_1 is never used, so we'll remove it. This changes the signature to void main(void).
  • Once this is done, we'll remove variables we no longer need and variables that aren't used.

After these changes, we get the following code:

c
1int main(void)
2{
3  int iVar1;
4  int i;

This is a lot easier to read. We can simplify this further by switching the do-while loop for a while loop where the condition is checked.

c
1while (i <= 0x23) {
2    if (enckey[i] != buffer[i]) {
Warning
Make sure you understand how we made this conversion. This is a tough thing to analyze and change. Although it's not required to rearrange the loop, it makes the code easier to read.

This means that we're checking the first 0x28 = 40 bytes of enc_key against buffer. If they match, we call win(). If they don't match, we call puts("You lose!") and return.

We have a few things to check out:

  • What is enc_key?
  • What is buffer?
  • What does the encode() function do?

If we look at enc_key and buffer, we notice these variables are colored aqua. This means they are global variables. Local variables are colored yellow. Double-clicking on enc_key and buffer takes us to the location where they are defined.

This is the definition of enc_key:

c
1enckey                                          XREF[4]:     Entry Point(*),
2															  encode:00011233(R),
3															  encode

enckey holds an address, making it a pointer. We can click on DAT_00012008 to see what's there.

c
1 DAT_00012008                                    XREF[4]:     encode:00011240

We notice that enc_key is 36 bytes big and starts at 0x12008. On the left side, we see this data is initialized. The second column of data is the value at each byte. We can get Ghidra to copy this as a Python List using Right Click -> Copy Special -> Python List. With no extra effort, here is the value of buffer:

python
enckey = [ 0x31, 0xc0, 0x50, 0x68, 0x62, 0x6f, 0x6f, 0x74, 0x68, 

Now that we have our data, we can look at encode():

c
1undefined * encode(int param_1)
2{
3	undefined4 local_c;
4
5	for 

From this, we notice three things:

  • As it stands, encode() takes an int argument.
  • encode() must cast the input to an int, and then cast it again inside the function.
  • Ghidra doesn't know the return type.

From this, we can gather that Ghidra got the parameter type wrong. We can help Ghidra by changing the type to what we think it is. It appears that this function is casting to a char* and a byte*. A byte* is simply a signed char*. Let's change the type in Ghidra to char* and see what happens. At the same time, we know that enckey is a char*, so we'll change the return type too.

c
1char * encode(char *s)
2{
3	int i;
4	for (

This is a lot better. This function takes a string, performs byte-wise operations on each byte, and stores it in enc_key. It performs the following operation:

c
out = (in ^ 0x55) + 8

Both these operations are undo-able, meaning:

c
in = (out - 8) ^ 0x55

This is our key to solving this problem! If we take the output buffer, and perform this operation on each byte, we'll get the correct input.

Writing a Solve Script

First, we must write a decode() function that reverses the input. This will take in the list and return a list with the decoded bytes.

python
1def decode(in_list):
2    out_list = []
3    for i in in_list:
4        out_list.

We'll take our buffer and decode it:

python
1enckey = [ 0x31, 0xc0, 0x50, 0x68, 0x62, 0x6f, 0x6f, 0x74, 0x68

Now, we need to convert this list to a string. We can do this with the bytes() function:

python
payload = bytes(payload)

Finally, we can send this payload to the server:

python
1proc = process()
2proc.sendline(payload)
3proc.interactive()

This gives us the flag:

$ python3 asd.py[*] './chall'    Arch:       i386-32-little    RELRO:      Partial RELRO    Stack:      Canary found    NX:         NX enabled    PIE:        PIE enabled    Stripped:   No[+] Starting local process './chall': pid 267985[*] Switching to interactive modeWhat is the password?> IFC{PL4C3H0LD3R_FL4G_H3R3!}[*] Got EOF while reading in interactive$[*] Process './chall' stopped with exit code 0 (pid 267985)[*] Got EOF while sending in interactive
int
in_GS_OFFSET
;
5 int local_3c;
6 char local_38 [37] = { 0 }; // Ghidra does it the long way
7 int local_14;
8 undefined *local_10;
9
10 local_10 = &stack0x00000004;
11 local_14 = *(int *)(in_GS_OFFSET + 0x14);
12 setbuf(_stdin,(char *)0x0);
13 setbuf(_stdout,(char *)0x0);
14 setbuf(_stderr,(char *)0x0);
15 printf("What is the password?\n> ");
16 fgets(local_38,0x25,_stdin);
17 encode(local_38);
18 local_3c = 0;
19 do {
20 if (0x23 < local_3c) {
21 win();
22LAB_00011397:
23 uVar1 = 0;
24 if (local_14 != *(int *)(in_GS_OFFSET + 0x14)) {
25 uVar1 = __stack_chk_fail_local();
26 }
27 return uVar1;
28 }
29 if (enckey[local_3c] != local_38[local_3c]) {
30 puts("You lose!");
31 goto LAB_00011397;
32 }
33 local_3c = local_3c + 1;
34 } while( true );
35}
5 char buffer [37] = { 0 };
6 int local_14;
7 undefined *local_10;
8
9 setbuf(stdin, NULL);
10 setbuf(stdout, NULL);
11 setbuf(stderr, NULL);
12
13 printf("What is the password?\n> ");
14 fgets(buffer,0x25,_stdin);
15 encode(buffer);
16 i = 0;
17 do {
18 if (0x23 < i) {
19 win();
20 return 0;
21 }
22 if (enckey[i] != buffer[i]) {
23 puts("You lose!");
24 return 0;
25 }
26 i = i + 1;
27 } while( true );
28}
3
puts
(
"You lose!"
)
;
4 return 0;
5 }
6 i = i + 1;
7}
8win();
9return 0;
:
00011255
(
R
)
,
4 main:0001134d(R)
500014024 08 20 01 00 addr DAT_00012008 = 31h 1
(
*
)
,
2 encode:00011255(*),
3 main:00011358(*), 00014024(*)
4 00012008 31 ?? 31h 1
5 00012009 c0 ?? C0h
6 0001200a 50 ?? 50h P
7 0001200b 68 ?? 68h h
8 0001200c 62 ?? 62h b
9 0001200d 6f ?? 6Fh o
10 0001200e 6f ?? 6Fh o
11 0001200f 74 ?? 74h t
12 00012010 68 ?? 68h h
13 00012011 6e ?? 6Eh n
14 00012012 2f ?? 2Fh /
15 00012013 72 ?? 72h r
16 00012014 65 ?? 65h e
17 00012015 68 ?? 68h h
18 00012016 2f ?? 2Fh /
19 00012017 73 ?? 73h s
20 00012018 62 ?? 62h b
21 00012019 69 ?? 69h i
22 0001201a 89 ?? 89h
23 0001201b e3 ?? E3h
24 0001201c 50 ?? 50h P
25 0001201d 66 ?? 66h f
26 0001201e 68 ?? 68h h
27 0001201f 2d ?? 2Dh -
28 00012020 66 ?? 66h f
29 00012021 89 ?? 89h
30 00012022 e6 ?? E6h
31 00012023 50 ?? 50h P
32 00012024 56 ?? 56h V
33 00012025 53 ?? 53h S
34 00012026 89 ?? 89h
35 00012027 e1 ?? E1h
36 00012028 b0 ?? B0h
37 00012029 0b ?? 0Bh
38 0001202a cd ?? CDh
39 0001202b 80 ?? 80h
40 0001202c 00 ?? 00h
0x6e
,
0x2f
,
0x72
,
0x65
,
0x68
,
0x2f
,
0x73
,
0x62
,
0x69
,
0x89
,
0xe3
,
0x50
,
0x66
,
0x68
,
0x2d
,
0x66
,
0x89
,
0xe6
,
0x50
,
0x56
,
0x53
,
0x89
,
0xe1
,
0xb0
,
0x0b
,
0xcd
,
0x80
]
(
local_c
=
0
;
*
(
char
*
)
(
param_1
+
local_c
)
!=
'\0'
;
local_c
=
local_c
+
1
)
{
6 *(byte *)(param_1 + local_c) = (*(byte *)(param_1 + local_c) ^ 0x55) + 8;
7 }
8 return param_1;
9}
i
=
0
;
s
[
i
]
!=
'\0'
;
i
=
i
+
1
)
{
5 s[i] = (s[i] ^ 0x55U) + 8;
6 }
7 return s;
8}
append
(
(
i
-
8
)
^
0x55
)
5 return out_list
,
0x6e
,
0x2f
,
0x72
,
0x65
,
0x68
,
0x2f
,
0x73
,
0x62
,
0x69
,
0x89
,
0xe3
,
0x50
,
0x66
,
0x68
,
0x2d
,
0x66
,
0x89
,
0xe6
,
0x50
,
0x56
,
0x53
,
0x89
,
0xe1
,
0xb0
,
0x0b
,
0xcd
,
0x80
]
2payload = decode(enckey)