This lesson comes last because it is the most intricate, and you do not need it to read or write everyday assembly. But the moment you start modifying binaries — patching an instruction, writing shellcode, hand-assembling a gadget — you need to know how a mnemonic becomes the exact bytes the CPU executes.
This information comes from the ISA Reference, Volume 2A. This document is extremely long and not fun to read. Everything you should need to know will be present in this document.
All x64 instructions follow the same format:
| Field | Length | Required | Definition |
|---|---|---|---|
| Legacy Prefixes | 1-4 bytes | Optional | Override various instructions to their x64 extensions. |
| Opcode (with prefix) | 1-4 bytes | Required | Provides the instruction with various prefixes. |
| MODR/M | 1 byte | Depends on Opcode | Indicates the arguments for the instruction executed. |
| SIB | 1 byte | Depends on Opcode | Used to demonstrate the scale when indexing values. |
| Displacement | 1, 2, 4, or 8 bytes | Depends on Opcode | Offsets to other parts of the program used in an instruction. |
| Immediate | 1, 2, 4, or 8 bytes | Depends on Opcode | Immediate (numeric) values used in an instruction. |
Opcodes stem from a long table containing each instruction. The opcode itself will always be one byte, but the prefixes are where the additional three bytes come in. The most common prefix is the REX Prefix, but others are also used.
This manual is the one I use to determine Opcodes for instructions. Note that instructions might have more than one Opcode based on its arguments, so its imperative you know which Opcode is the correct one for the desired functionality.
Register Extension (REX) is an Opcode used only in 64-bit Assembly to extend the register set and operand size to 64-bit mode.
mov rsi, rax), we need to override the default operand size with the REX prefix.The REX prefix is 1 byte and is packed with the following bits:
7 0
+---+---+---+---+---+---+---+---+
| 0 1 0 0 | W | R | X | B |
+---+---+---+---+---+---+---+---+Here is what these stand for:
| Field | Length | Description |
|---|---|---|
| 0100 | 4 bits | Fixed Bit Pattern |
| W | 1 bit | When W=1, a 64-bit operand is used. Otherwise, the default size operand (32-bit for most instructions) is used. |
| R | 1 bit | An extension to MODRM.reg: Used to include r8-r15 for source registers. |
| X | 1 bit | An extension to SIB.index: Used to include r8-r15 for index operations. |
| B | 1 bit | An extension to MODRM.rm or SIB.base: Used to include r8-r15 as base/destination registers. |
This is very confusing at first. Let's see a couple examples to understand these in action.
| Instruction | REX Value | Explanation |
|---|---|---|
mov rdi, rsi | 48 | We only need 64-bit operands. W=1 only. |
mov r8, rdi | 49 | We're using 64-bit operands (W=1). r8 is a base register, so B=1. |
mov rdi, r8 | 4c | 64-bit operands (W=1), and r8 is a source register (R=1). |
mov qword [rdi + r8], rsi | 4a | 64-bit operands (W=1), and r8 is an index register (X=1). |
mov qword [r8 + r9], r10 | 4f | 64-bit registers, r10 source, r8 base, r9 index (all bits set). |
The MODR/M (Modify Register/Memory) byte encodes a register or an opcode extension, and a register or memory address. The layout of the byte is as follows:
7 0
+---+---+---+---+---+---+---+---+
| mod | reg | rm |
+---+---+---+---+---+---+---+---+Here is what these stand for:
| Field | Length | Description |
|---|---|---|
MODRM.mod | 2 bits | Defines the type of addressing used. The most common are 11 (register-to-register) or 00 (register or [reg]) |
MODRM.reg | 3 bits | Specifies the source register operand with an optional displacement. |
MODRM.rm | 3 bits | Specifies the destination register operand with an optional displacement. |
Instances where MODRM.mod is not 00 or 11 is when indexing is used. MODRM.mod = 01 when 8-bit displacement is required, and MODRM.mod = 10 when 32-bit displacement is required.
The register table used is the following:
000 | 001 | 010 | 011 | 100 | 101 | 110 | 111 |
|---|---|---|---|---|---|---|---|
rax or r8 | rcx or r9 | rdx or r10 | rbx or r11 | rsp or r12 | rbp or r13 | rsi or r14 | rdi or r15 |
For example, when using a mov rdi, rsi instruction, here is the MODR/M byte:
MODRM.mod = 11.rsi, so MODRM.reg = 110.rdi, so MODRM.rm = 111.This makes the byte 11 110 111 = 1111 0111 = f7.
Put the prefix and this byte together and mov rdi, rsi assembles to 48 89 f7 — the 48 REX, the 89 opcode for mov r/m64, r64, and the f7 MODR/M we just built. The register-extension bits are just as predictable; assemble a few movs and check each prefix byte against the REX table above:
objdump -d -M intel enc0000000000401000 <_start>: 401000: 48 89 f7 mov rdi,rsi 401003: 49 89 f8 mov r8,rdi 401006: 4c 89 c7 mov rdi,r8 401009: c3 retmov rdi, rsi is 48 — W only. mov r8, rdi is 49 — W plus B, because r8 is the destination (base) register. mov rdi, r8 is 4c — W plus R, because r8 is now the source register. Exactly the pattern the table predicts.
Displacements are used in call and jmp instructions where an offset is required to another location in memory. Rather than using labels to define where to jump, the compiler instead elects to hard code the static offset to that location. This is done by counting the number of bytes between the two locations.
When calculating the offset, the compiler always chooses to compute the offset based on the address of the next instruction. Consider this example:
0x000000000040100a <+4>: e8 f1 ff ff ff call 0x401000 <win>
0x000000000040100f <+9>: 48 31 ff xor rdi,rdiThe first byte of the call instruction is its Opcode (e8). Then, the next four bytes represent the displacement. Since the win function is before the main function, the offset is negative (written in two's complement). The offset we see is 0xfffffff1 = -0xf. The address used for the displacement is the next instruction, or 0x40100f. Therefore, our offset makes sense: 0x401000-0x40100f=-0xf.
Even if there is no next instruction, we know the next instruction would be at the current instruction's address plus the length of the instruction. Since the displacement is almost always four bytes for a call, we can gather the next instruction is at current_instruction + 5.
Immediates are written the same way as displacements. Therefore, a mov rdi, 0x1 instruction would be compiled as:
0x000000000040100a <+4>: bf 01 00 00 00 mov edi,0x1Since the default operand size is 32 bits, it uses edi instead.