Table of Contents

IronForgeCyber

Courses

Resources

Blog

Log In



IronForgeCyber

AboutPrivacyTerms

© 2026 IronForgeCyber. All rights reserved.

  1. courses
  2. asm-101
  3. instruction-encoding
6 min read

Instruction Encoding

This lesson comes last because it is the most intricate, and you do not need it to read or write everyday assembly. But the moment you start modifying binaries — patching an instruction, writing shellcode, hand-assembling a gadget — you need to know how a mnemonic becomes the exact bytes the CPU executes.

This information comes from the ISA Reference, Volume 2A. This document is extremely long and not fun to read. Everything you should need to know will be present in this document.

Opcodes

All x64 instructions follow the same format:

FieldLengthRequiredDefinition
Legacy Prefixes1-4 bytesOptionalOverride various instructions to their x64 extensions.
Opcode (with prefix)1-4 bytesRequiredProvides the instruction with various prefixes.
MODR/M1 byteDepends on OpcodeIndicates the arguments for the instruction executed.
SIB1 byteDepends on OpcodeUsed to demonstrate the scale when indexing values.
Displacement1, 2, 4, or 8 bytesDepends on OpcodeOffsets to other parts of the program used in an instruction.
Immediate1, 2, 4, or 8 bytesDepends on OpcodeImmediate (numeric) values used in an instruction.

Opcodes stem from a long table containing each instruction. The opcode itself will always be one byte, but the prefixes are where the additional three bytes come in. The most common prefix is the REX Prefix, but others are also used.

This manual is the one I use to determine Opcodes for instructions. Note that instructions might have more than one Opcode based on its arguments, so its imperative you know which Opcode is the correct one for the desired functionality.

Register Extension (REX)

Register Extension (REX) is an Opcode used only in 64-bit Assembly to extend the register set and operand size to 64-bit mode.

Why do we need this?
Even though we are compiling 64-bit assembly, the default operand size is still 32 bits. Therefore, in instructions that require the use of 64 bit operands (i.e., mov rsi, rax), we need to override the default operand size with the REX prefix.

The REX prefix is 1 byte and is packed with the following bits:

  7                           0
+---+---+---+---+---+---+---+---+
| 0   1   0   0 | W | R | X | B |
+---+---+---+---+---+---+---+---+

Here is what these stand for:

FieldLengthDescription
01004 bitsFixed Bit Pattern
W1 bitWhen W=1, a 64-bit operand is used. Otherwise, the default size operand (32-bit for most instructions) is used.
R1 bitAn extension to MODRM.reg: Used to include r8-r15 for source registers.
X1 bitAn extension to SIB.index: Used to include r8-r15 for index operations.
B1 bitAn extension to MODRM.rm or SIB.base: Used to include r8-r15 as base/destination registers.

This is very confusing at first. Let's see a couple examples to understand these in action.

InstructionREX ValueExplanation
mov rdi, rsi48We only need 64-bit operands. W=1 only.
mov r8, rdi49We're using 64-bit operands (W=1). r8 is a base register, so B=1.
mov rdi, r84c64-bit operands (W=1), and r8 is a source register (R=1).
mov qword [rdi + r8], rsi4a64-bit operands (W=1), and r8 is an index register (X=1).
mov qword [r8 + r9], r104f64-bit registers, r10 source, r8 base, r9 index (all bits set).

Register Encoding

The MODR/M (Modify Register/Memory) byte encodes a register or an opcode extension, and a register or memory address. The layout of the byte is as follows:

  7                           0
+---+---+---+---+---+---+---+---+
|  mod  |    reg    |     rm    |
+---+---+---+---+---+---+---+---+

Here is what these stand for:

FieldLengthDescription
MODRM.mod2 bitsDefines the type of addressing used. The most common are 11 (register-to-register) or 00 (register or [reg])
MODRM.reg3 bitsSpecifies the source register operand with an optional displacement.
MODRM.rm3 bitsSpecifies the destination register operand with an optional displacement.

Instances where MODRM.mod is not 00 or 11 is when indexing is used. MODRM.mod = 01 when 8-bit displacement is required, and MODRM.mod = 10 when 32-bit displacement is required.

The register table used is the following:

000001010011100101110111
rax or r8rcx or r9rdx or r10rbx or r11rsp or r12rbp or r13rsi or r14rdi or r15

For example, when using a mov rdi, rsi instruction, here is the MODR/M byte:

  1. This is a register to register operation, so MODRM.mod = 11.
  2. The source register is rsi, so MODRM.reg = 110.
  3. The destination register is rdi, so MODRM.rm = 111.

This makes the byte 11 110 111 = 1111 0111 = f7.

Put the prefix and this byte together and mov rdi, rsi assembles to 48 89 f7 — the 48 REX, the 89 opcode for mov r/m64, r64, and the f7 MODR/M we just built. The register-extension bits are just as predictable; assemble a few movs and check each prefix byte against the REX table above:

objdump
$ objdump -d -M intel enc0000000000401000 <_start>:  401000:       48 89 f7                mov    rdi,rsi  401003:       49 89 f8                mov    r8,rdi  401006:       4c 89 c7                mov    rdi,r8  401009:       c3                      ret

mov rdi, rsi is 48 — W only. mov r8, rdi is 49 — W plus B, because r8 is the destination (base) register. mov rdi, r8 is 4c — W plus R, because r8 is now the source register. Exactly the pattern the table predicts.

Displacements and Immediates

Displacements are used in call and jmp instructions where an offset is required to another location in memory. Rather than using labels to define where to jump, the compiler instead elects to hard code the static offset to that location. This is done by counting the number of bytes between the two locations.

This is rather impractical for humans because we must retroactively determine the number of bytes between two locations. This is especially true if we decide to add more code or jump to code after the current instruction.

When calculating the offset, the compiler always chooses to compute the offset based on the address of the next instruction. Consider this example:

0x000000000040100a <+4>:     e8 f1 ff ff ff  call   0x401000 <win>
0x000000000040100f <+9>:     48 31 ff        xor    rdi,rdi

The first byte of the call instruction is its Opcode (e8). Then, the next four bytes represent the displacement. Since the win function is before the main function, the offset is negative (written in two's complement). The offset we see is 0xfffffff1 = -0xf. The address used for the displacement is the next instruction, or 0x40100f. Therefore, our offset makes sense: 0x401000-0x40100f=-0xf.

Even if there is no next instruction, we know the next instruction would be at the current instruction's address plus the length of the instruction. Since the displacement is almost always four bytes for a call, we can gather the next instruction is at current_instruction + 5.

Immediates are written the same way as displacements. Therefore, a mov rdi, 0x1 instruction would be compiled as:

0x000000000040100a <+4>:     bf 01 00 00 00  mov    edi,0x1

Since the default operand size is 32 bits, it uses edi instead.