x86 anti DECODE (PoC)
Axel Tillequin
@bdcht
Rump SSTIC 2014
ia32 instructions
... many "oddities"
(see https://code.google.com/p/corkami/wiki/x86oddities)
prefixes
MORE PFX/SSE oddities
+
66 0F 58 /r : ADDPD,
etc.
testing decoders...
WHAT THE CPU DOES...
what is decoded...
objdump FAIL :
WHAT IS DECODED...
IDA (6.x) FAIL :
WHAT IS DECODED...
LLVM FAIL :
WHAT IS DECODED...
capstone FAIL:
[FIXED]
WHAT IS DECODED...
beaengine : 
Hopper : 
WHAT IS DECODED...
HTE fail :
WTF ??!
(they got it almost right !!!)
conclusion
decoding IA32 is tricky...
- use last prefix within each group
- for SSE2, use pfx groups "precedence" rules (F2/F3 > 66)
- the decoder needs to be EASY to enhance/fix !!!
more testings needed :
- more CPUs (reliable behaviors ??)
- REX/VEX things... (64 bits)
- automated testing ? (ptrace validation)
amoco
github.com/bdcht/amoco
(how to do it and fix it)