Evolutive malware
0xKylm
Evolution can be decomposed into 3 steps:
1. Selection — Choose the fittest individuals based on a fitness function.
2. Reproduction (Crossover) — Combine genetic material to create offspring inheriting beneficial traits.
3. Mutation — Random changes to maintain diversity and escape local minima.
These 3 steps form one generation, repeated until a stopping criterion is met.
Genetic algorithms
An AST (Abstract Syntax Tree) is a way to represent a program as a tree structure:
- Nodes → operations (functions, operators)
- Leaves → terminals (variables, constants)
Widely used in compilers and Genetic Programming to represent and manipulate programs.
AST (abstract syntax tree)
DEAP, for distributed evolutionary algorithm in python, is a library that permit to use evolutionary algorithm for asts, and output & test them as python code
DEAP, The genetic programmation library
Objectif and search for minima
Finding the Global Minimum
The main challenge of GP and other ML algorithms: finding the global minimum rather than getting trapped in local minima.
Strategies:
- Gradient descent (SGD, Adam)
- Simulated annealing
- Particle swarm optimization
- NSGA-II / NSGA-III (multi-objective)
- Genetic algorithms / GP
Asts reproduction & mutation
During the algorithm, ASTs evolve through two main operations:
Mutation
- Modify, add, or remove a number of nodes within a single individual
- Introduces genetic diversity
- Helps escape local minima
- Types: point mutation, subtree mutation, hoist mutation
Crossover (Reproduction)
- Combine, concatenate, or slice parts from parent ASTs
- Exchanges subtrees between individuals
- Exploits beneficial traits from fit individuals
- Single or multi-point crossover
Setting up DEAP
To use DEAP, we need:
Primitives
The building blocks of our ASTs:
- Functions: add, sub, mul, if_then_else(a, b, c), custom operators...
- Terminals: constants and variables
> Grouped into a PrimitiveSetTyped or Creator
Algorithm configuration
- Selection method: tournament, roulette, rank...
- Crossover operator: one-point, uniform...
- Mutation operator: subtree, node replacement...
- Fitness function: evaluates individuals
- Hyperparameters: population size, generations, mutation/crossover rates
toolbox = base.Toolbox()
toolbox.register("expr", gen_tree, pset=pset, min_d=3, max_d=7)
toolbox.register("individual", toolbox.expr)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)
toolbox.register("compile", gp.compile, pset=pset)
def evaluate_multi(ind):
ux, uy = count_vars(ind)
if ux == 0 and uy == 0: return (1e10, 0, 0)
if ux == 0 or uy == 0: return (1e9, 0, 0)
try: func = toolbox.compile(expr=ind)
except: return (1e8, 0, 0)
tests = [(1,1),(2,3),(5,7),(10,4),(3,8),(6,2),(4,9),(7,5)]
err = 0.0
for x, y in tests:
try:
r = func(x, y)
if math.isnan(r) or math.isinf(r): err += 1e6
else: err += abs(r - TARGET_VALUE)
except: err += 1e6
obj1 = err / len(tests)
cf = count_cf(ind)
trig = count_trig(ind)
complexity = cf * 2 + trig
obj2 = -complexity
size_score = len(ind) + ind.height * 5
obj3 = -size_score
return (obj1, obj2, obj3)
Primitives
Primitives are the building blocks we can use in our AST nodes:
Mathematical expressions
- add, sub, mul, div, sin, cos...
Classic functions for symbolic regression
CFG-based primitives (Context-Free Grammar):
- if_then_else(a, b, c)
- while(cond, body)
- for(init, cond, step, body)
- Logical operators: and, or, not
- Custom domain-specific functions
Usage in malware dev
GP, can also be used in malware dev & obfuscation, by exemple here for opaque predicate
Generating code that generate code
Using the same idea, but based on a vm, it's possible to use gp use to use the vm opcode to generate code
static inline i32 vm_and(i32 a, i32 b) {
volatile u32 _a = (u32)a, _b = (u32)b;
volatile u32 r = 0;
volatile u32 T[] = {0x11111111u, 0x22222222u, 0x33333333u, 0x44444444u};
volatile u32 K[] = {_a, _b, _a^_b, ~_a, ~_b};
volatile u32 d = (_a - _b) & 3;
for(volatile u32 i = 0; i < 2; i++) { r -= T[0] - T[0]; }
for(volatile u32 i = 0; i < 2; i++) { r += K[0] - K[0]; }
for(volatile u32 i = 0; i < 2; i++) { r += K[0] - K[0]; }
switch(d) {
case 0: r = _a & _b; switch((r ^ 0) & 3) {
case 0: r += (K[0] ^ K[1]) & 0; break;
case 1: r ^= K[1] ^ K[1]; break;
case 2: r &= T[0] | 0xFFFFFFFF; break;
case 3: r ^= K[0] ^ K[0]; break;
case 4: r -= T[0] - T[0]; break;
default: r += T[0] - T[0]; break;
}; r &= T[0] | 0xFFFFFFFF; r ^= T[0] ^ T[0]; r += K[0] - K[0]; break;
case 1: r = _a; r &= _b; switch((r ^ 1) & 3) {
case 0: r -= T[0] - T[0]; break;
case 1: r -= T[0] - T[0]; break;
case 2: r ^= T[1] ^ T[1]; break;
case 3: r += T[0] - T[0]; break;
case 4: r += T[0] - T[0]; break;
default: r ^= T[1] ^ T[1]; break;
}; r += (K[0] ^ K[1]) & 0; r &= T[0] | 0xFFFFFFFF; r ^= T[1] ^ T[1]; break;
case 2: r = (_a ^ ~_b) & _a; switch((r ^ 2) & 3) {
case 0: r &= (K[4] | ~0u); break;
case 1: r += T[0] - T[0]; break;
case 2: r ^= K[1] ^ K[1]; break;
case 3: r &= (K[4] | ~0u); break;
case 4: r ^= K[0] ^ K[0]; break;
default: r &= T[0] | 0xFFFFFFFF; break;
}; r &= T[0] | 0xFFFFFFFF; r ^= K[2] ^ K[2]; r &= T[0] | 0xFFFFFFFF; break;
case 3: r = (_a ^ ~_b) & _a; switch((r ^ 3) & 3) {
case 0: r &= T[0] | 0xFFFFFFFF; break;
case 1: r ^= K[1] ^ K[1]; break;
case 2: r += d - d; break;
case 3: r ^= d ^ d; break;
case 4: r -= T[0] - T[0]; break;
default: r -= T[0] - T[0]; break;
}; r ^= d ^ d; r ^= K[2] ^ K[2]; r &= (K[4] | ~0u); break;
}
return (i32)r;
}
Evolutive malware
By 0xkylm
