Merge my pull request
How a rogue contributor nearly backdoored a huge percent of Linux servers
xz, liblzma
Compression
Compression
How did the contributor gain trust and access?
Lasse
- Original maintainer
- Based in Finland
- Maintained xz alone for many years
- Left cleaning up the mess
Jia
- Became a maintainer
- Introduced the backdoor
- Disappeared
jia's first PR
jia's first merged commit
Enter Jigar
Jigar starts digging at Lasse
Jia is sympathetic
Jigar repeats
The pile-on continues
The door opens
But the pressure continues
Lasse reveals his situation
Jigar continues
The situation is exploited
Jia is added as a maintainer
Lasse lets Jia takeover
Jia changes security contacts
ifunc support is added by unknown committer
... And then Jia disables a security warning
The attack begins
Hidden in binary files
Built and deployed
Valgrind warnings disabled
A clever trick
The target
- Debian-based systems
- Running sshd
- Installed binaries from build servers (i.e not built from source)
The mechanism
- If sshd is running, add a code path that accepts a particular ssh key
- Grant root privileges to that user
- Embed payload in a ssh key
The potential impact
- Most servers are running Debian or Ubuntu
- The backdoor could've been included in other distros
- Mac users had xz installed via Homebrew
Pressure is applied to Debian to update xz
... And also Ubuntu
The backdoor is discovered
Distros rush to revert
5.4.6
5.3.4
5.6.2-really-5.4.6
Known bad versions
5.6.0
5.6.1
Versions Jia had maintainer access to
5.4.2 to 5.6.1
Multiple emails?
Where in the world?
Where in the world?
Timezone strangeness
"People" who don't exist?
- 5+ years experience with C
- Comfortable using hex editors to modify binaries
- Familiarity with open source security projects
- Able to seem friendly and helpful
- Knowledgeable of Linux security architecture
- Know how to modify git metadata
- Extensive use of build systems
Fictional job ad:
Wanted! SecOps developer
What was the reaction of the open source community?
Raised awareness of social engineering
Lasse reviews the commits
Distros discuss dropping dependencies
What are the open questions that aren't resolved yet?
Who is Jia?
Can Lasse be trusted?
How many other repos have been through this?
Can open source rely on good intentions?
How did our company respond to the incident?
Take it seriously, but don't cause panic
- Figure out the potential impact
- Identify steps needed
- Roll out information
- Check-in while it's on-going
- Post mortem
What are some learnings for the future from this incident?
Indirectly or directly, the dependency chains are massive
Trust can be broken and hard to repair
Open source means people can rally together to respond
Tertiary news sources do not accurately report on issues
xz backdoor
By Noah Hall
