How to backdoor the Linux Kernel

(and fail?)

Konstantin Ryabitsev

Linux Security Summit

Vancouver, 2023

https://slides.com/mricon/backdoor-and-fail

• I live in Montréal, Québec

• Linux admin since 1997

• (formerly) head of LF Infrastructure Security

  • now just head of Core IT Projects team

• in charge of kernel.org for the past 10+ years

  • keeper of grounds and keys

About me

〞

Did anyone ever, you know, approach you with some shady offer?

– Common Question

No, they haven't

On parle français, se habla español, 厕所在哪里

Please don't contact me. It will be a very short awkward conversation and then I'll have to figure out how to contact both the FBI and the CSIS.

• A hidden vulnerability

• Installed by the victim

  • i.e. not added by attacker after a compromise

• Allowing remote access

• Or elevating privileges

• Or exfiltrating sensitive info

  • encryption keys

  • or secret docs

WHat do I mean by "backdoor"

This is not "How to Backdoor Linux for dummies."
Check if you're in the right talk.

Attacking the Pipeline

• Can you sneak malicious code inside a patch?
• Can you hack/threaten/bribe LF admins to add malicious code to a patch?
• Can you trick the maintainer to apply a malicious patch?

Submission stage

• Can you send a malicious pull request to Linus?
• Can you hack Linus and modify his repo?
• Can you make Linus sign a malicious tag?

Merge stage

• Can you make Greg KH sign the wrong tarball?
• Can you hack/threaten/bribe LF admins to replace a tarball with a malicious one?
• Can you MiTM a download from kernel.org?

Publish stage

• Developer submits code
• Maintainer reviews code
• Maintainer accepts code
• Maintainer sends a pull request

Attacking the patch workflow

What you think the workflow is like

What it's actually like

• Unlikely to succeed for complex backdoors
  • maybe a simple vulnerability to elevate privs
  • not complex code to sniff and exfiltrate secrets
• Hiding complex backdoors inside huge patchsets?
  • huge patchsets are rare for critical code paths
  • there are many eyes reviewing those patches
• backdooring a relatively obscure device driver?
  • just volunteer to be a maintainer for that driver

Overtly sending malicious code

• Probably not the best poster case
  • they were from “James Bond”
  • they were not trying very hard
  • the experience left us stirred, not shaken

UMN and "Hypocrite Commits"

• Patches reviewed vs. patches applied
  • patches retrieved from lore or patchwork
    • "I remember reviewing that series"
    • "It's from Alex, whom I trust"
    • "DKIM signatures check out"
  • "here is a v12 with minor wording changes"

Covertly sneaking in malicious code

• b4 verifications
  • DKIM signatures
    • fragile on many lists
    • requires trust in domain admins
    • typosquattable (bigco.com vs big-co.net)
  • Patatt signatures
    • DKIM-like, but end-to-end
    • Tailored to patches, resistant to ML junk
      
    • Can use PGP, SSH, or ED25519 keys
      
    • Require keyring management
      

End-to-end trust

• b4 shazam
  • can turn a patch series into a pull request
  • shazam first, then review
• b4 diff
  • can show you a diff between series
• b4 send
  • simplifies series management
  • auto-signs before sending
• b4 kr
  • keyring management (forthcoming)

New(er) features in b4

• Likely will still be found out
  • there are eyes watching commits
  • (but they may not be talking)
• Intentional bugs can still be caught by CI/bots
  • fuzzers
  • integration bots

If a malicious series got applied

• Likely will be almost immediately discovered
• There are tricks to make this more successful
  • arrange for their laptop to be stolen first
  • or their disk suddenly corrupted
  • force them to do a fresh full clone of the repo
• Signing git commits helps
  • Yes, I know this is super annoying
  • it's an effective way to quickly verify the repository after a full re-clone
  • or when using shared repositories

Can a LF admin backdoor a repo?

• Publishes information about each commit
• Replicates to multiple mirrors
• Tamper-evident (NOT tamper-resistant)
• Can be used to exonerate developers
  • and point fingers at kernel.org admins

Kernel.org transparency log

• kernel.org has been rooted before
  • the worst has already happened
  • no guarantee it won't happen again
• the #1 rule of kernel.org is "do not trust kernel.org"
  • we've always promoted zero-trust workflows
  • we promote end-to-end cryptography
  • we receive bits, we store bits, we send bits
    • it is your responsibility to verify them
    • we wrote tools to help with that

Trusting devs, not infra

• Linus receives a pull request
• Linus merges a pull request
• Linus tags a release

Attacking Linus

• All pull requests must be PGP-signed
  • assures the integrity of the entire git history
    • assuming we trust sha1
      • (which we still do)
        • ("ish")
• Linus does verify signatures on all pull requests
• It is very hard to get into Linus's keyring

This has been protected for a while

• Yes, but with limited success
  • They can only backdoor their own patches
  • The original author will likely notice changes
    • they are very protective of their code
    • this may break CI in suspicious ways
• If found out, their reputation will be destroyed
  • you probably want a key subsystem maintainer
  • so, this would be super expensive
  • there are much cheaper ways

Can you bribe a maintainer?

• Yes, this would be fairly effective
• There is a high chance it'll get caught
  • but it may not, until it's too late
  • There are people watching all commits
    • but they may not be talking to us
• Still only effective for planting "intentional bugs"
  • large changes would be too noticeable

Can you hack their workstation?

• Protect your digital identity above all
  • This usually means your crytographic keys
• Keep your PGP keys on smartcards
  • you can get one for free if you're a maintainer
  • or get your DAYJOB to buy you one
• Try to isolate your work env from your play env

Protect your workstations!

• Greg KH tags a release
• Kernel.org publishes a tarball
• User (or distro) downloads the tarball

Attacking Downloads

• All tarballs are signed with Greg's key
  • sigs are actually git notes in the stable repo
  • LF infra has no access to Greg's key
  • kernel.org verifies signature before release is allowed to happen
• The few times we messed up and broke the process, we were told about it very quickly
• You are responsible for checking the signature of the tarball you downloaded

Also been protected for a while

• We publish a tool that downloads random tarballs from kernel.org mirrors and checks their signatures
• The script runs from several undisclosed locations
• You can help run it, too!
• https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/sig-prover.py

The sig-prover tool

• For every powerful state-level actor, there is another equally or more powerful state-level actor who is paying someone to watch all Linux commits

  • It’s like building a shared arsenal of powerful weapons that anyone can launch

• For every powerful criminal syndicate there is another powerful criminal syndicate

  • They all keep money in the same Swiss banks

It won't be just *your* backdoor

〞

If you wait long enough, someone will backdoor the kernel for you.

– Kees Cook [citation needed]

KSPP Report

〞

Just wait until a critical vulnerability is fixed — it's not like manufacturers patch their kernels or anything.

– Greg KH [citation needed]

(c) user ebygomm on Flickr

• Despite the perception of "bazaar", the Linux development model is actually quite rigorous
• There is transparency and oversight at many levels
  • pre-commit and post-commit
• Being a maintainer is a well-paying job with long-term job security
  • bribing a maintainer is super expensive
  • there are cheaper ways to root someone
• Having root on kernel.org infrastructure gets you almost nothing in terms of attacking the Linux source code

Intentional? Probably not.

• The kernel you are using right now probably has a bad enough vulnerability in it to lead to local privilege escalation
  • just because it's not known to us, doesn't mean it's not known to the "bad guys"
  • and bad guys, what if it's known to your worst enemies and will be used against you?
• Reporting critical bugs in the Linux kernel is in everyone's interest.
• security@kernel.org

Unintentional? Probably yes.

FIN