How to backdoor the Linux Kernel

(and fail?)

Konstantin Ryabitsev

Linux Security Summit

Vancouver, 2023

 

https://slides.com/mricon/backdoor-and-fail

  • I live in Montréal, Québec

  • Linux admin since 1997

  • (formerly) head of LF Infrastructure Security

    • now just head of Core IT Projects team

  • in charge of kernel.org for the past 10+ years

    • keeper of grounds and keys

About me

Did anyone ever, you know, approach you with some shady offer?

– Common Question

Lite

Price $149 /mo

One zero-day

Two backdoors

7-day turnaround

Plausible deniability

PRO

Price $299 /mo

NEW

Three zero-days

Three backdoors

7-day turnaround

Plausible deniability

PREMIUM

Price $599 /mo

Up to five zero-days

Five backdoors

3-day turnaround

Plausible deniability

No, they haven't

On parle français, se habla español, 厕所在哪里

I'm kidding

Please don't contact me. It will be a very short awkward conversation and then I'll have to figure out how to contact both the FBI and the CSIS.

  • A hidden vulnerability

  • Installed by the victim

    • i.e. not added by attacker after a compromise

  • Allowing remote access

  • Or elevating privileges

  • Or exfiltrating sensitive info

    • encryption keys

    • or secret docs

WHat do I mean by "backdoor"

This is not "How to Backdoor Linux for dummies."
Check if you're in the right talk.

Attacking the Pipeline

Submit

  • developer submits patches
  • maintainer reviews patches
  • maintainer accepts patches

Merge

  • Maintainer sends git pull
  • Linus reviews git pull
  • Linus merges git pull

Publish

  • Linus tags a release
  • Greg KH signs a tarball
  • user downloads tarball
  • Can you sneak malicious code inside a patch?
  • Can you hack/threaten/bribe LF admins to add malicious code to a patch?
  • Can you trick the maintainer to apply a malicious patch?

Submission stage

  • Can you send a malicious pull request to Linus?
  • Can you hack Linus and modify his repo?
  • Can you make Linus sign a malicious tag?

Merge stage

  • Can you make Greg KH sign the wrong tarball?
  • Can you hack/threaten/bribe LF admins to replace a tarball with a malicious one?
  • Can you MiTM a download from kernel.org?

Publish stage

  • Developer submits code
  • Maintainer reviews code
  • Maintainer accepts code
  • Maintainer sends a pull request

Attacking the patch workflow

What you think the workflow is like

What it's actually like

  • Unlikely to succeed for complex backdoors
    • maybe a simple vulnerability to elevate privs
    • not complex code to sniff and exfiltrate secrets
  • Hiding complex backdoors inside huge patchsets?
    • huge patchsets are rare for critical code paths
    • there are many eyes reviewing those patches
  • backdooring a relatively obscure device driver?
    • just volunteer to be a maintainer for that driver

Overtly sending malicious code

  • Probably not the best poster case
    • they were from “James Bond”
    • they were not trying very hard
    • the experience left us stirred, not shaken

UMN and "Hypocrite Commits"

  • Patches reviewed vs. patches applied
    • patches retrieved from lore or patchwork
      • "I remember reviewing that series"
      • "It's from Alex, whom I trust"
      • "DKIM signatures check out"
    • "here is a v12 with minor wording changes"

Covertly sneaking in malicious code

  • b4 verifications
    • DKIM signatures
      • fragile on many lists
      • requires trust in domain admins
      • typosquattable (bigco.com vs big-co.net)
    • Patatt signatures
      • DKIM-like, but end-to-end
      • Tailored to patches, resistant to ML junk
      • Can use PGP, SSH, or ED25519 keys
      • Require keyring management

End-to-end trust

  • b4 shazam
    • can turn a patch series into a pull request
    • shazam first, then review
  • b4 diff
    • can show you a diff between series
  • b4 send
    • simplifies series management
    • auto-signs before sending
  • b4 kr
    • keyring management (forthcoming)

New(er) features in b4

  • Likely will still be found out
    • there are eyes watching commits
    • (but they may not be talking)
  • Intentional bugs can still be caught by CI/bots
    • fuzzers
    • integration bots

If a malicious series got applied

  • Likely will be almost immediately discovered
  • There are tricks to make this more successful
    • arrange for their laptop to be stolen first
    • or their disk suddenly corrupted
    • force them to do a fresh full clone of the repo
  • Signing git commits helps
    • Yes, I know this is super annoying
    • it's an effective way to quickly verify the repository after a full re-clone
    • or when using shared repositories

Can a LF admin backdoor a repo?

  • Publishes information about each commit
  • Replicates to multiple mirrors
  • Tamper-evident (NOT tamper-resistant)
  • Can be used to exonerate developers
    • and point fingers at kernel.org admins

Kernel.org transparency log

  • kernel.org has been rooted before
    • the worst has already happened
    • no guarantee it won't happen again
  • the #1 rule of kernel.org is "do not trust kernel.org"
    • we've always promoted zero-trust workflows
    • we promote end-to-end cryptography
    • we receive bits, we store bits, we send bits
      • it is your responsibility to verify them
      • we wrote tools to help with that

Trusting devs, not infra

  • Linus receives a pull request
  • Linus merges a pull request
  • Linus tags a release

Attacking Linus

  • All pull requests must be PGP-signed
    • assures the integrity of the entire git history
      • assuming we trust sha1
        • (which we still do)
          • ("ish")
  • Linus does verify signatures on all pull requests
  • It is very hard to get into Linus's keyring

This has been protected for a while

  • Yes, but with limited success
    • They can only backdoor their own patches
    • The original author will likely notice changes
      • they are very protective of their code
      • this may break CI in suspicious ways
  • If found out, their reputation will be destroyed
    • you probably want a key subsystem maintainer
    • so, this would be super expensive
    • there are much cheaper ways

Can you bribe a maintainer?

  • Yes, this would be fairly effective
  • There is a high chance it'll get caught
    • but it may not, until it's too late
    • There are people watching all commits
      • but they may not be talking to us
  • Still only effective for planting "intentional bugs"
    • large changes would be too noticeable

Can you hack their workstation?

  • Protect your digital identity above all
    • This usually means your crytographic keys
  • Keep your PGP keys on smartcards
    • you can get one for free if you're a maintainer
    • or get your DAYJOB to buy you one
  • Try to isolate your work env from your play env

Protect your workstations!

  • Greg KH tags a release
  • Kernel.org publishes a tarball
  • User (or distro) downloads the tarball

Attacking Downloads

  • All tarballs are signed with Greg's key
    • sigs are actually git notes in the stable repo
    • LF infra has no access to Greg's key
    • kernel.org verifies signature before release is allowed to happen
  • The few times we messed up and broke the process, we were told about it very quickly
  • You are responsible for checking the signature of the tarball you downloaded

Also been protected for a while

  • We publish a tool that downloads random tarballs from kernel.org mirrors and checks their signatures
  • The script runs from several undisclosed locations
  • You can help run it, too!
  • https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/sig-prover.py

The sig-prover tool

You don't want to backdoor the kernel.

  • For every powerful state-level actor, there is another equally or more powerful state-level actor who is paying someone to watch all Linux commits

    • It’s like building a shared arsenal of powerful weapons that anyone can launch

  • For every powerful criminal syndicate there is another powerful criminal syndicate

    • They all keep money in the same Swiss banks

It won't be just *your* backdoor

If you wait long enough, someone will backdoor the kernel for you.

– Kees Cook [citation needed]

KSPP Report

Just wait until a critical vulnerability is fixed — it's not like manufacturers patch their kernels or anything.

– Greg KH [citation needed]

Are there backdoors in the Linux Kernel?

(c) user ebygomm on Flickr

  • Despite the perception of "bazaar", the Linux development model is actually quite rigorous
  • There is transparency and oversight at many levels
    • pre-commit and post-commit
  • Being a maintainer is a well-paying job with long-term job security
    • bribing a maintainer is super expensive
    • there are cheaper ways to root someone
  • Having root on kernel.org infrastructure gets you almost nothing in terms of attacking the Linux source code

Intentional? Probably not.

  • The kernel you are using right now probably has a bad enough vulnerability in it to lead to local privilege escalation
    • just because it's not known to us, doesn't mean it's not known to the "bad guys"
    • and bad guys, what if it's known to your worst enemies and will be used against you?
  • Reporting critical bugs in the Linux kernel is in everyone's interest.
  • security@kernel.org

Unintentional? Probably yes.

FIN