Attack

of the mutant tags

Álvaro Iradier - @airadier

Attack of the mutant tags!

  • Basic concepts
  • What is a mutant tag?
  • Deployment issues
  • Protect against the mutations
  • Image registry pains
  • Immutable tags
  • Questions

Basic concepts

Registry, image, container

Image registry (i.e. docker.io)

myimage:tag1

myimage:tag2

container-1

container-2

container-3

container-4

container-5

Image tag, manifest, layer...

Image layers

e74c2290...

71bb4d21...

9080b345...

e74c2290...

71bb4d21...

9080b345...

myimage:tag1

1ab50232...

b604fa38...

612bc919...

23456789...

fe5b187a...

myimage repository

tag1

tag2

1ab50232...

Manifest

What is a mutant tag?

What is a mutant tag

myimage:tag1

What is a mutant tag

myimage:tag1

myimage repository

What is a mutant tag

myimage:tag1

1ab50232...

myimage repository

tag1

What is a mutant tag

tag1

1ab50232...

17b345ca...

myimage repository

myimage:tag1

myimage:tag1

What is a mutant tag

tag1

1ab50232...

17b345ca...

612bc919...

myimage repository

myimage:tag1

myimage:tag1

myimage:tag1

What is a mutant tag

tag1

1ab50232...

17b345ca...

612bc919...

myimage repository

myimage@sha256:1ab50232...

myimage@sha256:17b345ca

myimage:tag1
myimage:othertag
myimage@sha256:612c919...

 

othertag

Mutant tag: use cases

  • latest (default if not specified) and variants:
    • alpine
    • slim
    • ...
  • Per-environment:
    • dev
    • sta
    • qa
    • prod
    • ...
  • Versioning:
    • myimage:1.0 -> myimage:1.0.1, myimage:1.0.2,...

Deployment issues

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

couple of hours later in another developer laptop...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run myimage:latest

myimage@sha256:17b345ca

couple of hours later in another developer laptop...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run myimage:latest

myimage@sha256:17b345ca

couple of hours later in another developer laptop...

tomorrow, after a cache cleanup, or in other server...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run myimage:latest

myimage@sha256:17b345ca

docker run myimage:latest

myimage@sha256:612c919...

couple of hours later in another developer laptop...

tomorrow, after a cache cleanup, or in other server...

Kubernetes pod schedule

deploy with spec image: "myimage:latest"

node1

node2 - cordon

Kubernetes pod schedule

myimage@sha256:1ab50232...

deploy with spec image: "myimage:latest"

node1

node2 - cordon

Kubernetes pod schedule

myimage@sha256:1ab50232...

deploy with spec image: "myimage:latest"

node1

node2 - cordon

push other myimage:latest

Kubernetes pod schedule

myimage@sha256:1ab50232...

deploy with spec image: "myimage:latest"

node1

node2

Kubernetes pod schedule

myimage@sha256:1ab50232...

myimage@sha256:17b345ca

deploy with spec image: "myimage:latest"

node1

node2

Kubernetes pod schedule

myimage@sha256:1ab50232...

myimage@sha256:17b345ca

deploy with spec image: "myimage:latest"

node1

node2

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

Pod with image myimage:latest

1

2

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

Admission controller +
image scanner - TOCTOU

push other myimage:latest

6

Pod with image myimage:latest

1

2

3

4

5

7

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

7

8

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

7

8

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

7

8

push other myimage:latest

6

Protect against the mutations

Is this easy to reproduce?

Pushing image process to registry:

  • Start upload
  • For every layer, check and push layer
  • Finally, push image manifest


It is very easy to create a client that gets everything uploaded and ready and delays the last step 

Ways to mitigate it

Avoid tags like latest or similar

 

Don't mutate tags to avoid confusion

 

Use digest instead of tag: myimage@sha256:54c459100e

docker images --digests
kubectl inspect pod | grep sha256
...

Kubernetes resolves digest after scheduling the pod

 

TOCTOU issue: Mutating admission controller

Mutant vs Mutant

https://github.com/sysdiglabs/opa-image-scanner

Image registry pains

Docker registry internals 

Every layer is a content addressable blob

 

Addressed by the sha256 digest of the contents

 

Manifest of the image is another blob JSON doc:

  • Contains digest of image config blob
  • Digests of every image layer

 

 

https://hub.docker.com/_/registry
https://github.com/docker/distribution

 

Manifest example

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1511,
      "digest": "sha256:e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2802957,
         "digest": "sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9"
      }
   ]
}

Tag and digest storage

Base of registry at $DATA/registry/docker/registry/v2

$ docker push myregistry/library/test:mutant creates
# + repositories/library/test/_manifests/tags/mutant/current/link
# + repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link
# + blobs/sha256/b3/b3787bd1.../data
$ cat repositories/library/test/_manifests/tags/mutant/current/link                       sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat blobs/sha256/b3/b3787bd1.../data
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
...


Tag and digest storage

Now push a mutated version of tets:mutant

$ docker push myregistry/library/test:mutan
# ! repositories/library/test/_manifests/tags/mutant/current/link MODIFIED
$ cat repositories/library/test/_manifests/tags/mutant/current/link                       sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0
# = repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link
$ cat repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link
sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0

# = repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/revisions/sha256/5ae12116.../link
# = blobs/sha256/b3/b3787bd1.../data
# + blobs/sha256/5a/5ae12116.../data

Tag and digest storage

So, for each tag mutation:

  • blobs/sha256/<digestPrefix>/<digestPrefix><restOfDigest>/data
    is the blob for manifest file
     
  • repositories/library/test/_manifests/tags/mutant/current/link
    links to the current manifest for that tag
     
  • repositories/<org>/<repo>/_manifests/tags/<name>/index/sha256/<digest>.../link
    and

    repositories/<org>/<repo>/_manifests/revisions/sha256/<digest>/link
    are also links to the corresponding manifest blob
     
  • Index of current and past manifests is kept (for repo and tag)

Garbage collection issues

Garbage collector (mark & sweep):

  • Scans manifests on each repo
  • Marks referenced layers
  • Finally sweeps unused layers to release space

 

Consequence: as all past manifests are kept in the registry, layers for past mutations of a tag are not released

 

Fix: you need to completely remove the repository (all manifests)

The registry V2 API issue

API image removal is by digest:
DELETE /v2/<name>/manifests/<reference>

 

But the listing of images is by tag:

GET /v2/<name>/tags/list

 

Not easy to find all manifests! Need access to storage

 

https://docs.docker.com/registry/spec/api/

The registry V2 API issue

Unexpected behavior: if tag1 and tag2 both point to the same manifest, then deleting tag1 from Harbor UI also deletes tag2


https://github.com/goharbor/harbor/issues/2663

Thanks! Fixed in Harbor 1.11

Immutable tags

Tag immutability

Check before pushing (CI/CD?). Support in some registries like:

Mutable or immutable?

Thanks!

?

Copy of Attack of the mutant tags

By Álvaro José Iradier

Copy of Attack of the mutant tags

  • 360