Basic concepts

Registry, image, container

Image registry (i.e. docker.io)

Image tag, manifest, layer...

Image layers

e74c2290...

71bb4d21...

9080b345...

e74c2290...

71bb4d21...

9080b345...

What is a mutant tag?

What is a mutant tag

myimage:tag1

What is a mutant tag

myimage:tag1

myimage repository

What is a mutant tag

myimage:tag1

1ab50232...

myimage repository

What is a mutant tag

tag1

1ab50232...

17b345ca...

myimage repository

myimage:tag1

What is a mutant tag

tag1

1ab50232...

17b345ca...

612bc919...

myimage repository

myimage:tag1

What is a mutant tag

tag1

1ab50232...

17b345ca...

612bc919...

myimage repository

myimage@sha256:1ab50232...

myimage@sha256:17b345ca

myimage:tag1
myimage:othertag
myimage@sha256:612c919...

 

othertag

Mutant tag: use cases

• latest (default if not specified) and variants:
  • alpine
  • slim
  • ...
• Per-environment:
  • dev
  • sta
  • qa
  • prod
  • ...
• Versioning:
  • myimage:1.0 -> myimage:1.0.1, myimage:1.0.2,...

Deployment issues

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

couple of hours later in another developer laptop...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run myimage:latest

myimage@sha256:17b345ca

couple of hours later in another developer laptop...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run myimage:latest

myimage@sha256:17b345ca

couple of hours later in another developer laptop...

tomorrow, after a cache cleanup, or in other server...

docker run

docker run myimage:latest

myimage@sha256:1ab50232...

docker run myimage:latest

myimage@sha256:17b345ca

docker run myimage:latest

myimage@sha256:612c919...

couple of hours later in another developer laptop...

tomorrow, after a cache cleanup, or in other server...

Kubernetes pod schedule

deploy with spec image: "myimage:latest"

node1

node2 - cordon

Kubernetes pod schedule

myimage@sha256:1ab50232...

deploy with spec image: "myimage:latest"

node1

node2 - cordon

Kubernetes pod schedule

myimage@sha256:1ab50232...

deploy with spec image: "myimage:latest"

node1

node2 - cordon

push other myimage:latest

Kubernetes pod schedule

myimage@sha256:1ab50232...

deploy with spec image: "myimage:latest"

node1

node2

Kubernetes pod schedule

myimage@sha256:1ab50232...

myimage@sha256:17b345ca

deploy with spec image: "myimage:latest"

node1

node2

Kubernetes pod schedule

myimage@sha256:1ab50232...

myimage@sha256:17b345ca

deploy with spec image: "myimage:latest"

https://asciinema.org/a/snp1QZ6oYe3W1MFOMJjscGtUT

node1

node2

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

Pod with image myimage:latest

1

2

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

Admission controller +
image scanner - TOCTOU

push other myimage:latest

6

Pod with image myimage:latest

1

2

3

4

5

7

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

7

8

Admission controller +
image scanner - TOCTOU

Pod with image myimage:latest

1

2

3

4

5

7

8

Admission controller +
image scanner - TOCTOU

Protect against the mutations

Is this easy to reproduce?

Pushing image process to registry:

• Start upload
• For every layer, check and push layer
• Finally, push image manifest

It is very easy to create a client that gets everything uploaded and ready and delays the last step 

Ways to mitigate it

Avoid tags like latest or similar

 

Don't mutate tags to avoid confusion

 

Use digest instead of tag: myimage@sha256:54c459100e

docker images --digests
kubectl inspect pod | grep sha256
...

Kubernetes resolves digest after scheduling the pod

 

TOCTOU issue: Mutating admission controller

Mutant vs Mutant

https://github.com/sysdiglabs/opa-image-scanner

Image registry pains

Docker registry internals 

Every layer is a content addressable blob

 

Addressed by the sha256 digest of the contents

 

Manifest of the image is another blob JSON doc:

• Contains digest of image config blob
• Digests of every image layer

 

 

https://hub.docker.com/_/registry
https://github.com/docker/distribution

 

Manifest example

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1511,
      "digest": "sha256:e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2802957,
         "digest": "sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9"
      }
   ]
}

Tag and digest storage

Base of registry at $DATA/registry/docker/registry/v2

$ docker push myregistry/library/test:mutant creates
# + repositories/library/test/_manifests/tags/mutant/current/link
# + repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link

# + repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link
# + blobs/sha256/b3/b3787bd1.../data
$ cat repositories/library/test/_manifests/tags/mutant/current/link                       sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat blobs/sha256/b3/b3787bd1.../data
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
...

Tag and digest storage

Now push a mutated version of tets:mutant

$ docker push myregistry/library/test:mutan
# ! repositories/library/test/_manifests/tags/mutant/current/link MODIFIED
$ cat repositories/library/test/_manifests/tags/mutant/current/link                       sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0

# = repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link
$ cat repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link
sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0

# = repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/revisions/sha256/5ae12116.../link
# = blobs/sha256/b3/b3787bd1.../data
# + blobs/sha256/5a/5ae12116.../data

Tag and digest storage

So, for each tag mutation:

• blobs/sha256/<digestPrefix>/<digestPrefix><restOfDigest>/data
  is the blob for manifest file
   
• repositories/library/test/_manifests/tags/mutant/current/link
  links to the current manifest for that tag
   
• repositories/<org>/<repo>/_manifests/tags/<name>/index/sha256/<digest>.../link
  and
  repositories/<org>/<repo>/_manifests/revisions/sha256/<digest>/link
  are also links to the corresponding manifest blob
   
• Index of current and past manifests is kept (for repo and tag)

Garbage collection issues

Garbage collector (mark & sweep):

• Scans manifests on each repo
• Marks referenced layers
• Finally sweeps unused layers to release space

 

Consequence: as all past manifests are kept in the registry, layers for past mutations of a tag are not released

 

Fix: you need to completely remove the repository (all manifests)

The registry V2 API issue

API image removal is by digest:
DELETE /v2/<name>/manifests/<reference>

 

But the listing of images is by tag:

GET /v2/<name>/tags/list

 

Not easy to find all manifests! Need access to storage

 

https://docs.docker.com/registry/spec/api/

The registry V2 API issue

Unexpected behavior: if tag1 and tag2 both point to the same manifest, then deleting tag1 from Harbor UI also deletes tag2

https://github.com/goharbor/harbor/issues/2663

Thanks! Fixed in Harbor 1.11

Recent discussion on this topic:

https://github.com/opencontainers/distribution-spec/issues/102

Immutable tags

Tag immutability

Check before pushing (CI/CD?). Support in some registries like:

• Harbor since 1.10
• ECR - https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecr-now-supports-immutable-image-tags/
• ...

Mutable or immutable?