Attack
of the mutant tags
Álvaro Iradier - @airadier
Attack of the mutant tags!
- Basic concepts
- What is a mutant tag?
- Deployment issues
- Protect against the mutations
- Image registry pains
- Immutable tags
- Questions
Basic concepts
Registry, image, container
Image registry (i.e. docker.io)
myimage:tag1
myimage:tag2
container-1
container-2
container-3
container-4
container-5
Image tag, manifest, layer...
Image layers
e74c2290...
71bb4d21...
9080b345...
e74c2290...
71bb4d21...
9080b345...
myimage:tag1
1ab50232...
b604fa38...
612bc919...
23456789...
fe5b187a...
myimage repository
tag1
tag2
1ab50232...
Manifest
What is a mutant tag?
What is a mutant tag
myimage:tag1
What is a mutant tag
myimage:tag1
myimage repository
What is a mutant tag
myimage:tag1
1ab50232...
myimage repository
tag1
What is a mutant tag
tag1
1ab50232...
17b345ca...
myimage repository
myimage:tag1
myimage:tag1
What is a mutant tag
tag1
1ab50232...
17b345ca...
612bc919...
myimage repository
myimage:tag1
myimage:tag1
myimage:tag1
What is a mutant tag
tag1
1ab50232...
17b345ca...
612bc919...
myimage repository
myimage@sha256:1ab50232...
myimage@sha256:17b345ca
myimage:tag1
myimage:othertag
myimage@sha256:612c919...
othertag
Mutant tag: use cases
-
latest (default if not specified) and variants:
- alpine
- slim
- ...
- Per-environment:
- dev
- sta
- qa
- prod
- ...
- Versioning:
- myimage:1.0 -> myimage:1.0.1, myimage:1.0.2,...
Deployment issues
docker run
docker run myimage:latest
myimage@sha256:1ab50232...
docker run
docker run myimage:latest
myimage@sha256:1ab50232...
couple of hours later in another developer laptop...
docker run
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:17b345ca
couple of hours later in another developer laptop...
docker run
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:17b345ca
couple of hours later in another developer laptop...
tomorrow, after a cache cleanup, or in other server...
docker run
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:17b345ca
docker run myimage:latest
myimage@sha256:612c919...
couple of hours later in another developer laptop...
tomorrow, after a cache cleanup, or in other server...
Kubernetes pod schedule
deploy with spec image: "myimage:latest"
node1
node2 - cordon
Kubernetes pod schedule
myimage@sha256:1ab50232...
deploy with spec image: "myimage:latest"
node1
node2 - cordon
Kubernetes pod schedule
myimage@sha256:1ab50232...
deploy with spec image: "myimage:latest"
node1
node2 - cordon
push other myimage:latest
Kubernetes pod schedule
myimage@sha256:1ab50232...
deploy with spec image: "myimage:latest"
node1
node2
Kubernetes pod schedule
myimage@sha256:1ab50232...
myimage@sha256:17b345ca
deploy with spec image: "myimage:latest"
node1
node2
Kubernetes pod schedule
myimage@sha256:1ab50232...
myimage@sha256:17b345ca
deploy with spec image: "myimage:latest"
node1
node2
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
Pod with image myimage:latest
1
2
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
2
3
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
2
3
4
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
2
3
4
5
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
2
3
4
5
Admission controller +
image scanner - TOCTOU
push other myimage:latest
6
Pod with image myimage:latest
1
2
3
4
5
7
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
2
3
4
5
7
8
Admission controller +
image scanner - TOCTOU
Pod with image myimage:latest
1
2
3
4
5
7
8
Admission controller +
image scanner - TOCTOU
Protect against the mutations
Is this easy to reproduce?
Pushing image process to registry:
- Start upload
- For every layer, check and push layer
- Finally, push image manifest
It is very easy to create a client that gets everything uploaded and ready and delays the last step
Ways to mitigate it
Avoid tags like latest or similar
Don't mutate tags to avoid confusion
Use digest instead of tag: myimage@sha256:54c459100e
docker images --digests kubectl inspect pod | grep sha256 ...
Kubernetes resolves digest after scheduling the pod
TOCTOU issue: Mutating admission controller
Mutant vs Mutant
Image registry pains
Docker registry internals
Every layer is a content addressable blob
Addressed by the sha256 digest of the contents
Manifest of the image is another blob JSON doc:
- Contains digest of image config blob
- Digests of every image layer
https://hub.docker.com/_/registry
https://github.com/docker/distribution
Manifest example
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1511,
"digest": "sha256:e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2802957,
"digest": "sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9"
}
]
}
Tag and digest storage
Base of registry at $DATA/registry/docker/registry/v2
$ docker push myregistry/library/test:mutant creates # + repositories/library/test/_manifests/tags/mutant/current/link # + repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link
# + blobs/sha256/b3/b3787bd1.../data
$ cat repositories/library/test/_manifests/tags/mutant/current/link sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat blobs/sha256/b3/b3787bd1.../data
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
...
Tag and digest storage
Now push a mutated version of tets:mutant
$ docker push myregistry/library/test:mutan # ! repositories/library/test/_manifests/tags/mutant/current/link MODIFIED $ cat repositories/library/test/_manifests/tags/mutant/current/link sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0
# = repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link # + repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link $ cat repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0 # = repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link # + repositories/library/test/_manifests/revisions/sha256/5ae12116.../link # = blobs/sha256/b3/b3787bd1.../data # + blobs/sha256/5a/5ae12116.../data
Tag and digest storage
So, for each tag mutation:
-
blobs/sha256/<digestPrefix>/<digestPrefix><restOfDigest>/data
is the blob for manifest file
-
repositories/library/test/_manifests/tags/mutant/current/link
links to the current manifest for that tag
-
repositories/<org>/<repo>/_manifests/tags/<name>/index/sha256/<digest>.../link
and
repositories/<org>/<repo>/_manifests/revisions/sha256/<digest>/link
are also links to the corresponding manifest blob
- Index of current and past manifests is kept (for repo and tag)
Garbage collection issues
Garbage collector (mark & sweep):
- Scans manifests on each repo
- Marks referenced layers
- Finally sweeps unused layers to release space
Consequence: as all past manifests are kept in the registry, layers for past mutations of a tag are not released
Fix: you need to completely remove the repository (all manifests)
The registry V2 API issue
API image removal is by digest:
DELETE /v2/<name>/manifests/<reference>
But the listing of images is by tag:
GET /v2/<name>/tags/list
Not easy to find all manifests! Need access to storage
The registry V2 API issue
Unexpected behavior: if tag1 and tag2 both point to the same manifest, then deleting tag1 from Harbor UI also deletes tag2
https://github.com/goharbor/harbor/issues/2663
Thanks! Fixed in Harbor 1.11
Recent discussion on this topic:
https://github.com/opencontainers/distribution-spec/issues/102
Immutable tags
Tag immutability
Check before pushing (CI/CD?). Support in some registries like:
- Harbor since 1.10
- ECR - https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecr-now-supports-immutable-image-tags/
- ...
Mutable or immutable?
Thanks!
?
Check my blog post at https://sysdig.com/blog/toctou-tag-mutability
Attack of the mutant tags
By Álvaro José Iradier
Attack of the mutant tags
- 965