Álvaro Iradier - @airadier
Image registry (i.e. docker.io)
myimage:tag1
myimage:tag2
container-1
container-2
container-3
container-4
container-5
Image layers
e74c2290...
71bb4d21...
9080b345...
e74c2290...
71bb4d21...
9080b345...
myimage:tag1
1ab50232...
b604fa38...
612bc919...
23456789...
fe5b187a...
myimage repository
tag1
tag2
1ab50232...
Manifest
myimage:tag1
myimage:tag1
myimage repository
myimage:tag1
1ab50232...
myimage repository
tag1
tag1
1ab50232...
17b345ca...
myimage repository
myimage:tag1
myimage:tag1
tag1
1ab50232...
17b345ca...
612bc919...
myimage repository
myimage:tag1
myimage:tag1
myimage:tag1
tag1
1ab50232...
17b345ca...
612bc919...
myimage repository
myimage@sha256:1ab50232...
myimage@sha256:17b345ca
myimage:tag1
myimage:othertag
myimage@sha256:612c919...
othertag
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:1ab50232...
couple of hours later in another developer laptop...
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:17b345ca
couple of hours later in another developer laptop...
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:17b345ca
couple of hours later in another developer laptop...
tomorrow, after a cache cleanup, or in other server...
docker run myimage:latest
myimage@sha256:1ab50232...
docker run myimage:latest
myimage@sha256:17b345ca
docker run myimage:latest
myimage@sha256:612c919...
couple of hours later in another developer laptop...
tomorrow, after a cache cleanup, or in other server...
deploy with spec image: "myimage:latest"
node1
node2 - cordon
myimage@sha256:1ab50232...
deploy with spec image: "myimage:latest"
node1
node2 - cordon
myimage@sha256:1ab50232...
deploy with spec image: "myimage:latest"
node1
node2 - cordon
push other myimage:latest
myimage@sha256:1ab50232...
deploy with spec image: "myimage:latest"
node1
node2
myimage@sha256:1ab50232...
myimage@sha256:17b345ca
deploy with spec image: "myimage:latest"
node1
node2
myimage@sha256:1ab50232...
myimage@sha256:17b345ca
deploy with spec image: "myimage:latest"
node1
node2
Pod with image myimage:latest
1
Pod with image myimage:latest
1
2
Pod with image myimage:latest
1
2
3
Pod with image myimage:latest
1
2
3
4
Pod with image myimage:latest
1
2
3
4
5
Pod with image myimage:latest
1
2
3
4
5
push other myimage:latest
6
Pod with image myimage:latest
1
2
3
4
5
7
Pod with image myimage:latest
1
2
3
4
5
7
8
Pod with image myimage:latest
1
2
3
4
5
7
8
Pushing image process to registry:
It is very easy to create a client that gets everything uploaded and ready and delays the last step
Avoid tags like latest or similar
Don't mutate tags to avoid confusion
Use digest instead of tag: myimage@sha256:54c459100e
docker images --digests kubectl inspect pod | grep sha256 ...
Kubernetes resolves digest after scheduling the pod
TOCTOU issue: Mutating admission controller
Mutant vs Mutant
Every layer is a content addressable blob
Addressed by the sha256 digest of the contents
Manifest of the image is another blob JSON doc:
https://hub.docker.com/_/registry
https://github.com/docker/distribution
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1511,
"digest": "sha256:e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2802957,
"digest": "sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9"
}
]
}
Base of registry at $DATA/registry/docker/registry/v2
$ docker push myregistry/library/test:mutant creates # + repositories/library/test/_manifests/tags/mutant/current/link # + repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
# + repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link
# + blobs/sha256/b3/b3787bd1.../data
$ cat repositories/library/test/_manifests/tags/mutant/current/link sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link
sha256:b3787bd182d60ee3bd8d0bb53064e7eaa1073b817c31769dba3822895f9254d6
$ cat blobs/sha256/b3/b3787bd1.../data
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
...
Now push a mutated version of tets:mutant
$ docker push myregistry/library/test:mutan # ! repositories/library/test/_manifests/tags/mutant/current/link MODIFIED $ cat repositories/library/test/_manifests/tags/mutant/current/link sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0
# = repositories/library/test/_manifests/tags/mutant/index/sha256/b3787bd1.../link # + repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link $ cat repositories/library/test/_manifests/tags/mutant/index/sha256/5ae12116.../link sha256:5ae1211607a565d4fe5a8dba725dcc56265d6eadb4c013bf6f6407ab2d83ead0 # = repositories/library/test/_manifests/revisions/sha256/b3787bd1.../link # + repositories/library/test/_manifests/revisions/sha256/5ae12116.../link # = blobs/sha256/b3/b3787bd1.../data # + blobs/sha256/5a/5ae12116.../data
So, for each tag mutation:
Garbage collector (mark & sweep):
Consequence: as all past manifests are kept in the registry, layers for past mutations of a tag are not released
Fix: you need to completely remove the repository (all manifests)
API image removal is by digest:
DELETE /v2/<name>/manifests/<reference>
But the listing of images is by tag:
GET /v2/<name>/tags/list
Not easy to find all manifests! Need access to storage
Unexpected behavior: if tag1 and tag2 both point to the same manifest, then deleting tag1 from Harbor UI also deletes tag2
https://github.com/goharbor/harbor/issues/2663
Thanks! Fixed in Harbor 1.11
Recent discussion on this topic:
https://github.com/opencontainers/distribution-spec/issues/102
Check before pushing (CI/CD?). Support in some registries like:
Thanks!
?
Check my blog post at https://sysdig.com/blog/toctou-tag-mutability