Docker Image Prune Using --filter on Repository Name And Other Info

[loganknecht]

Preface

Hello there!

I've done some googling and I have not seen any clear documentation or discussion about this aspect of the docker image prune command.

I've read a fair amount of the documentation here:

• https://docs.docker.com/edge/engine/reference/commandline/system_prune/
• https://docs.docker.com/engine/reference/commandline/image_prune/
• Stack Overflow
• etc.

Problem

There appears to be no mechanism for filtering on information specific to the image name and/or tags.

User Story

I have a build process that builds intermediate images for my process. In addition to that I'm also pulling in other images from a docker registry, and this machine may be dirty from the build system using other build processes docker images.

Because of this I would like to be able to filter out images/tags that are irrelevant and/or relevant to my images.

For example, my docker images after the process is run, looks like so:

REPOSITORY                              TAG                  IMAGE ID            CREATED             SIZE
example.com/releng/api-db               latest               f19d60cab35c        6 days ago          281MB
example.com/releng/worker               latest               7fe81bdeb5d2        6 days ago          921MB
example.com/releng/api                  latest               5152cade7311        6 days ago          921MB
example.com/example_product/api         latest-development   de17f92e5504        7 days ago          333MB
rabbitmq                                3-management         046c07b70859        7 days ago          124MB
dinkel/clamavd                          latest               9a42e4755eb9        7 days ago          361MB
example.com/example_product/worker      latest-development   f57aab6d0d61        8 days ago          378MB
example.com/example_product/standalone  latest               8ac468f8a86f        2 weeks ago         2.25GB
example.com/example_product/halifax     latest-development   b7cbf1bda847        3 weeks ago         475MB
example.com/postgres                    9.4                  68c55c797542        5 weeks ago         263MB
ubuntu                                  latest               ccc7a11d65b1        2 months ago        120MB
redis                                   3.0.7                c44fa74ead88        3 months ago        91.6MB
nginx                                   1.12.0               313ec0a602bc        3 months ago        107MB
example.com/example_product/postgres    latest-development   4eb1ac130d39        6 months ago        265MB
python                                  2.7.12               0900c636e10d        10 months ago       676MB

Hypothetical Solution

What would be super convenient is the ability to filter based on the image name, the repo that houses the images, and/or the ability to filter by tags as well within that context.

Let's pretend I want to remove all images retrieved from the example.com repo.

Remove images based on repo fuzzy matching

docker image prune --force --filter "repository=example.com*"

Or maybe I want it based on image name

Remove images based on image name fuzzy matching

docker image prune --force --filter "repository=*example_product*"

Questions

Am I misunderstanding the --filter flag?

I realize grep'ing is a solution, but this seems like the tool itself would benefit more from creating this feature rather than relying on work arounds from within the environment's tooling?

I look forward to your feedback 😄

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:45:38 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 8
 Running: 0
 Paused: 0
 Stopped: 8
Images: 48
Server Version: 17.09.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 140
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: TERQ:LFYT:CWRT:4ZPB:37GA:BEWW:265O:6Q35:6DCW:6ISX:OMI2:2EQ7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 31
 System Time: 2017-10-18T00:08:35.693503549Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
OSX

[jadametz]

I was just thinking about the need for this functionality the other day. According to the docs it looks like some basic functionality exists around time filtering. I haven't looked at the code (yet), and would also like to hear the maintainers opinion, but if this isn't too difficult I'd even like to take a shot at the implementation! :)

[thaJeztah]

The docker system prune, and docker container/image/volume/network prune commands were added to have a generic "garbage collect" functionality. I think the general consensus is to not overload these functions, and only provide the basic functionality that's needed to collect unused resources, without being too specific.

The --filter option was added to set a general "policy" for what needs to be purged;

• --filter until=.. allows you to specify when resources should be considered "stale" (and prevents race conditions, for example, images that were just pulled, but no container was started yet).
• --filter label=xyz allows you to categorise resources, and either prevent them from being removed, even though they are considered "stale" (e.g. by setting a label do-not-remove)

If you want to perform more fine-grained deletes, you can combine existing commands, for example:

docker image rm $(docker image ls --filter reference=docker --quiet)

or

docker image ls --filter reference=docker --quiet | xargs docker image rm

So, while I cannot entirely exclude the possibility that new filters will be added to the prune commands, we'd like to be conservative adding new options.

[loganknecht]

@jadametz Did we just become best friends?!

Hey @thaJeztah!

Thank you for taking the time to respond to this feature request.

I would like to request a reconsideration for this feature. I super appreciate the examples you provided. My only complaint is that they're not particularly intuitive for those unfamiliar to docker.

In addition to that this isn't a feature that has not been sought out before.

If you look at google searches (google trend wasn't working for me):

Search	Results
docker prune images by name	124,000
docker prune images by time	117,00
docker remove images by name	1,850,000
docker remove images by time	247,000
docker prune filter by name	28,000
docker prune filter by time	32,500

This is not an amazing way to correlate it, but in each of these searches it is at least demonstrating that the feature for being able to remove by name is just as popular, if not more popular, than removing by time.

I realize that being conservative with expanding a feature is important, as outside sources might not understand the product directions, nuances, or technical limitations.

However, my interaction with docker users online and offline have demonstrated that there is a demand for the ability to have a more fine-tuned ability to remove images/containers with respect to the repo, name, etc, and that being only able to filter on time is not quite sufficient enough to solve that grievance.

Please reconsider the prioritization of this, as it directly affects my organization's efficacy in using your product to manage our solution. This would immediately reduce the footprint of my organization's codebase in managing docker images.

EDIT: Accidentally closed the ticket. Sorry!

[thaJeztah]

Well, the image prune functionality was to address:

Search	Result
docker remove all images	10,200,000

Joking aside, I appreciate your input, I'll definitely bring this up to see what other's opinions are.

My only complaint is that they're not particularly intuitive for those unfamiliar to docker.

The reason behind it comes from the Unix philosophy ("Do One Thing and Do It Well.") - combine commands instead of re-implement functionality that's already offered otherwise. However, I understand not everyone is equally fluent at doing that (and with the introduction of Docker on Windows, the landscape definitely changed).

[VimCommando]

You know, I landed on this feature request because I was exactly trying to figure out how to remove "old" images out of my automated build environment without removing my build dependencies. This means I can't just remove by age, because the nodejs image might not change for weeks, while my app builds can be worthless in literally minutes.

docker image rm $(docker image ls --filter reference=docker --quiet)

That little gem is exactly what I needed. I dropped my repository name in the reference variable (not the most self-explanatory.) Since I tag both the build number and latest the docker image rm command fails on the images I want to keep. I really don't like using daemon errors as a protection mechanism, but its effective.

I do understand not wanting to over complicate commands, but I feel like having documentation on how to accomplish a desired result (in my case I was searching removing docker images by tag) with existing tools would be a huge help. None of the --filter examples I saw in the official docs even mention --filter reference= as an option; so this is the first I've heard of it.

[thaJeztah]

The reference filter looks to be documented here; https://docs.docker.com/engine/reference/commandline/images/#filter-images-by-reference, but not on the new docker image ls page; https://docs.docker.com/engine/reference/commandline/image_ls/#usage. This is likely because they are aliases for each other, and the docker image ls documentation is generated. 🤔

@VimCommando can you open a separate issue for that?

[VimCommando]

You're right, reference is documented. It is something I probably overlooked because it didn't sound anything like "name" or "tag"; and "label" is something entirely different.

An FYI for anyone else who may land on this discussion, if you want to get a list of all images with a specific tag, use this:

docker images --filter reference="*:latest"

or even shorter

docker images *:latest

The documentation kinda implies "exact match only" behavior:

The [REPOSITORY[:TAG]] value must be an “exact match”. This means that, for example, docker images jav does not match the image java.

...but wildcards are supported. So the above commands return what you'd expect -f tag=latest to do. This will clear up my errors from above 👍

@thaJeztah For the filing the separate issue, are you referring to how all the links for the docker image sub-commands all link back to the parent command's content?

[jberkus]

@thaJeztah

--filter label=xyz allows you to categorise resources, and either prevent them from being removed, even though they are considered "stale" (e.g. by setting a label do-not-remove)

Last I checked, there is no way to change labels on a running container. So if you want to flag a container as do-not-remove after it's been started, you're out of luck. No?

[thaJeztah]

@jberkus correct, there's no option (yet) to update labels on a container (see moby/moby#21721 for a proposal) - more input is welcome there; it's been there for a while, so possibly that one should be revisited / discussed again.

[jberkus]

Bit of a catch-22, then, no?

1: You can't protect containers from removal by name, use labels.

2: You can't change labels on instantiated containers.

Therefore: you can't protect containers from pruning unless you knew you wanted to protect them before they were created.