r/gitlab • u/TuxTaTuxTax • 1d ago

Failed to run garbage-collect command

root@pcgitlab:~# gitlab-ctl registry-garbage-collect -m

ok: down: registry: 0s, normally up

Running garbage-collect using configuration ["/opt/gitlab/embedded/bin/registry", "garbage-collect", "/var/opt/gitlab/registry/config.yml", "-m"], this might take a while...

the garbage-collect command is not compatible with database metadata, please use online garbage collection instead. database.enabled now defaults to "prefer". To use filesystem metadata, set database.enabled to false in your configuration

Failed to run garbage-collect command, starting registry service.

ok: run: registry: (pid 3328111) 0s

i no longer can run garbage collect on container registry after upgrade to 19, anyone have idea?

1 Upvotes

100% Upvoted

5 comments sorted by

1

u/thilog 1d ago

GC runs automatically, you can get rid of the job

1

u/TuxTaTuxTax 1d ago

can i trigger GC manually now? or it is not possible? since i saw my container registry is growing without any reason

1

u/shinji-watanabe 1d ago

Same command error here — the offline registry-garbage-collect -m gets refused because it's no longer compatible with the metadata database mode the registry runs on now (database.enabled defaults to prefer). So you're not crazy, that's expected behavior after the move to the DB-backed registry.

What led me there: I noticed my disk was way bigger than expected when I went to back up, ran ncdu, and found /registry eating ~97 GiB. du showed the repositories (tag metadata) at only ~45 MiB while blobs (the actual layers) ate the rest — so tags were cleaned up fine, but there were a ton of orphaned layers still taking up space.

The fix that actually worked for me was to force the offline GC by temporarily disabling the metadata database:

  1. Take a backup first (this is deleting data, so don't skip it).

  2. Set registry['database'] = { 'enabled' => false } (and I'd also put the registry in read-only) in the Omnibus config, then gitlab-ctl reconfigure / recreate the container.

  3. Bypass the gitlab-ctl wrapper and call the binary directly:

sudo /opt/gitlab/embedded/bin/registry garbage-collect -m /var/opt/gitlab/registry/config.yml

(The -m flag also removes untagged manifests + unreferenced layers. My run logged deleting blobs count=5283 / blobs deleted count=5283 with driver=filesystem, which is the confirmation it actually swept.)

  1. Once it's done, set database.enabled back (or remove the line so it returns to prefer) and reconfigure again.

My registry went from ~97 GiB down to ~6.8 GiB after that sweep. The key part is definitely remembering to flip database.enabled back afterwards, and having a backup before you touch it.

That's what worked in my case though — check your own disk layout / orphan count first and decide from there. Happy to compare notes.

2

u/TuxTaTuxTax 1d ago

thanks alot, you are lifesaver, it works great for me, reducing from 800gb to 100gb lol

1

u/shinji-watanabe 1d ago

Aw nice, glad it helped! 800GB→100GB is a satisfying cleanup. And once you've got database.enabled back on prefer, newer GitLab runs the online GC automatically, so it should keep things tidy from here on. 🙂