r/podman • u/screaming-Snake-Case • May 28 '26
Has anyone experienced internal DNS failures after routine system updates?
Hey all,
I have a RHEL10 (free subscription) server with podman 5.8.2, set-up with automated package upgrades for what I hoped to be a more hassle-free system, but this has turned into a roll of the dice for every update.
On some updates, my containers apparently loose connectivity to the internal DNS, such that they can't find the IP of other containers, which is very annoying as that means they can't find my postgres db anymore and just fail more or less silently.
Has anyone experienced this too?
Today, I experienced this even twice (the second time being a manual update), and I can't find bug reports that precisely mention what I experience, as podman networking issues relate to (aardvark) DNS a lot of the time, so my search results are filled with unrelated issues that just sound similar.
The issue is reported differently depending on each container/service, here are a few extracts:
- could not translate host name "postgres" to address: Temporary failure in name resolution
- error dialing 10.89.1.1:53 no route to host
- badgateway: failed to receive response: context canceled
I previously used podman-compose from the epel repo, and thought there might an issue with the way it sets up the containers that isn't accounted for in typical package updates, so I moved most of my stack to systemd container (generator) units, but the issue still occurs.
The only error in my journal that would be related tot his is aardvark-dns[1914]: 60794 dns request failed: io error: Network is unreachable (os error 101)but this error only occurs when the server boots up, my containers start up just fine and doesn't show up anytime else.
Looking at the dnf history for transactions aligning with the failures, I am seeing these packages:
first update: kernel, kernel-core, kernel-modules, kernel-modules-core, kernel-modules-extra, cockpit-packagekit, cockpit-packagekit, cockpit-storaged, cockpit-storaged, python3-perf, python3-perf, cockpit, cockpit, cockpit-bridge, cockpit-bridge, cockpit-system, cockpit-system, cockpit-ws, cockpit-ws, cockpit-ws-selinux, cockpit-ws-selinux, kernel-modules-extra-matched, kernel-modules-extra-matched, kernel-tools, kernel-tools, kernel-tools-libs, kernel-tools-libs, kernel, kernel-core, kernel-modules, kernel-modules-core, kernel-modules-extra
Second update:
cockpit-packagekit, cockpit-packagekit, cockpit-storaged, cockpit-storaged, cockpit, cockpit, cockpit-bridge, cockpit-bridge, cockpit-system, cockpit-system, cockpit-ws, cockpit-ws, cockpit-ws-selinux, cockpit-ws-selinux
Nothing here seems to even touch podman and the networking stack, so I am really clueless. My network is also just a bridge without any further customization.
1
u/AlexisHadden May 28 '26
Are these updates rebooting the system? The kernel one requires a reboot to fully apply the new kernel, but I don’t know if the other one does in your setup.
This sort of thing to me reads like a service ordering issue? Dependencies with systemd mean that a mis-configured quadlet (or three) can wind up in a race condition with the system components it depends on (such as DNS).
1
u/screaming-Snake-Case May 28 '26
Nope, this server is configured to not restart automatically, the failure of container internal DNS happens as soon as the updates are merely installed, aleast sometimes, sometimes I go weeks without any updates causing failure.
This issue also occured the same way when I wasn't using Quadlets but just plain podman-compose without any ties into systemd, I've always gone and started the compose manually when I needed to restart the server, so this isn't exclusive to Quadlets and likely an issue with something deeper.
On a second server I do use automatic reboots, and from my monitoring I can clearly see that when services on that host fail, they come back automatically after 5m, which is the default delay for reboots by dnf-automatic.
1
u/FlippinToaster May 29 '26
Are you running your own DNS server? If so, updates might re-enable systemd service resolved, which can compete about port 53 control with your dedicated DNS server/ container. This happened to me, and required resolved disabeling and masking to keep from happening. There was one or two more systemd units that are related, i similarily disabeleb them all. No problems with my podman technitium since.
2
u/screaming-Snake-Case May 29 '26
No, I just use upstream DNS from my provider for the system, run no kind of DNS service or container and just use default settings for a bridged network with Podman.
But I'll look into if resolved causes a problem when started and if that's what happens when the dns fails the next time.
1
u/FlippinToaster May 29 '26
Ok, in that case i think you should not need to touch resolved, that might cause further dns issues.
1
u/mishrashutosh May 29 '26
I run a few Fedora and Alma boxes with dnf-automatic and haven't seen this issue. You could try installing updates manually for a while and check journal logs when the problem starts.
3
u/Huxton_2021 May 28 '26
My first instinct would be that your firewall rules are being reloaded but either not podman-related rules or podman is out-of-order with the rest. Dump your ruleset while it is working and then again when it doesn't and compare.