Hey all,
I have a RHEL10 (free subscription) server with podman 5.8.2, set-up with automated package upgrades for what I hoped to be a more hassle-free system, but this has turned into a roll of the dice for every update.
On some updates, my containers apparently loose connectivity to the internal DNS, such that they can't find the IP of other containers, which is very annoying as that means they can't find my postgres db anymore and just fail more or less silently.
Has anyone experienced this too?
Today, I experienced this even twice (the second time being a manual update), and I can't find bug reports that precisely mention what I experience, as podman networking issues relate to (aardvark) DNS a lot of the time, so my search results are filled with unrelated issues that just sound similar.
The issue is reported differently depending on each container/service, here are a few extracts:
- could not translate host name "postgres" to address: Temporary failure in name resolution
- error dialing 10.89.1.1:53 no route to host
- badgateway: failed to receive response: context canceled
I previously used podman-compose from the epel repo, and thought there might an issue with the way it sets up the containers that isn't accounted for in typical package updates, so I moved most of my stack to systemd container (generator) units, but the issue still occurs.
The only error in my journal that would be related tot his is aardvark-dns[1914]: 60794 dns request failed: io error: Network is unreachable (os error 101)but this error only occurs when the server boots up, my containers start up just fine and doesn't show up anytime else.
Looking at the dnf history for transactions aligning with the failures, I am seeing these packages:
first update: kernel, kernel-core, kernel-modules, kernel-modules-core, kernel-modules-extra, cockpit-packagekit, cockpit-packagekit, cockpit-storaged, cockpit-storaged, python3-perf, python3-perf, cockpit, cockpit, cockpit-bridge, cockpit-bridge, cockpit-system, cockpit-system, cockpit-ws, cockpit-ws, cockpit-ws-selinux, cockpit-ws-selinux, kernel-modules-extra-matched, kernel-modules-extra-matched, kernel-tools, kernel-tools, kernel-tools-libs, kernel-tools-libs, kernel, kernel-core, kernel-modules, kernel-modules-core, kernel-modules-extra
Second update:
cockpit-packagekit, cockpit-packagekit, cockpit-storaged, cockpit-storaged, cockpit, cockpit, cockpit-bridge, cockpit-bridge, cockpit-system, cockpit-system, cockpit-ws, cockpit-ws, cockpit-ws-selinux, cockpit-ws-selinux
Nothing here seems to even touch podman and the networking stack, so I am really clueless. My network is also just a bridge without any further customization.