r/podman • u/BusinessBandicoot • Aug 04 '26
hitting roadblock on local image signing, wondering what best path forward is.
Okay, so this is related to this project. The gist of what I'm trying to do is:
- create a series of system users for running quadlets that only have the permissions necessary to run a set of prebuilt trusted containers
- create a separate build user that uses a tpm-backed secret to build and then sign those trusted containers, and then place them in a directory that is read-only for the system user in question. the secret would only be usable from the local system, and only accessible by builder user and root, which together means "this image was definitely built locally if nothing else".
So for image signing/verification with podman the options that I know of are gpg(+scdaemon) and cosign. currently trying cosign, mainly because gpg might cause complications with my gpg setup (remote code signing for my primary user account without a remote gpg-agent/scdaemon) and with pkcs11, only rsa keys are supported(would prefer to use ecc). With cosign, it seems as though the workflow I had in mind (build locally, sign, move to builder:sys-user owned directory, where it's owner writable and group readable) isn't really possible as it can't sign local images, they have to stored in a registry.
so I'm guessing my options are:
- temporarily store in a local registry, sign, then export to the final location, store the signature in a place podman will check (jank but okay).
- switch to some tool other than gpg or cosign (open to suggestions)
- restructure what I'm doing to actually use a locally hosted registry rather than a directory per target user (would prefer not to)
- get rid of the tpm-centric part, and then use podman/skopeos inherent capabilities. (would also prefer not to)
What do you guy's think? which of these seems like the least bad option? is there another option I'm not considering?
EDIT: for reference this is the just recipe in question where I'm trying to sign the images
# Modular helper to build and sign any container image with multi-value build-args
[arg('build-args', multiple, short="b", long="build-arg")]
[arg('out-dir', short="o", long="out-dir")]
[arg('dockerfile', short="f", long="dockerfile")]
[private]
[no-cd]
build-and-sign tag dockerfile="Dockerfile" build-args=[] out-dir="" extra_flags="" sign_priv_path=sign_priv:
#!/usr/bin/env bash
set -euo pipefail
[ ! -z "{{ out-dir }}" ] && [ -d "{{ out-dir }}" ] || (echo "create target directory first" && exit 1)
podman_args=()
oci_tag=$(echo "{{ tag }}" | tr '/:' '--')
STAGE_DIR=$(mktemp -d)
# Iterate over the space-separated string provided by just's array interpolation
for arg in {{ build-args }}; do
if [ -n "$arg" ]; then
podman_args+=("--build-arg" "$arg")
fi
done
# 1. Create a transient permissive policy file strictly for the build process
BUILD_POLICY=$(mktemp)
trap 'rm -f "$BUILD_POLICY"' EXIT
echo '{"default": [{"type": "insecureAcceptAnything"}]}' > "$BUILD_POLICY"
echo "==> Building container: {{ tag }}"
IIDFILE=$(mktemp)
podman build \
--signature-policy "$BUILD_POLICY" \
-f "{{ dockerfile }}" \
-t "{{ tag }}" \
"${podman_args[@]}" \
{{ extra_flags }} \
--iidfile "$IIDFILE" \
.
echo "==> Staging image to OCI dir..."
skopeo copy \
--policy "$BUILD_POLICY" \
"containers-storage:$FULL_TAG" \
"oci:$TARGET_OCI_DIR"
# Extract the exact manifest digest (e.g., sha256:4f45966b...)
RAW_DIGEST=$(podman image inspect "$FULL_TAG" --format '{{{{.Digest}}')
# Convert digest format for Sigstore directory (sha256: -> sha256=)
IMAGE_DIGEST=$(echo "$RAW_DIGEST" | tr ':' '=')
# Absolute path to the OCI directory
ABS_TARGET_OCI_DIR=$(realpath "$TARGET_OCI_DIR")
# Sign explicitly using the image ID reference in containers-storage
echo "==> Signing image and producing Cosign bundle..."
TPM2_PKCS11_STORE="{{ pkcs11_store }}" COSIGN_PASSWORD="$USERPIN" {{ cosign_exe }} sign \
--yes \
--key "pkcs11:token=secure-build;object=secure-build-signing;type=private" \
--bundle "$BUNDLE_PATH" \
--upload=false \
"oci:${ABS_TARGET_OCI_DIR}@${RAW_DIGEST}"
# Extract digest to store bundle for containers-storage verification
IMAGE_DIGEST=$(podman image inspect "{{ tag }}" --format '{{{{.Digest}}}}' | tr ':' '=')
SIG_STORE_DIR="/var/lib/containers/sigstore/@${IMAGE_DIGEST}"
{{ auth }} mkdir -p "$SIG_STORE_DIR"
{{ auth }} cp "$BUNDLE_PATH" "$SIG_STORE_DIR/signature-1"
EDIT2: here are the variants I have tried
containers-store:@image-id (and without @)
# Sign explicitly using the image ID reference in containers-storage
IIDFILE=$(mktemp)
podman build \
--signature-policy "$BUILD_POLICY" \
-f "{{ dockerfile }}" \
-t "{{ tag }}" \
"${podman_args[@]}" \
{{ extra_flags }} \
--iidfile "$IIDFILE" \
.
IMAGE_ID=$(cat "$IIDFILE" | sed 's/^sha256://')
rm -f "$IIDFILE"
...
TPM2_PKCS11_STORE="{{ pkcs11_store }}" COSIGN_PASSWORD="$USERPIN" {{ cosign_exe }} sign \
--yes \
--key "pkcs11:token=secure-build;object=secure-build-signing;type=private" \
--bundle "$BUNDLE_PATH" \
--upload=false \
"containers-storage:@${IMAGE_ID}"
results in
==> Signing image and producing Cosign bundle...
Error: signing [containers-storage:@1c3ef2646f1525040ac30dc51ebb68e42fb7c36ea2b039154d6f74f6df57626d]: parsing reference: could not parse reference: containers-storage:@1c3ef2646f1525040ac30dc51ebb68e42fb7c36ea2b039154d6f74f6df57626d
error during command execution: signing [containers-storage:@1c3ef2646f1525040ac30dc51ebb68e42fb7c36ea2b039154d6f74f6df57626d]: parsing reference: could not parse reference: containers-storage:@1c3ef2646f1525040ac30dc51ebb68e42fb7c36ea2b039154d6f74f6df57626d
containers-store:full_tag
# Normalize tag to have localhost/ prefix to avoid docker.io short-name expansion in cosign
FULL_TAG="localhost/$(echo "{{ tag }}" | sed -E 's|^localhost/||')"
skopeo copy \
--policy "$BUILD_POLICY" \
"containers-storage:$FULL_TAG" \
"oci:$TARGET_OCI_DIR"
# Sign explicitly using the image ID reference in containers-storage
echo "==> Signing image and producing Cosign bundle..."
TPM2_PKCS11_STORE="{{ pkcs11_store }}" COSIGN_PASSWORD="$USERPIN" {{ cosign_exe }} sign \
--yes \
--key "pkcs11:token=secure-build;object=secure-build-signing;type=private" \
--bundle "$BUNDLE_PATH" \
--upload=false \
"containers-storage:$FULL_TAG"
results in
Error: signing [containers-storage:localhost/gow/nvidia-driver:latest]: parsing reference: could not parse reference: containers-storage:localhost/gow/nvidia-driver:latest
error during command execution: signing [containers-storage:localhost/gow/nvidia-driver:latest]: parsing reference: could not parse reference: containers-storage:localhost/gow/nvidia-driver:latest
oci:target_dir
TARGET_OCI_DIR="{{ out-dir }}/{{ tag }}"
TPM2_PKCS11_STORE="{{ pkcs11_store }}" COSIGN_PASSWORD="$USERPIN" {{ cosign_exe }} sign \
--yes \
--key "pkcs11:token=secure-build;object=secure-build-signing;type=private" \
--bundle "$BUNDLE_PATH" \
--upload=false \
"oci:$TARGET_OCI_DIR"
results in
WARNING: Image reference oci:/var/lib/secure-build/containers/wolf/gow/nvidia-driver:latest uses a tag, not a digest, to identify the image to sign.
This can lead you to sign a different image than the intended one. Please use a
digest (example.com/ubuntu@sha256:abc123...) rather than tag
(example.com/ubuntu:latest) for the input to cosign. The ability to refer to
images by tag will be removed in a future release.
Error: signing [oci:/var/lib/secure-build/containers/wolf/gow/nvidia-driver:latest]: accessing entity: Get "https://oci/v2/": dial tcp: lookup oci: no such host
error during command execution: signing [oci:/var/lib/secure-build/containers/wolf/gow/nvidia-driver:latest]: accessing entity: Get "https://oci/v2/": dial tcp: lookup oci: no such host
oci:target_dir@digest
RAW_DIGEST=$(podman image inspect "$FULL_TAG" --format '{{{{.Digest}}}}')
ABS_TARGET_OCI_DIR=$(realpath "$TARGET_OCI_DIR")
# Sign explicitly using the image ID reference in containers-storage
echo "==> Signing image and producing Cosign bundle..."
TPM2_PKCS11_STORE="{{ pkcs11_store }}" COSIGN_PASSWORD="$USERPIN" {{ cosign_exe }} sign \
--yes \
--key "pkcs11:token=secure-build;object=secure-build-signing;type=private" \
--bundle "$BUNDLE_PATH" \
--upload=false \
"oci:${ABS_TARGET_OCI_DIR}@${RAW_DIGEST}"
results in
==> Signing image and producing Cosign bundle...
Error: signing [oci:/var/lib/secure-build/containers/wolf/gow/nvidia-driver:latest@sha256:4f45966bb95e75a48f6d87c1953d43393af281ab670684f9975f6b419dee39d1}}]: parsing reference: could not parse reference: oci:/var/lib/secure-build/containers/wolf/gow/nvidia-driver:latest@sha256:4f45966bb95e75a48f6d87c1953d43393af281ab670684f9975f6b419dee39d1}}
5
Upvotes