Docker on ¬ just serendipity 🍀

Kubernetes: one-off env exec

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 23 Mar 2026 11:56:33 +0100

♠ Problem statement: run an one-off script and/or binary within a kubernetes pod. It needs a certain environment variable (VAULT_TOKEN) to be set.

Wrong:

ubuntu@my-deploy-server:~ $  kubectl exec -n vault vault-1 -- VAULT_TOKEN=hvs.mysupersecrettoken vault read auth/kubernetes/role/vault-init
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "7455bd5880fe41c177a47702e3dfacbe4240207d53a19ac13edf74b62c626e61": OCI runtime exec failed: exec failed: unable to start container process: exec: "VAULT_TOKEN=hvs.mysupersecrettoken": executable file not found in $PATH

It is trying to look for a file named VAULT_TOKEN=hvs.mysupersecrettoken in the $PATH. Surely that’s not we want here!

Right:

ubuntu@my-deploy-server:~ $  kubectl exec -n vault vault-1 -- env VAULT_TOKEN=hvs.mysupersecrettoken vault read auth/kubernetes/role/vault-init
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
bound_service_account_names                 [vault-init]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [vault]
policies                                    [vault-init]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                168h
token_policies                              [vault-init]
token_ttl                                   0s
token_type                                  default

…just pass it via env(1):

env — run a program in a modified environment

env [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]

Also, if you pay attention, you’ll notice there are two spaces between $ and kubectl. This is intentional. See Why is bash not storing commands that start with spaces? for context. bash won’t record the command in the shell history if it starts with spaces — though that behavior depends on the HISTCONTROL setting. We do not want to store our VAULT_TOKEN in plain text in the shell history (~/.bash_history).

— § —

Reply via email

#dev #docker

Docker: AWS ECR: list recent image tags

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 18 Aug 2025 15:38:21 +0200

♠ Problem statement: Given a docker repository hosted in ECR (AWS), list its most recent image tags.

Assuming you’re already authenticated to AWS in the CLI:

% aws ecr describe-images \
  --repository-name {corp}/{repo} \
  --filter tagStatus=TAGGED \
  --query 'reverse(sort_by(imageDetails, &imagePushedAt))[*].{Tag: imageTags[0], PushedAt: imagePushedAt}' \
  [--output table]

The output looks like this:

-------------------------------------------------------------------------------------------
|                                     DescribeImages                                      |
+-----------------------------------+-----------------------------------------------------+
|             PushedAt              |                         Tag                         |
+-----------------------------------+-----------------------------------------------------+
|  2025-08-14T14:44:49.285000+02:00 |  1.0.2                                              |
|  2025-08-14T14:07:53.526000+02:00 |  app-839403421                                      |
|  2025-08-07T19:38:21.452000+02:00 |  app-1134bfa1d                                      |
|  2025-08-06T14:54:23.314000+02:00 |  app-pr-5033-07f01dd66                              |
|  2025-08-06T14:33:27.171000+02:00 |  app-pr-5033-643536a2b                              |
|  2025-07-23T09:36:20.368000+02:00 |  app-pr-4947-fe5f4c618                              |
|  2025-04-17T03:17:30.937000+02:00 |  1.0.1                                              |
|  2024-12-18T18:50:18.990000+01:00 |  latest                                             |
|  2024-12-05T17:52:09.775000+01:00 |  1.0.0                                              |
+-----------------------------------+-----------------------------------------------------+

If you omit --output table, it emits JSON instead.

This is faster than logging into the AWS Console.

— § —

Reply via email

#dev #docker

Dockerfile: CMD and ENTRYPOINT: shell and exec forms

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 18 Aug 2025 15:09:28 +0200

♠ Today I filed a github issue against the dockerfmt project, which I copy verbatim here given its surprising behavior.

reteps/dockerfmt#33 #

Consider this sample Dockerfile (abridged version of a real-world scenario):

ENTRYPOINT sleep infinity
CMD sleep infinity

Let’s run dockerfmt (v0.3.7) in it:

% dockerfmt Dockerfile
ENTRYPOINT ["sleep", "infinity"]
CMD ["sleep", "infinity"]

At first glance, everything looks OK. The problem is that this transformation is not a no-op.

Looking at the docs:

If CMD is used to provide default arguments for the ENTRYPOINT instruction, both the CMD and ENTRYPOINT instructions should be specified in the exec form.

The difference is:

the original code yields sleep infinity (correct!)
the transformed code yields sleep infinity sleep infinity, which is an error (sleep: no such argument sleep)

One could argue that the original code is odd – the following would be conceptually simpler, and more idiomatic:

ENTRYPOINT ["sleep"]
CMD ["infinity"]

Or even:

ENTRYPOINT ["sleep", "infinity"]

Nonetheless, IMHO dockerfmt should not change the behavior in this case.

Notes #

shell and exec forms have distinct forms:

INSTRUCTION ["executable", "param1", "param2"] (exec form)
INSTRUCTION command param1 param2 (shell form)

The exec form makes it possible to avoid shell string munging, and to invoke commands using a specific command shell, or any other executable. It uses a JSON array syntax, where each element in the array is a command, flag, or argument.

The shell form is more relaxed, and emphasizes ease of use, flexibility, and readability. The shell form automatically uses a command shell, whereas the exec form does not.

dockerfmt prefers the exec form, for whatever reason.

There’s even a code comment:

// this can technically change behavior. https://docs.docker.com/reference/dockerfile/#understand-how-cmd-and-entrypoint-interact

Surprise surprise! The issue I reported is precisely due to a change of behavior.

— § —

Reply via email

#dev #docker

docker: dive into images

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 11 Aug 2025 13:20:08 +0200

♠ Recently bitnami has pulled the plug off their public docker images offering¹.

We are relying on their bitnami/kubectl image. Now what?

We could build our own, or use another upstream image. Looking at the Docker Hub catalog, I was pleased to find rancher/kubectl (backed by SUSE) there.

This is enough for our needs.

It would have been trivial to build our own image otherwise if needed:

FROM alpine:3.22
RUN apk add --no-cache kubectl

How can we ensure these images are equivalent, for practical purposes?

One way is to use dive to introspect each of them:

A tool for exploring each layer in a docker image

% dive bitnami/kubectl:latest
% dive rancher/kubectl:v1.30.14

The rancher image has 50MB in size and contains only the kubectl binary. There’s no shell, not even /bin/sh.

The bitnami image has 326MB and contains a bunch of other stuff.

It turns out we will get some (unexpected) efficiency gains out of this migration!

“““Recession”””. ↩︎

— § —

Reply via email

#dev #docker

Alpine Linux: Dockerfile: install package from edge

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 04 Jun 2025 12:37:47 +0200

♠ Problem statement: Given a Dockerfile based on a fixed release of Alpine Linux (e.g. FROM alpine:3.22), install a package from edge (i.e. a package that did not yet make it to a fixed release).

The approach is similar to package pinning, by providing a custom --repository to apk add:

RUN apk add --no-cache --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing gliderlabs-sigil vals

Replace “testing” with “community” or “main” as needed.

— § —

Reply via email

#alpine-linux #dev #docker

★ skopeo: operate container images and registries

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 17 Dec 2024 17:49:31 -0300

♠ When working with docker and private image registries, a common workflow is to copy images from one private registry in the cloud to another. This can be done with skopeo.

This post includes some common recipes for it.

Usage 1) Default / Root to Staging #

From the default / root account registry to the staging registry:

skopeo sync \
  --src-creds "AWS:$(aws ecr get-login-password --region {region} --profile default)" \
  --dest-creds "AWS:$(aws ecr get-login-password --region {region} --profile staging)" \
  --override-os linux --override-arch amd64 \
  --src docker --dest docker \
  {account_id_root}.dkr.ecr.{region}.amazonaws.com/{org}/{repository}:{tag} \
  {account_id_staging}.dkr.ecr.{region}.amazonaws.com/{org}

Example values¹:

repository: argocd-gitops-tools
tag (version): 1.0.1
region: us-east-1

Usage 2) Public to MFA #

From a public registry to a private registry that uses MFA (multi-factor authentication).

First, it’s necessary to get MFA credentials. Source the script below (china.mfa.sh):

#!/usr/bin/env bash
# shellcheck disable=SC2155

export AWS_PROFILE=china

echo -n "Enter the MFA token code for your AWS China account: " && read -r token

mfa_arn="$(aws iam --profile "$AWS_PROFILE" get-user --output text --query User.Arn | sed 's/:user\//:mfa\//')"
credentials="$(aws --profile "$AWS_PROFILE" sts get-session-token --serial-number "$mfa_arn" --token-code "$token" --duration-seconds 86400)"

echo "Got credentials: $credentials"

export AWS_ACCESS_KEY_ID=$(echo "$credentials" | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo "$credentials" | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo "$credentials" | jq -r '.Credentials.SessionToken')

[[ -n "$AWS_SESSION_TOKEN" ] && echo "Success!"

Important: It is necessary to do source china-mfa.sh. Doing ./china-mfa.sh will not work.

Then we can proceed with the image sync:

skopeo sync \
  --dest-creds "AWS:$(aws ecr get-login-password --region cn-north-1)" \
  --override-os linux --override-arch amd64 \
  --src docker \
  --dest docker \
  quay.io/argoproj/argocd:v2.12.6 \
  {account_id_mfa}.dkr.ecr.cn-north-1.amazonaws.com.cn/quay.io/argoproj

Notes:

A public registry does not need authentication, hence there’s no --src-creds.
--dest-creds does not specify a --profile. Likewise, no AWS_PROFILE env var should be defined.
In this example, China (cn-north-1) is an AWS account with MFA enabled.

Potpourri #

It’s possible to pass --scoped to prefix images at destination using the full source image path as scope.
To sync multiple image architectures, pass -a or --all.

If syncing from/to Azure (ACR), use az acr login. Find the username in the Azure portal. For example:

az acr login -n {container registry name} --expose-token | jq -r '.accessToken' | skopeo login {registry} --password-stdin --username {username}

image = registry (includes the org name) + repository + tag ↩︎

— § —

Reply via email

#aws #bestof #dev #docker #kubernetes

Docker build: ignore cache

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 10 Dec 2024 19:45:42 -0300

♠ To build a docker image completely from scratch, without reusing cache layers on your system:

docker build --no-cache -f Dockerfile . -t {image name}:{image tag} --platform
linux/amd64

Pass an explicit platform in case e.g. you’re building on a Mac M1 (arm64) with the intent of running it on Linux in the cloud (amd64 typically).
The docker image full specification is: image = {registry}/{repository}:{tag}. For example, the registry could be an AWS ECR URL, whereas the repository is something like “bitnami/sealed-secrets”, and the tag typically follows semantic versioning.

— § —

Reply via email

#aws #dev #docker

Docker image size

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 05 Dec 2024 13:05:44 +0100

♠ When trying to optimize docker images in terms of size, the workflow is as follows:

Terminal 1 #

% docker build -f Dockerfile . -t argocd-download-tools

…possibly paired up with entr or fswatch depending on your eagerness for automatic builds.

Terminal 2 #

% docker image inspect argocd-download-tools | jq -r '.[0].Size' | numfmt --to=iec
134M

…possibly prepended with watch.

I wasn’t aware of numfmt:

numfmt — Convert numbers from/to human-readable strings

…instead of displaying plain bytes, it converts numbers to human-readable strings (à la df -h or free -h).

Regarding the units: they don’t matter much, because we are only interested in an approximation, no need to shave bytes off here. SI has 1k = 1000, whereas IEC has 1Ki = 1024.

— § —

Reply via email

#dev #docker

Github Actions: multiple docker images

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 05 Nov 2024 17:05:22 +0100

♠ Problem statement: Given a monorepo on Github with multiple docker images in it, write a github workflow to build and push all of them.

Here is an abridged version of the resulting workflow. The images are listed one by one for fine-grained control purposes, but it would also be possible to glob them with a single command.

name: Global services

permissions:
  id-token: write
  contents: read

# Replace with workflow trigger conditions.
on: {}

jobs:
  global-services:
    name: ${{ matrix.image }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          # keep-sorted start
          - dockerfile: path/to/one/Dockerfile
            image: org/one
          - dockerfile: path/to/two/Dockerfile
            image: org/two
          # keep-sorted end
    steps:
      - name: Check out source code
        uses: actions/checkout@v4

      - name: Get git SHA
        id: git-sha
        run: |
          ./scripts/get-versions.sh
          echo "sha=$(grep '^services_sha' versions.yaml | cut -d' ' -f 2)" >> "$GITHUB_OUTPUT"

      - uses: ./.github/actions/setup-ecr-buildx
        id: setup-ecr-buildx

      - name: Build and push global service images
        uses: docker/build-push-action@6.9.0
        with:
            cache-from: type=gha
            cache-to: type=gha,mode=max
            context: path/to
            file: ${{ matrix.dockerfile }}
            provenance: false
            push: true
            tags: ${{ steps.setup-ecr-buildx.outputs.ecr_registry }}/${{ matrix.image }}:${{ steps.git-sha.outputs.sha }}

A matrix strategy kicks off independent multiple build jobs all at once.

Docs: https://github.com/docker/build-push-action

— § —

Reply via email

#dev #docker

Check if docker image exists in registry

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 25 Oct 2024 14:11:14 +0200

♠ Given a docker registry, image and tag, run the following command:

% docker manifest inspect 9876543210.foo.ecr.us-east-1.amazonaws.com/org/image:123456789

If it does not exist, you’ll see:

no such manifest: 9876543210.foo.ecr.us-east-1.amazonaws.com/org/image:123456789

Ensure you are authenticated and/or connected to the VPN in case of a private registry.

When using ECR (AWS’s registry), you can authenticate this way:

% which docker_login
docker_login () {
	aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 9876543210.foo.ecr.us-east-1.amazonaws.com
}

— § —

Reply via email

#aws #dev #docker

Docker on Alpine Linux

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 30 Jul 2024 22:56:58 +0200

♠ In this post: how to properly start docker on an Alpine Linux server.

First, install docker and friends:

doas apk add docker docker-cli docker-compose

Then start the docker service:

doas service docker start

Check if it started successfully:

service docker status

If not, then look at the logs:

less /var/log/docker.log

I got an error:

failed to start daemon: error initializing graphdriver: driver not supported

The suggestion was to change the driver to overlay2:

% $EDITOR /etc/docker/daemon.json
{
  "storage-driver": "overlay2"
}

Then restart docker.

I got another error:

level=error msg="failed to mount overlay: no such device" storage-driver=overlay2

The suggestion was to reboot:

doas reboot

Then start docker again:

doas service docker start

And now everything works!

— § —

Reply via email

#alpine-linux #dev #docker

Adding a healthcheck to chartmuseum in AWS Fargate

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 17 May 2024 11:15:09 +0200

♠ Assume that you have a Chartmuseum container running in AWS Fargate.

Chartmuseum is a repository for helm charts. AWS Fargate is an Amazon service to run containers (“serverless”), being part of ECS (Elastic Container Service).

Problem statement: Add a container healthcheck to the chartmuseum task definition associated with the container.

The official docs suggest using curl:

["CMD-SHELL", "curl -f http://localhost/ || exit 1"]

For Chartmuseum specifically we’re interested in its /health endpoint, as per this reference:

["CMD-SHELL", "curl -f http://localhost/health || exit 1"]

But we’re using port 8080:

["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]

If you use this healthcheck for the official chartmuseum image (ghcr.io/helm/chartmuseum) it will fail, because the Alpine Linux environment it uses does not contain curl.

A straightforward fix is to use wget instead:

["CMD-SHELL", "wget -q --spider http://localhost:8080/health || exit 1"]

--spider is needed because we do not want to download anything, -q is optional and short for “quiet”.

The /health endpoint merely returns a simple JSON:

{"healthy":true}

References: https://stackoverflow.com/questions/47722898/how-can-i-make-a-docker-healthcheck-with-wget-instead-of-curl

— § —

Reply via email

#aws #dev #docker #kubernetes