Kubernetes on ¬ just serendipity 🍀

when Gen AI cites docs that don't exist

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 16 Jul 2026 12:34:22 +0200

♠ Today in AGI: two Claude agents, same annotation, opposite answers.

I had a PR touching an ArgoCD PostSync hook. Ordering hooks within a phase needs an annotation, so I asked; Claude Code (Opus 4.8) and Claude in GitHub disagreed:

Claude Code: argocd.argoproj.io/hook-weight does not exist.
Claude in GitHub: it does exist.

Both stated it flatly. So I pushed the GitHub agent, and it doubled down — citing documentation(!):

This claim is incorrect. argocd.argoproj.io/hook-weight is a documented ArgoCD annotation for controlling hook execution order within a phase. From the ArgoCD resource hooks documentation:

Citing non-existent docs is the clanker equivalent of pulling rank.

Confident. Sourced. Wrong. The actual annotation for ordering is argocd.argoproj.io/sync-wave:

They are defined by the argocd.argoproj.io/sync-wave annotation. The value is an integer that defines the ordering.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"

There is no hook-weight. The “documentation” it quoted from doesn’t say what it says it says.

Eventually it folded:

You are correct, and I apologize for the confident but incorrect claim in round 3.

Round 3. It took three rounds and a fabricated citation before the model conceded. The failure mode isn’t the hallucination — it’s the escalation to """docs""" as authority to defend it.

Same model family, same question, one right and one wrong. Verify against the upstream source, not against the more confident agent.

Human-in-the-loop is still very much needed, folks.

🤖 Drafted with /bloggify.

— § —

Reply via email

#ai #dev #kubernetes

kubectl: atomic upsert

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 09 Jun 2026 16:16:12 +0200

♠ Problem statement: A cronjob refreshes an image-pull secret in every namespace so nodes can authenticate to a private registry. It does the obvious™ thing: delete the old secret, then create the new one right away:

kubectl delete secret --ignore-not-found aws-ecr-token -n "$namespace"
kubectl create secret docker-registry aws-ecr-token -n "$namespace" \
    --docker-server="https://$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com" \
    --docker-username="AWS" \
    --docker-password="$(aws ecr get-login-password --region "$AWS_REGION")"

Every couple of hours, for a fraction of a second, that secret does not exist in the cluster (booo!). Most of the time nobody notices. Occasionally a deploy step that copies the secret into a tenant namespace runs into exactly that gap, its kubectl get returns non-zero, and the whole deployment fails with ImagePullError. Occasional flakiness, our favorite kind.

The idiomatic fix is to never delete.

The solution is to employ kubectl create with a client-side dry-run that renders the object to YAML without touching the cluster; pipe that into kubectl apply and we get an upsert:

kubectl create secret docker-registry aws-ecr-token -n "$namespace" \
    --docker-server="https://$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com" \
    --docker-username="AWS" \
    --docker-password="$(aws ecr get-login-password --region "$AWS_REGION")" \
    --dry-run=client -o yaml | kubectl apply -f -

The first run creates the secret; every run afterwards updates it in place. There is no moment wherein it’s absent.

The thing I wanted to be sure of before trusting it: does apply over a secret that was originally created actually rotate the password, or does it merge in some way that leaves stale data behind? kubectl apply does a 3-way merge against the last-applied-configuration annotation, and that merge has bitten me before. A secret with no prior annotation falls back to a 2-way merge — so this is worth checking, just in case.

So I checked. Two consecutive upserts on a throwaway namespace, rotating the password between them:

% kubectl get secret aws-ecr-token -n "$ns" -o jsonpath='{.metadata.uid}'
38d4c676-7169-4d1d-9911-9de068bba29a   # run 1
# [...]
38d4c676-7169-4d1d-9911-9de068bba29a   # run 2 — same uid: never deleted

% kubectl get secret aws-ecr-token -n "$ns" \
    -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
{"auths":{"...":{"username":"AWS","password":"PASSWORD_V2_ROTATED",...}}}

The uid is stable across runs — proof the object was never deleted and recreated, only updated. resourceVersion bumped, the password rotated to the new value, and no trace of the old one lingered. The 2-way-merge worry didn’t materialize: for a flat field like .data.\.dockerconfigjson, apply replaces the value outright.

A uid that survives a refresh is the whole proof. Delete-then-create mints a new uid every cycle; an in-place update keeps it. If we ever want to know whether something replaced an object or mutated it, the uid is the way to go.

Two more things worth highlighting:

It’s atomic from a consumer’s point of view — the secret is never missing. It is not a transaction. apply is still a single API write; we’re trading a delete-then-create pair (two writes, with a gap) for one write (no gap).
--dry-run=client is the spelling since k8s 1.18. Older clusters used a bare --dry-run or --dry-run=true.

The same pattern works for configmaps, or anything else kubectl create knows how to build. create ... --dry-run=client -o yaml | apply -f - is the way to do whenever we’d otherwise reach for delete-then-create.

— § —

Reply via email

#dev #kubernetes

ArgoCD: persistent PDB selector diff

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 22 May 2026 16:51:12 +0200

♠ Previously.

Problem statement: An ArgoCD application shows a persistent diff on a PodDisruptionBudget — spec.selector.matchExpressions is in the rendered manifest but missing from the live object. Sync doesn’t clear it. Hard refresh doesn’t clear it. Syncing with replace or with server-side apply or with force doesn’t clear it. Ugh!

policy/PodDisruptionBudget/default/my-super-cool-app:

selector:
  matchExpressions:
    - key: job-name
      operator: DoesNotExist
  matchLabels:
    release: my-super-cool-app

The chart was updated to add matchExpressions to the PDB selector. The diff shows up. Sync runs. Diff stays.

First idea: Kubernetes mutates PDB selectors server-side — no. The PDB registry strategy in pkg/registry/policy/poddisruptionbudget/strategy.go only touches Status and Generation. No selector mutation, no admission plugin in tree that injects this. Patterns like matchExpressions: [{key: job-name, operator: DoesNotExist}] come from chart authors, not from K8s. This is all according to the LLM.

Second idea: PDB selectors are immutable on older K8s — is also wrong here. Selector mutability landed in 1.27. The cluster is on 1.33+.

The actual culprit is client-side kubectl apply 3-way merge, which is what ArgoCD uses by default. The merge needs three inputs:

The previous applied state, read from metadata.annotations.kubectl.kubernetes.io/last-applied-configuration.
The live object.
The new desired manifest.

Check the annotation on the live PDB:

% kubectl get pdb my-super-cool-app -n default -o yaml | yq '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"'
{"apiVersion":"policy/v1","kind":"PodDisruptionBudget","metadata":{...},"spec":{"maxUnavailable":1,"selector":{"matchLabels":{"release":"my-super-cool-app"}}}}

No matchExpressions. So the 3-way merge sees: the previous apply didn’t have the field, the live object doesn’t have the field, the new manifest has the field. It treats this as an addition — but the existing implementation in this path doesn’t reliably propagate the new key into the live selector when the parent (spec.selector) is owned as an opaque value. The managedFields confirm it:

- manager: argocd-controller
  operation: Update
  fieldsV1:
    f:spec:
      f:maxUnavailable: {}
      f:selector: {}   # opaque — no sub-field ownership

f:selector: {} means argocd-controller owns the whole selector blob with no sub-field tracking. New fields under spec.selector get lost in the merge.

The fix is to delete the live PDB and let ArgoCD recreate it on the next sync:

% kubectl delete pdb my-super-cool-app -n default

This can alternatively be done via the ArgoCD Web UI.

It is a safe operation: a PDB only gates voluntary disruptions (drains, evictions). A brief gap doesn’t affect running pods. The recreated object gets a fresh last-applied-configuration containing matchExpressions, and the diff clears.

The longer-term fix is to switch the application to server-side apply, which tracks ownership at the field level and handles this correctly:

syncPolicy:
  syncOptions:
    - ServerSideApply=true

It’s always a dilemma whether or not to use server-side apply.

— § —

Reply via email

#dev #kubernetes

ArgoCD: stale Vault secret after population

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 13 May 2026 12:34:36 +0200

♠ Problem statement: An ArgoCD application references a Vault secret via helm-secrets / vals (e.g. vault://kv-v2/services/foo#/apiKey). The application is created before the secret exists in Vault, or while the corresponding key is blank. The key is populated later on. The application keeps behaving as if the value is still missing — usually a 404 from Vault, or a permission denied on the cached lookup:

Failed to load target state: failed to generate manifest for source 1 of 2:
rpc error: code = Unknown desc = Manifest generation error (cached): failed to
execute helm template command: [...] vals error: expand
vault://kv-v2/services/foo#/apiKey: Error making API request. URL: GET
http://vault.infra-services:8200/v1/sys/internal/ui/mounts/kv-v2/services/foo
Code: 404.

Note the (cached) part of the error. The argocd-repo-server caches the result of manifest generation, including failed vals / helm-secrets lookups. A normal sync or refresh happily reuses the cached (stale) failure and never re-resolves the Vault reference. So the freshly-populated secret goes unnoticed. A blank to non-blank update counts as well, besides outright token rotations.

The fix is to perform a Hard Refresh on the application — that’s what busts the manifest cache and forces a fresh Vault lookup:

argocd app get <app> --hard-refresh

Or, in the web UI: open the application → REFRESH ▾ dropdown → HARD REFRESH.

If a hard refresh alone doesn’t pick up the new value, restart the argocd repo server:

kubectl rollout restart deployment -n argocd argocd-repo-server

A regular refresh won’t do. The cache key for the failed lookup outlives the refresh.

By default, the application cache lasts for 24h (though it can be changed via helm values); hence, another way to resolve the issue is to simply sit out on it and wait.

— § —

Reply via email

#dev #kubernetes

ArgoCD: sync stuck in terminating

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 05 May 2026 13:51:10 -0200

♠ Problem statement: An ArgoCD application’s sync operation is stuck in Terminating for hours. The web UI shows operation is terminating due to timeout. A PreSync hook resource keeps appearing as Running even though the underlying Job already completed in the cluster.

% kubectl get application <app> -n argocd -o yaml | yq '.status.operationState.phase'
Terminating

The application controller is stuck in a busy loop:

"Resuming in-progress operation. phase: Terminating, message: operation is terminating due to timeout"
"No operation updates necessary to '<ns>/<app>'. Skipping patch"

In my case this followed a transient ImagePullBackOff on a PreSync hook job: the image was being mirrored to a regional ECR, the first hook attempt failed, and by the time the image landed and the retry succeeded, the parent operation had already moved into Terminating and never recovered.

The instinct is to clear the operation:

kubectl patch application <app> -n argocd --type merge -p '{"operation": null}'

It’s a no-op. The wedge doesn’t live in spec.operation (the request) — it lives in status.operationState (the controller’s bookkeeping). And status is owned by the controller, so a regular --type merge patch won’t touch it.

We need the status subresource:

kubectl patch application <app> -n argocd --type merge \
  -p '{"operation": null}'
kubectl patch application <app> -n argocd --subresource status --type merge \
  -p '{"status":{"operationState":null}}'

Both patches together. Neither touches cluster resources — they only modify ArgoCD’s view of the operation. After that, re-trigger the sync from the UI and the application reconciles cleanly.

Restarting argocd-application-controller does not help: the stuck state sits in etcd, and the freshly-elected leader picks it up on startup and resumes the same loop.

One caveat: verify the underlying hook resource has actually finished before clearing. kubectl describe job -n argocd <hook-job> should show 1 Succeeded. Otherwise we’ll lose in-flight progress.

— § —

Reply via email

#dev #kubernetes

external-secrets: bust the ClusterSecretStore cache

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 24 Apr 2026 01:22:06 +0200

♠ This post is hyper-specific but I am quite sure I’ll forget how to do it the next time it happens again, so I’m better off documenting it.

External Secrets:

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Secrets Manager, Pulumi ESC and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

Problem statement: a ClusterSecretStore is stuck on a past authentication failure long after the Vault-side config was fixed, and every ExternalSecret that references it reports SecretSyncedError: the desired SecretStore is not ready.

Verify the store really is healthy now:

% kubectl describe clustersecretstore external-secrets-vault-backend
...
Status:
  Conditions:
    Last Transition Time:  2026-04-21T23:16:45Z   # <- stale
    Message:               unable to create client
    Reason:                InvalidProviderConfig
    Status:                False
    Type:                  Ready

Last Transition Time is from before I fixed the auth role. The operator hasn’t re-evaluated since. Poke it with an annotation to force a reconcile:

% kubectl annotate clustersecretstore external-secrets-vault-backend \
    force-reconcile=$(date +%s) --overwrite

Now it validates:

% kubectl describe clustersecretstore external-secrets-vault-backend | grep -A3 Status:
Status:
  Capabilities:  ReadWrite
  Conditions:
    Last Transition Time:  2026-04-23T23:11:30Z
    Message:               store validated

But the ExternalSecrets are still stuck — the operator caches store readiness per-ES, not just per-CSS. Same trick:

% kubectl annotate externalsecret -n argocd --all \
    force-reconcile=$(date +%s) --overwrite
externalsecret.external-secrets.io/argocd-external-secret annotated
externalsecret.external-secrets.io/argocd-external-secret-github-webhook annotated
...

% kubectl get externalsecret -n argocd
NAME                       STORE                            STATUS         READY
argocd-external-secret     external-secrets-vault-backend   SecretSynced   True
...

No controller restart is necessary.

The annotation value doesn’t matter — any change to the object metadata re-queues it.

— § —

Reply via email

#dev #kubernetes

Kubernetes: pod memory usage

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 26 Sep 2025 11:54:57 +0200

♠ Problem statement: Given a pod in Kubernetes, check its memory (RAM) usage.

Stack Overflow:

kubectl top pod -n {namespace} {pod name} [--containers]

--containers=false:
    If present, print usage of containers within a pod.

This only works if metrics server is installed (it should be!).

There’s an alternative:

kubectl describe PodMetrics {pod name}

It must be PodMetrics. PodMetric does not work (ugh!¹):

% kubectl get PodMetric
error: the server doesn't have a resource type "PodMetric"

If metrics server is not available for some reason, then one needs to dive into the container:

kubectl exec -it {pod name} -n {namespace} -- /bin/sh

…which won’t always work; not every container has /bin/sh. Then read the contents of /sys/fs/cgroup/memory/memory.usage_in_bytes.

Who designed this? Is it so hard to achieve consistency? Both kubectl get pod and kubectl get pods work. Why can’t it be the same with PodMetric? And, no, PodMetricses does not work either. ↩︎

— § —

Reply via email

#dev #kubernetes

kubectl: verbose logging

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 05 Sep 2025 18:10:33 +0200

♠ When running kubectl commands, to get a glimpse of what is happening under the hood, pass -v={N} to it.

For example:

% kubectl get pod -n argocd -v=6
I0905 16:10:19.893576 3929356 loader.go:395] Config loaded from file:  /home/ubuntu/.kube/config
I0905 16:10:19.919935 3929356 round_trippers.go:553] GET https://{REDACTED}.us-east-1.elb.amazonaws.com/api/v1/namespaces/argocd/pods?limit=500 200 OK in 17 milliseconds
NAME                                                  READY   STATUS    RESTARTS      AGE
argo-workflows-server-6ffcfbb779-kxdzk                1/1     Running   0             4d18h
argo-workflows-workflow-controller-85c667cf96-rm9dw   1/1     Running   0             2d4h
argocd-application-controller-0                       1/1     Running   0             48m
[...]

-v=10 is super verbose. Fine-tune as needed.

Another example:

% kubectl get deploy -n argocd -w -v6
I0905 16:12:49.277471 3929550 loader.go:395] Config loaded from file:  /home/ubuntu/.kube/config
I0905 16:12:49.302783 3929550 round_trippers.go:553] GET https://{REDACTED}.us-east-1.elb.amazonaws.com/apis/apps/v1/namespaces/argocd/deployments?limit=500 200 OK in 17 milliseconds
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
argo-workflows-server                1/1     1            1           91d
argo-workflows-workflow-controller   1/1     1            1           44d
argocd-applicationset-controller     1/1     1            1           23d
argocd-dex-server                    1/1     1            1           451d
argocd-notifications-controller      1/1     1            1           449d
argocd-redis                         1/1     1            1           451d
argocd-repo-server                   1/1     1            1           451d
argocd-server                        3/3     3            3           428d
I0905 16:12:49.316961 3929550 round_trippers.go:553] GET https://{REDACTED}.us-east-1.elb.amazonaws.com/apis/apps/v1/namespaces/argocd/deployments?resourceVersion=695374153&watch=true 200 OK in 2 milliseconds

— § —

Reply via email

#dev #kubernetes

Uninstall Kyverno

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 05 Sep 2025 17:03:04 +0200

♠ It’s always a bit frustrating to do it.

Despite the excellent upstream documentation, our Kubernetes clusters often leave cruft behind when doing so.

% kubectl rollout restart sts -n argocd argocd-redis-ha-server
E0905 12:29:24.284879 3908207 memcache.go:287] couldn't get resource list for wgpolicyk8s.io/v1alpha2: the server is currently unable to handle the request
E0905 12:29:24.287856 3908207 memcache.go:287] couldn't get resource list for reports.kyverno.io/v1: the server is currently unable to handle the request
E0905 12:29:24.290250 3908207 memcache.go:121] couldn't get resource list for reports.kyverno.io/v1: the server is currently unable to handle the request
E0905 12:29:24.292666 3908207 memcache.go:121] couldn't get resource list for wgpolicyk8s.io/v1alpha2: the server is currently unable to handle the request
E0905 12:29:24.295959 3908207 memcache.go:121] couldn't get resource list for reports.kyverno.io/v1: the server is currently unable to handle the request
E0905 12:29:24.298226 3908207 memcache.go:121] couldn't get resource list for wgpolicyk8s.io/v1alpha2: the server is currently unable to handle the request
E0905 12:29:24.302703 3908207 memcache.go:121] couldn't get resource list for reports.kyverno.io/v1: the server is currently unable to handle the request
E0905 12:29:24.305009 3908207 memcache.go:121] couldn't get resource list for wgpolicyk8s.io/v1alpha2: the server is currently unable to handle the request
[...]

That output junk is emitted on every kubectl command.

This time the culprit was:

% kubectl get apiservice | egrep -i "(kyverno|wgpolicy)"
v1.reports.kyverno.io                           reports-server/kyverno-reports-server   False (MissingEndpoints)   76d
v1alpha2.wgpolicyk8s.io                         reports-server/kyverno-reports-server   False (MissingEndpoints)   76d

Deleting these two entries resolves the issue:

% kubectl delete apiservice v1.reports.kyverno.io v1alpha2.wgpolicyk8s.io -n reports-server

— § —

Reply via email

#dev #kubernetes

helm: diff manifests

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 28 Aug 2025 17:57:32 +0200

♠ Lately I’ve been employing (with great success) the following pattern in order to test whether changes to helm charts are harmless (and expected):

cd path/to/my/helm/chart
git checkout origin/master  # or main
helm template -f values.yaml . > golden
git switch --create thiagowfx/my-feature-branch
# develop, develop, develop
helm template -f values.yaml . > candidate
diff candidate golden  # or icdiff

The diff output will reliably demonstrate whether your changes are WAI.

Note: If needed, pass --release-name {foo} to helm template.

This pattern is comparable to running a golden helm install followed by a helm-diff (helm diff upgrade) candidate in the cluster.

— § —

Reply via email

#dev #kubernetes

ArgoCD: git index.lock issue

serendipity@perrotta.dev (Thiago Perrotta) — Sat, 23 Aug 2025 12:39:14 +0200

♠ Problem statement: Every ArgoCD application is stuck in Unknown state, with the following error:

Failed to load target state: failed to generate manifest for source 1 of 2:
rpc error: code = Unknown desc = failed to initialize repository resources: rpc
error: code = Internal desc = Failed to checkout FETCH_HEAD: failed to checkout
FETCH_HEAD: `git checkout --force FETCH_HEAD` failed exit status 128: fatal:
Unable to create '<path to cached source>/.git/index.lock': File exists. Another
git process seems to be running in this repository, e.g. an editor opened by
'git commit'. Please make sure all processes are terminated then try again. If
it still fails, a git process may have crashed in this repository earlier:
remove the file manually to continue.

It is a lock contention race condition issue within the ArgoCD repo server component, with the git index.lock.

The easiest way to address it is to restart the repo server:

kubectl rollout restart deploy -n argocd argocd-repo-server

Afterwards, refresh all applications in the cluster via the web UI:

click Applications on the left-side nav bar
click Refresh apps at the top
select ALL applications, set refresh type to NORMAL
click refresh

— § —

Reply via email

#dev #git #kubernetes

★ Kubernetes: list all ImagePullBackOff container images

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 21 Aug 2025 13:34:27 +0200

♠ Problem statement Upgrade the kube-prometheus-stack helm chart in Azure China to the latest version. Side effect: Many container images are now missing from our internal ACR (Azure Container Registry). They need to be synced with skopeo.

Which images need to be synced? Use the command below, it is quite readable¹:

kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(any(.status.containerStatuses[]; .state.waiting.reason == "ImagePullBackOff")) | .metadata.namespace + "/" + .metadata.name + "\t" + ( [.status.containerStatuses[] | select(.state.waiting.reason == "ImagePullBackOff").image] | join(", ") )'

Sample run:

ubuntu@universe:~ $ kubectl get pods --all-namespaces -o json | jq -r '.items[]
      | select(any(.status.containerStatuses[]; .state.waiting.reason == "ImagePullBackOff"))
      | .metadata.namespace + "/" + .metadata.name + "\t" +
        ( [.status.containerStatuses[] | select(.state.waiting.reason == "ImagePullBackOff").image] | join(", ") )'
monitoring/clustermon-kube-prometheus-operator-67c8b6c87c-w72km {my-registry-name}.azurecr.cn/quay.io/prometheus-operator/prometheus-operator:v0.83.0
monitoring/clustermon-prometheus-node-exporter-v7bpt    {my-registry-name}.azurecr.cn/quay.io/prometheus/node-exporter:v1.9.1

How to skopeo? First, log in:

echo -n "{password}" | skopeo login {my-registry-name}.azurecr.cn --password-stdin --username {my-username}

How do you get these credentials? Log in to the Azure China Portal, find the ACR resource, navigate to Settings > Access keys.

Then sync each image²:

% skopeo sync \
  --override-os linux --override-arch amd64 \
  --src docker --dest docker \
  quay.io/prometheus/node-exporter:v1.9.1 \
  {my-registry-name}.azurecr.cn/quay.io/prometheus

I cannot promise I am able to craft this by hand without looking up man pages or getting assistance from an LLM. C’est la vie. ↩︎
If there are many images, consider using an array + for loop. ↩︎

— § —

Reply via email

#bestof #dev #kubernetes

ArgoCD: app version notifications

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 12 Aug 2025 12:07:20 +0200

♠ Problem statement: when using ArgoCD notifications, starting off from their catalog, print the app chart version in addition to existing information.

More specifically, currently, a Slack notification message from Argo looks like the following:

Cluster 37: Application heart-of-gold is unhealthy.

We want to improve it to:

Cluster 37: Application heart-of-gold (~1.18.3) is unhealthy.

The following diff (under a template block) accomplishes that:

-          {{if eq .serviceType "slack"}}:interrobang:{{end}} Cluster <{{.context.argocdUrl}}|*{{.context.clusterId}}*>: Application <{{.context.argocdUrl}}/applications/{{.app.metadata.name}}|`{{.app.metadata.name}}`> _sync_ by {{.app.status.operationState.operation.initiatedBy.username | default "_automation_" }} is _unknown_.
+          {{if eq .serviceType "slack"}}:interrobang:{{end}} Cluster <{{.context.argocdUrl}}|*{{.context.clusterId}}*>: Application <{{.context.argocdUrl}}/applications/{{.app.metadata.name}}|`{{.app.metadata.name}}`>{{if .app.spec.source.targetRevision}} ({{.app.spec.source.targetRevision}}){{else}}{{range .app.spec.sources}}{{if .targetRevision}} ({{.targetRevision}}){{break}}{{end}}{{end}}{{end}} _sync_ by {{.app.status.operationState.operation.initiatedBy.username | default "_automation_" }} is _unknown_.

Diving into it with pretty-printing¹:

{{if .app.spec.source.targetRevision}}
  ({{.app.spec.source.targetRevision}})
{{else}}
  {{range .app.spec.sources}}
    {{if .targetRevision}}
      ({{.targetRevision}})
      {{break}}
    {{end}}
  {{end}}
{{end}}

If there’s a single source in the helm chart, we extract its targetRevision.

Otherwise, when using a multi-source app, we extract the first available targetRevision.

These are golang templates. ↩︎

— § —

Reply via email

#dev #kubernetes

ArgoCD: recovery when traefik is broken

serendipity@perrotta.dev (Thiago Perrotta) — Sat, 09 Aug 2025 15:24:53 +0200

♠ It’s Friday. The best day to push to prod™. An internal scheduled upgrade for traefik goes awry. Internal services are no longer accessible. HTTP requests won’t be properly routed.

The most sensible SRE / DevOps will simply roll it back. Roll back first, investigate later: it’s crucial to minimize downtime.

How do we roll back? We’re using GitOps, the most sensible approach is to make a new git commit by submitting a pull request. However, this is an emergency situation. Can we roll back faster?

Sure, let’s make temporary changes directly in the cluster.

The traefik app is managed with ArgoCD. Let’s log in to the ArgoCD web UI and roll back to the previous version.

Oh, wait. We can’t. The ArgoCD web UI isn’t accessible, because traefik is broken. Sighs.

Let’s use the ArgoCD CLI then.

Aaaah, no. The ArgoCD CLI is really a REST client for the ArgoCD API. HTTP(S) is not accessible, so the CLI won’t work either.

What’s next? A plain helm upgrade should work. But helm is picky and will refuse to touch manifests it doesn’t own. The manifests are managed by Argo. helm won’t budge. Oh gosh.

Last option: the ArgoCD controllers are still running. We don’t need the ArgoCD HTTPS API to communicate with them. Let’s simply edit the k8s manifests directly.

First we patch the root app (app-of-apps pattern):

kubectl edit app -n argocd root

We remove the automated block, to effectively pause auto sync, otherwise our subsequent changes to traefik would be immediately reverted:

spec:
#  automated:
#    prune: [..]
#    selfHeal: [...]

You can comment or delete the automated: block, either is fine.

Next, we need to remove traefik from all application project (app project) sync windows, so that changes to it will be immediately applied:

kubectl edit appproject -n argocd cluster

The traefik entry got removed from all sync windows.

Now, finally, let’s instruct Argo to roll traefik back.

The broken version is 2.5.1, the previous working version was 2.4.1.

kubectl edit app -n argocd traefik

Update targetRevision: 2.4.1 under the helm block.

Wait a few seconds. Observe traefik pods as they are rolling restarted:

kubectl get deploy -n kube-system traefik -w

Finally, all services are once again accessible!

From this point on, we can go back to git and do a proper pull request revert to reconcile the codebase with the live cluster state.

This was an interesting exercise. I’m quite familiar with ArgoCD and GitOps at this point, and I instinctively knew what to do in this situation, without needing to consult the official documentation or even an LLM. I wouldn’t have minded doing either of these if needed, but the point is that it feels good to be well familiar with how a system works and how to apply your knowledge to fix day-to-day problems.

— § —

Reply via email

#dev #kubernetes

ArgoCD: all applications stuck on refresh

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 01 Aug 2025 14:30:58 +0200

♠ The problem is very likely the Application Controller.

You can restart it:

kubectl rollout restart sts -n argocd argocd-application-controller

Or inspect its logs:

kubectl logs -n argocd argocd-application-controller-0 --tail=100 | less

While we’re here, make sure there is at least one replica running:

% kubectl get sts -n argocd argocd-application-controller
NAME                            READY   AGE
argocd-application-controller   1/1     415d

Upstream issue.

— § —

Reply via email

#dev #kubernetes

ArgoCD: deployment spec.selector: field is immutable

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 30 Jun 2025 13:04:51 +0200

♠ When using ArgoCD to manage an app, the Deployment manifest needs to undergo significant changes, updating its selector labels. When trying to sync the argocd app, we get the following error:

one or more objects failed to apply, reason: error when patching "/dev/shm/3469418025": Deployment.apps "myapp" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:v1.LabelSelectorRequirement(nil)}: field is immutable. Retrying attempt #2 at 11:02AM.

This error doesn’t come directly from Argo, it is from kubectl apply.

There are two ways to address it:

Sync the argo app with the “force” option enabled
Run kubectl apply with --force (which is ultimately what the aforementioned option maps to).

In action (-w is short for --watch):

% kubectl get endpoints -w
NAME         ENDPOINTS         AGE
myapp        <none>            5y44d
myapp                          5y44d
myapp        10.1.13.40:8080   5y44d

— § —

Reply via email

#dev #kubernetes

Helm repository with a trailing slash

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 24 Jun 2025 11:32:50 +0200

♠ A repository URL with a trailing slash, and the same URL without a trailing slash, are treated as different repos. We found this out the hard way:

Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = error building helm chart dependencies: failed to add helm repository https://chartmuseum.{corp intranet}/orgs/{corp}/: failed to add repository: failed to get command args to log: `helm repo add https:--chartmuseum.{corp intranet}-orgs-{corp}- https://chartmuseum.{corp intranet}/orgs/{corp}/` failed exit status 1: Error: looks like "https://chartmuseum.{corp intranet}/orgs/{corp}/" is not a valid chart repository or cannot be reached: failed to fetch https://chartmuseum.{corp intranet}/orgs/{corp}/index.yaml : 401 Unauthorized

Let’s put aside the fact the error message REALLY likes to repeat the repo name.

This is a repository that we’ve already configured, with proper authentication credentials. Why would it start to fail now, all of a sudden?

You already got spoiled in the post title. Given Chart.yaml:

apiVersion: v2
name: corp-postgres-operator
description: Corp Postgres Operator
type: application
version: 1.1.13

dependencies:
  - name: postgres-operator
    repository: https://chartmuseum.{corp intranet}/orgs/archive

And also:

apiVersion: v2
name: corp-postgres-operator
description: Corp Postgres Operator
type: application
version: 1.1.13

dependencies:
  - name: postgres-operator
    repository: https://chartmuseum.{corp intranet}/orgs/archive/

These two dependencies end up being treated differently by Argo, even though they are effectively the same. It’s an upstream bug.

To work around it, I decided to simply remove all trailing slashes from helm repository dependencies. Canonicalize everything!

A git pre-commit.com hook is a good mechanism to do so:

repos:
  repo: local
  hooks:
    - id: helm-repo-trailing-slash
      name: Helm repositories must not contain trailing slashes in Chart.yaml
      files: Chart.yaml$
      language: pygrep
      entry: '\s*repository: [\w:/.]+/$'

Run pre-commit run -a helm-repo-trailing-slash, fix all violations, and then never think about this again.

Should the upstream bug ever be fixed, we can remove this workaround.

— § —

Reply via email

#dev #kubernetes #pre-commit

★ ArgoCD: every application in unknown state

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 13 Jun 2025 22:34:33 +0200

♠ Symptom: Every single application in the Kubernetes cluster that is managed by Argo is in Unknown sync status. A disaster! 😱

This has only happened in one of our clusters. More specifically, in an AKS cluster (from Microsoft Azure).

An observed effect:

% kubectl get --raw='/openapi/v2' | head -c 200
Error from server (ServiceUnavailable): the server is currently unable to handle the request

Aha! The Kubernetes OpenAPI server is not responding properly. That explains why Argo is having so much trouble to figure out the state of affairs in the cluster.

It was quite difficult to troubleshoot this issue. There are barely any resources about it in the open web. We filed an Azure Support ticket, but it was completely useless. They offered to restart¹ the OpenAPI server, but that didn’t help (of course it wouldn’t!).

Because this is a managed Kubernetes cluster, we have limited access to it, which includes lack of access to raw logs. The Azure portal doesn’t make it easy but, after digging quite deep into it, eventually we found this error message in the logs:

 {"pod":"kube-apiserver-7785d59f4f-scqmp","stream":"stderr","containerID":"925345f6501626aa6cd40dd187d5cf57837389eec9cb39b2292fb9ae3f66242f","log":"E0512 12:01:36.876768       1 handler.go:160] Error in OpenAPI handler: failed to build merge specs: unable to merge: duplicated path \/apis\/reports.kyverno.io\/v1\/clusterephemeralreports\n"}

That was the first clue! Now we know it’s related to Kyverno.

That error eventually led us to this issue in Kubernetes:

We have the same issue here, actually also triggered by installing the kyverno reports-server with both apiservices (different groups served by the same api-server).

Aha!

Deleting the duplicate CRDs installed by kyverno reports server has immediately resolved the OpenAPI server unresponsiveness. It wasn’t even necessary to restart it:

% kubectl delete crd ephemeralreports.reports.kyverno.io clusterephemeralreports.reports.kyverno.io clusterpolicyreports.wgpolicyk8s.io policyreports.wgpolicyk8s.io
customresourcedefinition.apiextensions.k8s.io "ephemeralreports.reports.kyverno.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterephemeralreports.reports.kyverno.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterpolicyreports.wgpolicyk8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "policyreports.wgpolicyk8s.io" deleted

And now:

$ kubectl get --raw='/openapi/v2' | head -c 200
{"swagger":"2.0","info":{"title":"Kubernetes","version":"v1.30.11"},"paths":{"/.well-known/openid-configuration/":{"get":{"description":"get service account issuer OpenID configuration, also known as

That was a tricky one! Interestingly: it has only occurred in AKS. EKS was fine.

Hello, IT. Have you tried turning it off and on again?. ↩︎

— § —

Reply via email

#bestof #dev #kubernetes

★ ArgoCD: custom health check for StatefulSet (OnDelete)

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 13 Jun 2025 21:40:01 +0200

♠ It turns out the built-in health check for StatefulSets in ArgoCD is not comprehensive enough.

It covers RollingUpdate well enough, but OnDelete is incomplete.

More specifically:

Observed: whenever you push an update to an STS whose .spec.updateStrategy is OnDelete, its health status stays as Healthy.

Expected: there should be an indication that the STS pods are not up-to-date, and hence need to be restarted to incorporate the latest changes.

There is an open bug: Statefulset healthcheck does not work #6527 for it.

I managed to work around it by defining my own health check. This is an use case where Gen AI is helpful, though it does not shine. I could not have Gen AI produce a fully working health check for this scenario. Instead, I used it to bootstrap a basic example, then had to tweak it until it worked. This has been tested multiple times, I am reasonably confident it works well (some edge cases could be missing though).

health_status = {}

if obj.status == nil then
  health_status.status = "Progressing"
  health_status.message = "Waiting for status to be reported"
  return health_status
end

local strategy = obj.spec.updateStrategy and obj.spec.updateStrategy.type or "RollingUpdate"
local spec_replicas = obj.spec.replicas or 1
local ready_replicas = obj.status.readyReplicas or 0
local updated_replicas = obj.status.updatedReplicas or 0
local status_replicas = obj.status.replicas or 0
local current_revision = obj.status.currentRevision or ""
local update_revision = obj.status.updateRevision or ""

if strategy == "OnDelete" then
  if ready_replicas == status_replicas and updated_replicas == spec_replicas then
    health_status.status = "Healthy"
    health_status.message = "All replicas are ready and updated (OnDelete strategy)"
  elseif updated_replicas == 0 then
    health_status.status = "Degraded"
    health_status.message = "No replicas have been updated (OnDelete strategy): 0/" .. spec_replicas .. " updated. Still running an old revision!"
  else
    health_status.status = "Progressing"
    health_status.message = "Progressing (OnDelete): " .. ready_replicas .. "/" .. status_replicas .. " ready, " .. updated_replicas .. "/" .. spec_replicas .. " updated"
  end
else
  if ready_replicas == status_replicas and updated_replicas == spec_replicas then
    if current_revision ~= update_revision then
      health_status.status = "Progressing"
      health_status.message = "Waiting for rollout to complete: revision mismatch (" .. current_revision .. " ≠ " .. update_revision .. ")"
    else
      health_status.status = "Healthy"
      health_status.message = "All replicas updated, ready, and revisions match"
    end
  else
    health_status.status = "Progressing"
    health_status.message = "RollingUpdate progressing: " .. ready_replicas .. "/" .. status_replicas .. " ready, " .. updated_replicas .. "/" .. spec_replicas .. " updated"
  end
end
return health_status

It is necessary to distinguish between OnDelete and RollingUpdate.

OnDelete should not use currentRevision / update_revision, as these values don’t mean anything for that update strategy. This can be observed upstream as well.

It is easy to write / tweak the health check when you look at the live STS resource / manifest in the cluster. obj refers to the STS manifest (kubectl get sts -o yaml). The most relevant field is status:.

If you deploy ArgoCD via its helm chart, add that check for argocd-cm. The values.yaml file:

argo-cd:
  configs:
    cm:
      resource.customizations.health.apps_StatefulSet: |
        health_status = {}
        [...]

This custom health check was partly inspired by a Prometheus alert we have:

alert: StatefulSetUpdateNotRolledOut
expr: max without (revision) (kube_statefulset_status_current_revision unless kube_statefulset_status_update_revision) * (kube_statefulset_replicas != kube_statefulset_status_replicas_updated)

How long will it take until LLMs ~~steal~~ adopt this novel solution from me?

— § —

Reply via email

#bestof #dev #kubernetes

ArgoCD: application stuck in unknown

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 12 Jun 2025 11:51:32 +0200

♠ Problem statement: An application in ArgoCD that was previously running smoothly has suddenly started to fail. Its status has changed to “Unknown”. The “Refresh” button is grayed out in the Argo web UI.

I found the following works in this scenario, which is typically a transient issue:

restart the argo server
restart the argo repo server

In terms of commands:

% kubectl rollout restart deployment argocd-server -n argocd
% kubectl rollout restart deployment argocd-repo-server -n argocd

— § —

Reply via email

#dev #kubernetes

ArgoCD: upgrade to v3, remove helm labels

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 02 Jun 2025 13:03:24 +0200

♠ I am leading the effort to upgrade ArgoCD to its newest major version (v3) across our fleet.

One of its breaking changes is the removal of helm labels from argo applications, like such:

metadata:
# [...]
  labels:
    app.kubernetes.io/managed-by: Helm

Applying a normal sync does not get rid of the label.

I found out that doing an one-off server-side apply sync addresses the label removal, and it is safe to do (incurs no downtime):

However, there are some cases where you want to use kubectl apply --server-side over kubectl apply:

[…]

Patching of existing resources on the cluster that are not fully managed by Argo CD.

There’s no need to make the server-side apply a permanent config, i.e.:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

…though there’s no harm either in doing so.

— § —

Reply via email

#dev #kubernetes

PDB: minAvailable and maxUnavailable cannot be both set

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 02 Jun 2025 12:00:05 +0200

♠ In Kubernetes it’s not possible to have a PodDisruptionBudget (PDB) with both minAvailable and maxUnavailable set simultaneously:

one or more objects failed to apply, reason: PodDisruptionBudget.policy "pgbouncer-pgbouncer" is invalid: spec: Invalid value: policy.PodDisruptionBudgetSpec{MinAvailable:(*intstr.IntOrString)(0xc02c03dd20), Selector:(*v1.LabelSelector)(0xc02c03dd40), MaxUnavailable:(*intstr.IntOrString)(0xc02c03dd60), UnhealthyPodEvictionPolicy:(*policy.UnhealthyPodEvictionPolicyType)(nil)}: minAvailable and maxUnavailable cannot be both set (retried 5 times).

Our manifests did not violate this rule though. Then where did the error message come from?

It turns out ArgoCD was creating an intermediate state containing both properties:

# [...]
spec:
  minAvailable: 1
  maxUnavailable: 1
# [...]

One solution is to sync the underlying application PDB manifest with replace.

Warning: It’s better to “sync with replace” only the PDB. Do not do it in the entire application.

— § —

Reply via email

#dev #kubernetes

kubernetes: list pending pods in daemonset

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 20 May 2025 16:41:00 +0200

♠ Problem statement: Given a DaemonSet with a couple of pods stuck in “Pending” state during their rollout, find out which nodes did not yet complete the pod initialization.

% kubectl get nodes -o name | grep -vFf <(kubectl get pods -n {namespace} -l {ds-label-selector} -o wide --no-headers | awk '{print "node/" $7}')
node/aks-database-12345678-vmss000000
node/aks-database-12345678-vmss000002
node/aks-systempool-87654321-vmss000000
node/aks-systempool-87654321-vmss000001

Notes:

kubectl get nodes -o name: list all node names in the node/<name> form
kubectl get pods [...] -o wide: get the node name for each running DaemonSet pod
awk '{print "node/" $7}': get the node name matching the output of get nodes
grep [...]: find the nodes we are looking for

Surely there must be an easier way to accomplish this?

— § —

Reply via email

#dev #kubernetes

helm-secrets + vals: string interpolation

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 08 May 2025 15:25:14 +0200

♠ helm-secrets is:

a helm plugin that helps to manage secrets with Git workflow and store them anywhere.

It uses vals underneath, which is a:

Helm-like configuration values loader with support for various sources

…including, in particular, HashiCorp Vault.

The vals syntax is documented in the vals documentation.

Problem statement: construct a string composed of the following two vault entries:

vault://kv-v2/services/common/postgres/#url
vault://kv-v2/services/common/postgres/#params

…to yield {url}?{params}.

How do we concatenate two vault entries? The secret is in the +:

Finally, the optional trailing + is the explicit “end” of the expression. You usually don’t need it, as if omitted, it treats anything after ref+ and before the new-line or the end-of-line as an expression to be evaluated. An explicit + is handy when you want to do a simple string interpolation. That is, foo ref+SECRET1+ ref+SECRET2+ bar evaluates to foo SECRET1_VALUE SECRET2_VALUE bar.

The final result:

secrets+literal://ref+vault://kv-v2/services/common/postgres/#url+?ref+vault://kv-v2/services/common/postgres/#params

…wherein secrets+literal:// comes from helm-secrets. It needs to appear only once. ref+vault:// comes from vals, it appears once for each vault entry.

— § —

Reply via email

#dev #kubernetes #security

HashiCorp Vault from helm values

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 23 Apr 2025 16:56:16 +0200

♠ Problem statement: Populate HashiCorp Vault with select keys from helm get values for a given chart release.

The following script works:

#!/bin/bash
# Usage: Tweak entries as needed and then run the script.

set -euo pipefail

# It's an associative array: a dictionary, a hash map
#
#   Key: the entry to be populated in vault
#   Value: a `jq` JSON selector from the output of `helm get values -o json`
declare -A entries=(
    # keep-sorted start
    [config.aws.accessKey]=".config.aws.accessKey"
    [config.aws.secretKey]=".config.aws.secretKey"
    [config.cronjob.apiPassword]=".config.cronjob.apiPassword"
    [config.cronjob.apiUser]=".config.cronjob.apiUser"
    [config.postgresql.adminpass]=".config.postgresql.adminpass"
    [config.postgresql.password]=".config.postgresql.password"
    [config.postgresql.ropass]=".config.postgresql.ropass"
    [config.postgresql.rwpass]=".config.postgresql.rwpass"
    [config.slack.token]=".config.slack.token"
    [config.slack.verificationToken]=".config.slack.verificationToken"
    # keep-sorted end
)

main() {
        vault login

        FIRST=1
        for entry in "${!entries[@]}"; do
                key="$entry"
                value="${entries[$entry]}"
                content="$(helm get values -n tools mychart -o json | jq -r "$value")"

                echo "Populating vault $key from helm value $key: $content..."

                if [[ "$FIRST" == "1" ]]; then
                  vault kv put -mount="kv-v2" "services/mychart/" "$key=$content"
                  FIRST=0
                else
                  vault kv patch -mount="kv-v2" "services/mychart/" "$key=$content"
                fi
        done

        vault kv get -mount="kv-v2" "services/mychart"

        rm ~/.vault-token
}

main

FIRST: In the first time we need to create an entry (vault kv put), in the second time we need to patch it (vault kv patch).

— § —

Reply via email

#dev #kubernetes #security

ArgoCD: vertical-pod-autoscaler sync loop

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 13 Feb 2025 13:17:36 +0100

♠ We’ve had a long time issue wherein our vertical-pod-autoscaler (VPA) ArgoCD application, deployed via its helm chart, got stuck in a sync loop.

More specifically: the helm chart automatically generates a self-signed certificate via the GenCA helm function upon every change to our gitops repository. It turns out this is documented and we can see its corresponding manifest in the chart template.

So that’s where the sync loop was coming from! ArgoCD was constantly reporting (and resolving) a diff in the admission controller secret.

The manifest:

apiVersion: v1
data:
  ca.crt: ++++++++
  tls.crt: ++++++++
  tls.key: ++++++++
kind: Secret
metadata:
[...]
  name: vertical-pod-autoscaler-admission-controller-tls
  namespace: kube-system
[...]

The problematic are all the ones under data:.

I confirmed that was indeed the case in the CNCF Slack #argo-cd. Thanks Tim!

How to address it?

We leverage ignoreDifferences (c.f. Stack Overflow) in the ArgoCD app manifest:

% git diff HEAD~2..HEAD
diff --git apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml
index 92bfbcc..90d8435 100644
--- apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml
+++ apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml
@@ -34,3 +34,15 @@ spec:
         maxDuration: 1m
     syncOptions:
       - CreateNamespace=true
+      - RespectIgnoreDifferences=true
+  ignoreDifferences:
+    - group: ""
+      kind: Secret
+      name: vertical-pod-autoscaler-admission-controller-tls
+      jsonPointers:
+        - /data
+    - group: apps
+      kind: Deployment
+      name: vertical-pod-autoscaler-admission-controller
+      jsonPointers:
+        - /spec/template/metadata/annotations/checksum~1tls-secret

In JSON Pointers (RFC 6901), one escapes the slash character with a ~1 (c.f. section 3 in the spec):

Because the characters '~' (%x7E) and '/' (%x2F) have special meanings in
JSON Pointer, '~' needs to be encoded as '~0' and '/' needs to be encoded as
'~1' when these characters appear in a reference token.

Problem resolved!

In general all helm charts that generate self-signed certificates (with genCA) may experience this sync loop when used with a CD framework such as Argo.

— § —

Reply via email

#dev #kubernetes

Kubernetes: create a pod in pending state

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 07 Feb 2025 13:58:36 +0100

♠ We are making a non-trivial change to our cluster monitoring architecture, and I needed to verify the new rollout is properly working in our infratesting environment.

A simple way to do so is to create a pod that never leaves the “Pending” state, because we have an alert (a prometheus rule) that fires whenever a pod is in that stack for too long (15 minutes is our threshold in most cases).

Use the following manifest:

apiVersion: v1
kind: Pod
metadata:
  name: pending-pod
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: "non/existent"
            operator: In
            values:
            - "true"
        topologyKey: "kubernetes.io/hostname"
  containers:
  - name: busybox
    image: busybox
    command: ['sh', '-c', 'echo Hello, Kubernetes! && sleep 3600']

…it creates a pending-pod in the default namespace that should never be scheduled, due to its strict pod affinity rules.

Save it as pending-pod.yaml, deploy it with kubectl apply -f pending-pod.yaml.

$ kubectl get pod pending-pod -w
NAME          READY   STATUS    RESTARTS   AGE
pending-pod   0/1     Pending   0          21s
 pending-pod   0/1     Pending   0          65s
pending-pod   0/1     Pending   0          74s
pending-pod   0/1     Pending   0          2m6s
pending-pod   0/1     Pending   0          3m8s
pending-pod   0/1     Pending   0          4m8s
pending-pod   0/1     Pending   0          4m18s

At some point the alert fires, as expected. And now we can roll out these changes to prod!

— § —

Reply via email

#dev #kubernetes

ArgoCD: cache: key is missing

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 09 Jan 2025 20:20:03 -0300

♠ If you experience the title error message in ArgoCD (e.g. via its web UI and/or in pod logs), it’s related to argo-cd#5068. The full error message is:

Unable to load data: cache: key is missing

As I commented in the bug, here’s how to fix it:

The following workaround sufficed for me: kubectl delete pod -n argocd argocd-application-controller-0.

Initially I ran kubectl rollout restart deployment -n argocd argocd-application-controller but it didn’t work.

In my experience, this error only happens once, ephemerally, during ArgoCD bootstrapping. Once fixed it does not reoccur.

Update(2025-06-06): This error reoccurs intermittently 🤷.

Another potential workaround is to disable redis / caching altogether, as ArgoCD can fully operate without it, but I wouldn’t recommend that.

Helm: migrate chart dependencies from HTTPS to OCI

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 08 Jan 2025 18:45:57 -0300

♠ I got aware of this due to Bitnami’s blog post.

In Chart.yaml, instead of depending on:

- https://charts.bitnami.com/bitnami/airflow-18.3.2.tgz

Instead, depend on:

- oci://registry-1.docker.io/bitnamicharts/airflow:18.3.2

The switch is transparent, it works out-of-the-box.

However there’s one caveat: You cannot helm repo add oci://registry-1.docker.io/bitnamicharts. That only works for https:// registries.

The workaround is to helm pull oci://[...] --version {version} and/or to browse available tags/versions via docker hub.

— § —

Reply via email

#dev #kubernetes

★ skopeo: operate container images and registries

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 17 Dec 2024 17:49:31 -0300

♠ When working with docker and private image registries, a common workflow is to copy images from one private registry in the cloud to another. This can be done with skopeo.

This post includes some common recipes for it.

Usage 1) Default / Root to Staging #

From the default / root account registry to the staging registry:

skopeo sync \
  --src-creds "AWS:$(aws ecr get-login-password --region {region} --profile default)" \
  --dest-creds "AWS:$(aws ecr get-login-password --region {region} --profile staging)" \
  --override-os linux --override-arch amd64 \
  --src docker --dest docker \
  {account_id_root}.dkr.ecr.{region}.amazonaws.com/{org}/{repository}:{tag} \
  {account_id_staging}.dkr.ecr.{region}.amazonaws.com/{org}

Example values¹:

repository: argocd-gitops-tools
tag (version): 1.0.1
region: us-east-1

Usage 2) Public to MFA #

From a public registry to a private registry that uses MFA (multi-factor authentication).

First, it’s necessary to get MFA credentials. Source the script below (china.mfa.sh):

#!/usr/bin/env bash
# shellcheck disable=SC2155

export AWS_PROFILE=china

echo -n "Enter the MFA token code for your AWS China account: " && read -r token

mfa_arn="$(aws iam --profile "$AWS_PROFILE" get-user --output text --query User.Arn | sed 's/:user\//:mfa\//')"
credentials="$(aws --profile "$AWS_PROFILE" sts get-session-token --serial-number "$mfa_arn" --token-code "$token" --duration-seconds 86400)"

echo "Got credentials: $credentials"

export AWS_ACCESS_KEY_ID=$(echo "$credentials" | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo "$credentials" | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo "$credentials" | jq -r '.Credentials.SessionToken')

[[ -n "$AWS_SESSION_TOKEN" ] && echo "Success!"

Important: It is necessary to do source china-mfa.sh. Doing ./china-mfa.sh will not work.

Then we can proceed with the image sync:

skopeo sync \
  --dest-creds "AWS:$(aws ecr get-login-password --region cn-north-1)" \
  --override-os linux --override-arch amd64 \
  --src docker \
  --dest docker \
  quay.io/argoproj/argocd:v2.12.6 \
  {account_id_mfa}.dkr.ecr.cn-north-1.amazonaws.com.cn/quay.io/argoproj

Notes:

A public registry does not need authentication, hence there’s no --src-creds.
--dest-creds does not specify a --profile. Likewise, no AWS_PROFILE env var should be defined.
In this example, China (cn-north-1) is an AWS account with MFA enabled.

Potpourri #

It’s possible to pass --scoped to prefix images at destination using the full source image path as scope.
To sync multiple image architectures, pass -a or --all.

If syncing from/to Azure (ACR), use az acr login. Find the username in the Azure portal. For example:

az acr login -n {container registry name} --expose-token | jq -r '.accessToken' | skopeo login {registry} --password-stdin --username {username}

image = registry (includes the org name) + repository + tag ↩︎

— § —

Reply via email

#aws #bestof #dev #docker #kubernetes

Kubernetes: tail logs from pods with stern

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 17 Dec 2024 16:37:16 -0300

♠ You can always use kubectl logs -n {namespace} {pod} [-c {container}] -f to inspect logs from a specific pod¹.

Nonetheless that doesn’t scale when you don’t know which pod you want in the first place.

You could start with deployments, dive into replica sets, and then into individual pods, one by one, but… that is tedious and slow.

We can do better with stern:

⎈ Multi pod and container log tailing for Kubernetes

Usage:

stern pod-query [flags]

The pod-query is a regular expression or a Kubernetes resource in the form <resource>/<name>.

Example:

% stern -n argocd argocd

We can also grep for specific strings in log lines with --include:

% stern -n argocd argocd --include level=error

Or grep -v with --exclude.

It’s better to use --include / --exclude than to | grep, because it preserves color attributes (each pod has a different color in the aggregated log output).

The logs stream by default (like tail -f, or kubectl logs -f).

-c to specify a container within it, and -f to stream logs à la tail -f. ↩︎

— § —

Reply via email

#dev #kubernetes

Kubernetes: debugging services

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 16 Dec 2024 10:14:11 -0300

♠ I am deploying a new helm chart to our staging environments, and needed a way to ensure its dependencies in values.yaml are properly configured.

From the official kubernetes documentation:

An issue that comes up rather frequently for new installations of Kubernetes is that a Service is not working properly. You’ve run your Pods through a Deployment (or other workload controller) and created a Service, but you get no response when you try to access it. This document will hopefully help you to figure out what’s going wrong.

Run commands in a brand new pod #

% kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh

…quite verbose, but it works reliably, spawning a basic pod with busybox.

I prefer to have a package manager at my fingertips, hence alpine is arguably a better choice:

% kubectl run -it --rm --restart=Never busybox --image=alpine ash

You don’t need to remember that alpine uses ash by default, sh works as well. In fact:

/ # ls -al /bin | grep sh
lrwxrwxrwx    1 root     root            12 Dec  5 12:17 ash -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Dec  5 12:17 fdflush -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Dec  5 12:17 sh -> /bin/busybox

Then I could do, for example, apk update + apk add mtr nmap.

Alpine is marvellous for this because apk is very lightweight (and, hence, fast!) and the available packages from upstream are quite comprehensive.

Run commands in an already existing pod #

% kubectl exec <POD-NAME> -c <CONTAINER-NAME> -- <COMMAND>

…however you cannot choose a custom image this way. You’re stuck with whatever the pod is running.

The way around this is with the use of ephemeral debug containers. Newer versions of kubernetes have the kubectl debug command:

% kubectl debug -it <POD-NAME> --image=alpine [--target=<POD-NAME>]

Happy debugging!

— § —

Reply via email

#dev #kubernetes

Helm: JSON schema generation

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 14 Nov 2024 15:05:49 +0100

♠ Helm charts support the inclusion of a values.schema.json file to validate values.yaml. Documentation: https://helm.sh/docs/topics/charts/#schema-files

A JSON schema is akin to defining the structure of and type-annotating a JSON file. It helps to “shift left” the lifecycle of your helm releases.

For a pre-existing values.yaml file, it may be quite tedious to generate the schema from scratch, by hand.

Can we automate this process?

Of course we can.

One possibility is to use GenAI / LLMs. This is out of scope of this post.

Another possibility is to leverage a helm plug-in to do so.

I had a good experience with https://github.com/karuppiah7890/helm-schema-gen:

helm plugin install https://github.com/karuppiah7890/helm-schema-gen.git
cd path/to/helm/chart
helm schema-gen values.yaml | tee values.schema.json

Afterwards, test the generated schema with:

helm lint path/to/helm/chart

Usually I need to do some follow-up edits to the generated file, including:

add string enum types for a tighter validation of strings
mark certain fields as required
mark certain fields with additionalProperties: "false"
augment ["null"] to ["null", "string"] for keys that do not have a default value

It’s worth to observe that, at the time of this publication, the following notice is present in the repository of the plug-in:

This repository has been archived by the owner on Aug 31, 2021. It is now read-only.

Nonetheless, it still works well.

— § —

Reply via email

#dev #kubernetes

★ Helm: enforce the directory name matches the chart name

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 12 Nov 2024 11:13:06 +0100

♠ Problem statement: Given a helm chart called foo, enforce that its Chart.yaml file lives in a directory called foo¹.

Background #

In 2016, this used to be the default behavior in Helm:

fix(helm): produce error if package name is inconsistent

In 2018, this enforcement was removed:

remove dirname constraint on helm package

We would like to reintroduce this requirement in our Helm charts codebase, as a best practice, to prevent chart name collisions.

What would be the most native way to accomplish that?

I would probably write a git pre-commit hook if there is no native way (e.g. via some helm lint flag).

Solution #

Use the following script with pre-commit:

#!/usr/bin/env bash
#
# Check that the directory name matches the chart name in Chart.yaml.
#
# Examples:
#   - foo/Chart.yaml with "name: hey-foo" fails the check.
#   - foo/Chart.yaml with "name: foo" passes the check.
#
# Usage: $0 [path/to/chart/Chart.yaml ...]

for chart in "$@"; do
	dirname="$(basename "$(dirname "$chart")")"

	# Remove trailing slash.
	dirname="${dirname%/}"

	# Fetch chart name from Chart.yaml.
	chart_name="$(yq e '.name' "$chart")"

	if [[ $dirname != "$chart_name" ]]; then
		echo "error: directory name '${dirname}' does not match chart name '${chart_name}'"
		exit 1
	fi
done

Reference #

I asked and self-answered this question on Stack Overflow.

Why? For ease of management, simplicity, consistency & uniformity. ↩︎

— § —

Reply via email

#bestof #dev #kubernetes

kubectl: list all node pods

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 16 Oct 2024 12:26:24 +0200

♠ Recipe to list all pods that belong to a given node:

$ kubectl get pod -o wide --field-selector spec.nodeName={node_name} -A

Source: https://stackoverflow.com/questions/39231880/kubernetes-api-get-pods-on-specific-nodes

This ought to be easier to remember…

— § —

Reply via email

#dev #kubernetes

kubectl: force delete pods

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 15 Oct 2024 13:14:40 +0200

♠ Our installation of traefik via helm got stuck today in such a way that its pods did not terminate, even with successive kubectl delete.

--force did not work either :mildshock:

--grace-period=0 (alongside --force) did the trick:

kubectl delete pod traefik-{...} --grace-period=0 --force --namespace kube-system

Source: https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status

— § —

Reply via email

#dev #kubernetes

New APKBUILD: argocd

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 09 Oct 2024 23:03:32 +0200

♠ ArgoCD is a widely used GitOps software for Kubernetes Continuous Delivery (see USERS.md).

I am quite surprised no one bothered to create an Alpine Linux package for it.

Until…now, by yours truly.

This APKBUILD took a bit longer to create than the usual. There were a couple of issues with -buildmode=pie, addressed with export CGO_ENABLED=1 (via make CGO_FLAG=1).

Also, not every architecture is compatible with it. The following error message appears in ARM builds:

cannot use math.MaxInt64 (untyped int constant 9223372036854775807) as int value in argument to env.ParseNumFromEnv (overflows)

Anyway, once it is merged upstream, enjoy!

% doas apk add argocd

— § —

Reply via email

#alpine-linux #dev #kubernetes

helm: list recent chart upgrades

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 24 Sep 2024 14:21:20 +0200

♠ Problem statement: Given a kubernetes cluster with many helm charts in different namespaces, how to use helm to query the list of the last recently “touched” charts?

By “touched” we mean either installed (via helm install) or upgraded (via helm upgrade).

Something like: helm ls --all --sort-by updated. Using kubectl directly would also be OK.

Solution #

The brute force way:

helm ls --max 99999 -A -o json | jq -r '.[] | "\(.updated)\t\(.name)"' | sort | tail | column -t

-A for all namespaces
--max 99999 to “disable” paging (there’s no better way as of 2024-09-24)
jq to filter out on the “updated” field

Note that a sample json entry looks like the following:

{
  "name": "hoth-cb7f8a327",
  "namespace": "default",
  "revision": "2",
  "updated": "2024-06-14 15:02:16.775174131 +0000 UTC",
  "status": "deployed",
  "chart": "hoth-0.1.0",
  "app_version": "1.0.0"
}

Sample output of the aforementioned command:

2024-09-24  10:56:15.655674586  +0000  UTC  geonosis
2024-09-24  10:56:15.731630075  +0000  UTC  coruscant
2024-09-24  11:01:35.156973247  +0000  UTC  endor
2024-09-24  11:02:30.314014351  +0000  UTC  hoth

— § —

Reply via email

#dev #kubernetes

Helm: list all installed charts in the cluster

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 26 Aug 2024 17:57:07 +0200

♠ In theory this ought to be really easy:

$ helm ls -h
[...]
  -a, --all                  show all releases without any filter applied
  -A, --all-namespaces       list releases across all namespaces

So…one of these should work, right?

helm ls -a
helm ls -A

I have a sms-service running in the cluster in the default namespace. If I grep any of the previous commands to sms, it does not show up though.

Then…maybe helm ls -a -A? Nope.

Here’s what works:

$ helm ls -a --max 9999 | grep sms
sms-service                  	default  	356     	2024-08-26 14:09:50.705277885 +0000 UTC	deployed	sms-v0.1.0

Kinda ridiculous, eh?

 -m, --max int              maximum number of releases to fetch (default 256)

From https://helm.sh/docs/helm/helm_list/:

By default, up to 256 items may be returned. To limit this, use the ‘–max’ flag. Setting ‘–max’ to 0 will not return all results. Rather, it will return the server’s default, which may be much higher than 256. Pairing the ‘–max’ flag with the ‘–offset’ flag allows you to page through results.

Therefore I settle with this form:

$ helm ls -a -A --max 9999 | grep sms

Happy helming.

— § —

Reply via email

#dev #kubernetes

kubectl: print all secret values in plain text

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 06 Aug 2024 13:37:31 +0200

♠ Problem statement: Given a kubernetes secret with more than one key-value pair, print all of them in plain text (i.e. base64-decoded in this context).

Edit(2024-08-08): It turns out I made a mistake. The effect of the post below is akin to kubectl get [...] -o yaml. I’ll keep the post for bookkeeping purposes anyway.

Basically, something like:

kubectl cat secret -n infra-services my-cool-secret

Except that there is no kubectl cat, what a shame. It would be really great if we had it, for consistency with systemctl cat. Oh well.

There is kubectl edit though – amen, just like systemctl edit. This will open your $EDITOR. It’s often inconvenient to copy text to the clipboard from your terminal-based editor though, due to intricacies of different terminals, shells, terminal multiplexers, and OSC-52, therefore I don’t deem this as an acceptable solution.

There is a simple trick though:

EDITOR=cat kubectl edit secret -n infra-services my-cool-secret

…which works exactly as you would expect kubectl cat to behave. It displays a warning at the end:

Edit cancelled, no changes made.

…however it can be duly ignored.

If we didn’t have this trick, we could have:

written a bloated tool in golang to pipe to e.g. https://github.com/ashleyschuett/kubernetes-secret-decode (via)
used the kubectl edit trick above, with a decent terminal and OSC-52 setup
used jq with its map and base64d constructs (c.f. Stack Overflow)
used kubectl describe with -o jsonpath, specifying every single field, one by one (super tedious) c.f. this previous post

— § —

Reply via email

#dev #kubernetes