Argocd on ¬ just serendipity 🍀

ArgoCD: persistent PDB selector diff

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 22 May 2026 16:51:12 +0200

Problem statement: An ArgoCD application shows a persistent diff on a PodDisruptionBudget — spec.selector.matchExpressions is in the rendered manifest but missing from the live object. Sync doesn’t clear it. Hard refresh doesn’t clear it. Syncing with replace or with server-side apply or with force doesn’t clear it. Ugh!

policy/PodDisruptionBudget/default/my-super-cool-app:

selector:
  matchExpressions:
    - key: job-name
      operator: DoesNotExist
  matchLabels:
    release: my-super-cool-app

The chart was updated to add matchExpressions to the PDB selector. The diff shows up. Sync runs. Diff stays.

First idea: Kubernetes mutates PDB selectors server-side — no. The PDB registry strategy in pkg/registry/policy/poddisruptionbudget/strategy.go only touches Status and Generation. No selector mutation, no admission plugin in tree that injects this. Patterns like matchExpressions: [{key: job-name, operator: DoesNotExist}] come from chart authors, not from K8s. This is all according to the LLM.

Second idea: PDB selectors are immutable on older K8s — is also wrong here. Selector mutability landed in 1.27. The cluster is on 1.33+.

The actual culprit is client-side kubectl apply 3-way merge, which is what ArgoCD uses by default. The merge needs three inputs:

The previous applied state, read from metadata.annotations.kubectl.kubernetes.io/last-applied-configuration.
The live object.
The new desired manifest.

Check the annotation on the live PDB:

% kubectl get pdb my-super-cool-app -n default -o yaml | yq '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"'
{"apiVersion":"policy/v1","kind":"PodDisruptionBudget","metadata":{...},"spec":{"maxUnavailable":1,"selector":{"matchLabels":{"release":"my-super-cool-app"}}}}

No matchExpressions. So the 3-way merge sees: the previous apply didn’t have the field, the live object doesn’t have the field, the new manifest has the field. It treats this as an addition — but the existing implementation in this path doesn’t reliably propagate the new key into the live selector when the parent (spec.selector) is owned as an opaque value. The managedFields confirm it:

- manager: argocd-controller
  operation: Update
  fieldsV1:
    f:spec:
      f:maxUnavailable: {}
      f:selector: {}   # opaque — no sub-field ownership

f:selector: {} means argocd-controller owns the whole selector blob with no sub-field tracking. New fields under spec.selector get lost in the merge.

The fix is to delete the live PDB and let ArgoCD recreate it on the next sync:

% kubectl delete pdb my-super-cool-app -n default

This can alternatively be done via the ArgoCD Web UI.

It is a safe operation: a PDB only gates voluntary disruptions (drains, evictions). A brief gap doesn’t affect running pods. The recreated object gets a fresh last-applied-configuration containing matchExpressions, and the diff clears.

The longer-term fix is to switch the application to server-side apply, which tracks ownership at the field level and handles this correctly:

syncPolicy:
  syncOptions:
    - ServerSideApply=true

It’s always a dilemma whether or not to use server-side apply.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: stale Vault secret after population

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 13 May 2026 12:34:36 +0200

♠ Problem statement: An ArgoCD application references a Vault secret via helm-secrets / vals (e.g. vault://kv-v2/services/foo#/apiKey). The application is created before the secret exists in Vault, or while the corresponding key is blank. The key is populated later on. The application keeps behaving as if the value is still missing — usually a 404 from Vault, or a permission denied on the cached lookup:

Failed to load target state: failed to generate manifest for source 1 of 2:
rpc error: code = Unknown desc = Manifest generation error (cached): failed to
execute helm template command: [...] vals error: expand
vault://kv-v2/services/foo#/apiKey: Error making API request. URL: GET
http://vault.infra-services:8200/v1/sys/internal/ui/mounts/kv-v2/services/foo
Code: 404.

Note the (cached) part of the error. The argocd-repo-server caches the result of manifest generation, including failed vals / helm-secrets lookups. A normal sync or refresh happily reuses the cached (stale) failure and never re-resolves the Vault reference. So the freshly-populated secret goes unnoticed. A blank to non-blank update counts as well, besides outright token rotations.

The fix is to perform a Hard Refresh on the application — that’s what busts the manifest cache and forces a fresh Vault lookup:

argocd app get <app> --hard-refresh

Or, in the web UI: open the application → REFRESH ▾ dropdown → HARD REFRESH.

If a hard refresh alone doesn’t pick up the new value, restart the argocd repo server:

kubectl rollout restart deployment -n argocd argocd-repo-server

A regular refresh won’t do. The cache key for the failed lookup outlives the refresh.

By default, the application cache lasts for 24h (though it can be changed via helm values); hence, another way to resolve the issue is to simply sit out on it and wait.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: sync stuck in terminating

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 05 May 2026 13:51:10 -0200

♠ Problem statement: An ArgoCD application’s sync operation is stuck in Terminating for hours. The web UI shows operation is terminating due to timeout. A PreSync hook resource keeps appearing as Running even though the underlying Job already completed in the cluster.

% kubectl get application <app> -n argocd -o yaml | yq '.status.operationState.phase'
Terminating

The application controller is stuck in a busy loop:

"Resuming in-progress operation. phase: Terminating, message: operation is terminating due to timeout"
"No operation updates necessary to '<ns>/<app>'. Skipping patch"

In my case this followed a transient ImagePullBackOff on a PreSync hook job: the image was being mirrored to a regional ECR, the first hook attempt failed, and by the time the image landed and the retry succeeded, the parent operation had already moved into Terminating and never recovered.

The instinct is to clear the operation:

kubectl patch application <app> -n argocd --type merge -p '{"operation": null}'

It’s a no-op. The wedge doesn’t live in spec.operation (the request) — it lives in status.operationState (the controller’s bookkeeping). And status is owned by the controller, so a regular --type merge patch won’t touch it.

We need the status subresource:

kubectl patch application <app> -n argocd --type merge \
  -p '{"operation": null}'
kubectl patch application <app> -n argocd --subresource status --type merge \
  -p '{"status":{"operationState":null}}'

Both patches together. Neither touches cluster resources — they only modify ArgoCD’s view of the operation. After that, re-trigger the sync from the UI and the application reconciles cleanly.

Restarting argocd-application-controller does not help: the stuck state sits in etcd, and the freshly-elected leader picks it up on startup and resumes the same loop.

One caveat: verify the underlying hook resource has actually finished before clearing. kubectl describe job -n argocd <hook-job> should show 1 Succeeded. Otherwise we’ll lose in-flight progress.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: git index.lock issue

serendipity@perrotta.dev (Thiago Perrotta) — Sat, 23 Aug 2025 12:39:14 +0200

♠ Problem statement: Every ArgoCD application is stuck in Unknown state, with the following error:

Failed to load target state: failed to generate manifest for source 1 of 2:
rpc error: code = Unknown desc = failed to initialize repository resources: rpc
error: code = Internal desc = Failed to checkout FETCH_HEAD: failed to checkout
FETCH_HEAD: `git checkout --force FETCH_HEAD` failed exit status 128: fatal:
Unable to create '<path to cached source>/.git/index.lock': File exists. Another
git process seems to be running in this repository, e.g. an editor opened by
'git commit'. Please make sure all processes are terminated then try again. If
it still fails, a git process may have crashed in this repository earlier:
remove the file manually to continue.

It is a lock contention race condition issue within the ArgoCD repo server component, with the git index.lock.

The easiest way to address it is to restart the repo server:

kubectl rollout restart deploy -n argocd argocd-repo-server

Afterwards, refresh all applications in the cluster via the web UI:

click Applications on the left-side nav bar
click Refresh apps at the top
select ALL applications, set refresh type to NORMAL
click refresh

— § —

Reply via email

#argocd #dev #git #kubernetes

ArgoCD: app version notifications

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 12 Aug 2025 12:07:20 +0200

♠ Problem statement: when using ArgoCD notifications, starting off from their catalog, print the app chart version in addition to existing information.

More specifically, currently, a Slack notification message from Argo looks like the following:

Cluster 37: Application heart-of-gold is unhealthy.

We want to improve it to:

Cluster 37: Application heart-of-gold (~1.18.3) is unhealthy.

The following diff (under a template block) accomplishes that:

-          {{if eq .serviceType "slack"}}:interrobang:{{end}} Cluster <{{.context.argocdUrl}}|*{{.context.clusterId}}*>: Application <{{.context.argocdUrl}}/applications/{{.app.metadata.name}}|`{{.app.metadata.name}}`> _sync_ by {{.app.status.operationState.operation.initiatedBy.username | default "_automation_" }} is _unknown_.
+          {{if eq .serviceType "slack"}}:interrobang:{{end}} Cluster <{{.context.argocdUrl}}|*{{.context.clusterId}}*>: Application <{{.context.argocdUrl}}/applications/{{.app.metadata.name}}|`{{.app.metadata.name}}`>{{if .app.spec.source.targetRevision}} ({{.app.spec.source.targetRevision}}){{else}}{{range .app.spec.sources}}{{if .targetRevision}} ({{.targetRevision}}){{break}}{{end}}{{end}}{{end}} _sync_ by {{.app.status.operationState.operation.initiatedBy.username | default "_automation_" }} is _unknown_.

Diving into it with pretty-printing¹:

{{if .app.spec.source.targetRevision}}
  ({{.app.spec.source.targetRevision}})
{{else}}
  {{range .app.spec.sources}}
    {{if .targetRevision}}
      ({{.targetRevision}})
      {{break}}
    {{end}}
  {{end}}
{{end}}

If there’s a single source in the helm chart, we extract its targetRevision.

Otherwise, when using a multi-source app, we extract the first available targetRevision.

These are golang templates. ↩︎

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: recovery when traefik is broken

serendipity@perrotta.dev (Thiago Perrotta) — Sat, 09 Aug 2025 15:24:53 +0200

♠ It’s Friday. The best day to push to prod™. An internal scheduled upgrade for traefik goes awry. Internal services are no longer accessible. HTTP requests won’t be properly routed.

The most sensible SRE / DevOps will simply roll it back. Roll back first, investigate later: it’s crucial to minimize downtime.

How do we roll back? We’re using GitOps, the most sensible approach is to make a new git commit by submitting a pull request. However, this is an emergency situation. Can we roll back faster?

Sure, let’s make temporary changes directly in the cluster.

The traefik app is managed with ArgoCD. Let’s log in to the ArgoCD web UI and roll back to the previous version.

Oh, wait. We can’t. The ArgoCD web UI isn’t accessible, because traefik is broken. Sighs.

Let’s use the ArgoCD CLI then.

Aaaah, no. The ArgoCD CLI is really a REST client for the ArgoCD API. HTTP(S) is not accessible, so the CLI won’t work either.

What’s next? A plain helm upgrade should work. But helm is picky and will refuse to touch manifests it doesn’t own. The manifests are managed by Argo. helm won’t budge. Oh gosh.

Last option: the ArgoCD controllers are still running. We don’t need the ArgoCD HTTPS API to communicate with them. Let’s simply edit the k8s manifests directly.

First we patch the root app (app-of-apps pattern):

kubectl edit app -n argocd root

We remove the automated block, to effectively pause auto sync, otherwise our subsequent changes to traefik would be immediately reverted:

spec:
#  automated:
#    prune: [..]
#    selfHeal: [...]

You can comment or delete the automated: block, either is fine.

Next, we need to remove traefik from all application project (app project) sync windows, so that changes to it will be immediately applied:

kubectl edit appproject -n argocd cluster

The traefik entry got removed from all sync windows.

Now, finally, let’s instruct Argo to roll traefik back.

The broken version is 2.5.1, the previous working version was 2.4.1.

kubectl edit app -n argocd traefik

Update targetRevision: 2.4.1 under the helm block.

Wait a few seconds. Observe traefik pods as they are rolling restarted:

kubectl get deploy -n kube-system traefik -w

Finally, all services are once again accessible!

From this point on, we can go back to git and do a proper pull request revert to reconcile the codebase with the live cluster state.

This was an interesting exercise. I’m quite familiar with ArgoCD and GitOps at this point, and I instinctively knew what to do in this situation, without needing to consult the official documentation or even an LLM. I wouldn’t have minded doing either of these if needed, but the point is that it feels good to be well familiar with how a system works and how to apply your knowledge to fix day-to-day problems.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: all applications stuck on refresh

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 01 Aug 2025 14:30:58 +0200

♠ The problem is very likely the Application Controller.

You can restart it:

kubectl rollout restart sts -n argocd argocd-application-controller

Or inspect its logs:

kubectl logs -n argocd argocd-application-controller-0 --tail=100 | less

While we’re here, make sure there is at least one replica running:

% kubectl get sts -n argocd argocd-application-controller
NAME                            READY   AGE
argocd-application-controller   1/1     415d

Upstream issue.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: deployment spec.selector: field is immutable

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 30 Jun 2025 13:04:51 +0200

♠ When using ArgoCD to manage an app, the Deployment manifest needs to undergo significant changes, updating its selector labels. When trying to sync the argocd app, we get the following error:

one or more objects failed to apply, reason: error when patching "/dev/shm/3469418025": Deployment.apps "myapp" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:v1.LabelSelectorRequirement(nil)}: field is immutable. Retrying attempt #2 at 11:02AM.

This error doesn’t come directly from Argo, it is from kubectl apply.

There are two ways to address it:

Sync the argo app with the “force” option enabled
Run kubectl apply with --force (which is ultimately what the aforementioned option maps to).

In action (-w is short for --watch):

% kubectl get endpoints -w
NAME         ENDPOINTS         AGE
myapp        <none>            5y44d
myapp                          5y44d
myapp        10.1.13.40:8080   5y44d

— § —

Reply via email

#argocd #dev #kubernetes

Helm repository with a trailing slash

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 24 Jun 2025 11:32:50 +0200

♠ A repository URL with a trailing slash, and the same URL without a trailing slash, are treated as different repos. We found this out the hard way:

Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = error building helm chart dependencies: failed to add helm repository https://chartmuseum.{corp intranet}/orgs/{corp}/: failed to add repository: failed to get command args to log: `helm repo add https:--chartmuseum.{corp intranet}-orgs-{corp}- https://chartmuseum.{corp intranet}/orgs/{corp}/` failed exit status 1: Error: looks like "https://chartmuseum.{corp intranet}/orgs/{corp}/" is not a valid chart repository or cannot be reached: failed to fetch https://chartmuseum.{corp intranet}/orgs/{corp}/index.yaml : 401 Unauthorized

Let’s put aside the fact the error message REALLY likes to repeat the repo name.

This is a repository that we’ve already configured, with proper authentication credentials. Why would it start to fail now, all of a sudden?

You already got spoiled in the post title. Given Chart.yaml:

apiVersion: v2
name: corp-postgres-operator
description: Corp Postgres Operator
type: application
version: 1.1.13

dependencies:
  - name: postgres-operator
    repository: https://chartmuseum.{corp intranet}/orgs/archive

And also:

apiVersion: v2
name: corp-postgres-operator
description: Corp Postgres Operator
type: application
version: 1.1.13

dependencies:
  - name: postgres-operator
    repository: https://chartmuseum.{corp intranet}/orgs/archive/

These two dependencies end up being treated differently by Argo, even though they are effectively the same. It’s an upstream bug.

To work around it, I decided to simply remove all trailing slashes from helm repository dependencies. Canonicalize everything!

A git pre-commit.com hook is a good mechanism to do so:

repos:
  repo: local
  hooks:
    - id: helm-repo-trailing-slash
      name: Helm repositories must not contain trailing slashes in Chart.yaml
      files: Chart.yaml$
      language: pygrep
      entry: '\s*repository: [\w:/.]+/$'

Run pre-commit run -a helm-repo-trailing-slash, fix all violations, and then never think about this again.

Should the upstream bug ever be fixed, we can remove this workaround.

— § —

Reply via email

#argocd #dev #kubernetes #pre-commit

★ ArgoCD: every application in unknown state

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 13 Jun 2025 22:34:33 +0200

♠ Symptom: Every single application in the Kubernetes cluster that is managed by Argo is in Unknown sync status. A disaster! 😱

This has only happened in one of our clusters. More specifically, in an AKS cluster (from Microsoft Azure).

An observed effect:

% kubectl get --raw='/openapi/v2' | head -c 200
Error from server (ServiceUnavailable): the server is currently unable to handle the request

Aha! The Kubernetes OpenAPI server is not responding properly. That explains why Argo is having so much trouble to figure out the state of affairs in the cluster.

It was quite difficult to troubleshoot this issue. There are barely any resources about it in the open web. We filed an Azure Support ticket, but it was completely useless. They offered to restart¹ the OpenAPI server, but that didn’t help (of course it wouldn’t!).

Because this is a managed Kubernetes cluster, we have limited access to it, which includes lack of access to raw logs. The Azure portal doesn’t make it easy but, after digging quite deep into it, eventually we found this error message in the logs:

 {"pod":"kube-apiserver-7785d59f4f-scqmp","stream":"stderr","containerID":"925345f6501626aa6cd40dd187d5cf57837389eec9cb39b2292fb9ae3f66242f","log":"E0512 12:01:36.876768       1 handler.go:160] Error in OpenAPI handler: failed to build merge specs: unable to merge: duplicated path \/apis\/reports.kyverno.io\/v1\/clusterephemeralreports\n"}

That was the first clue! Now we know it’s related to Kyverno.

That error eventually led us to this issue in Kubernetes:

We have the same issue here, actually also triggered by installing the kyverno reports-server with both apiservices (different groups served by the same api-server).

Aha!

Deleting the duplicate CRDs installed by kyverno reports server has immediately resolved the OpenAPI server unresponsiveness. It wasn’t even necessary to restart it:

% kubectl delete crd ephemeralreports.reports.kyverno.io clusterephemeralreports.reports.kyverno.io clusterpolicyreports.wgpolicyk8s.io policyreports.wgpolicyk8s.io
customresourcedefinition.apiextensions.k8s.io "ephemeralreports.reports.kyverno.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterephemeralreports.reports.kyverno.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterpolicyreports.wgpolicyk8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "policyreports.wgpolicyk8s.io" deleted

And now:

$ kubectl get --raw='/openapi/v2' | head -c 200
{"swagger":"2.0","info":{"title":"Kubernetes","version":"v1.30.11"},"paths":{"/.well-known/openid-configuration/":{"get":{"description":"get service account issuer OpenID configuration, also known as

That was a tricky one! Interestingly: it has only occurred in AKS. EKS was fine.

Hello, IT. Have you tried turning it off and on again?. ↩︎

— § —

Reply via email

#argocd #bestof #dev #kubernetes

★ ArgoCD: custom health check for StatefulSet (OnDelete)

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 13 Jun 2025 21:40:01 +0200

♠ It turns out the built-in health check for StatefulSets in ArgoCD is not comprehensive enough.

It covers RollingUpdate well enough, but OnDelete is incomplete.

More specifically:

Observed: whenever you push an update to an STS whose .spec.updateStrategy is OnDelete, its health status stays as Healthy.

Expected: there should be an indication that the STS pods are not up-to-date, and hence need to be restarted to incorporate the latest changes.

There is an open bug: Statefulset healthcheck does not work #6527 for it.

I managed to work around it by defining my own health check. This is an use case where Gen AI is helpful, though it does not shine. I could not have Gen AI produce a fully working health check for this scenario. Instead, I used it to bootstrap a basic example, then had to tweak it until it worked. This has been tested multiple times, I am reasonably confident it works well (some edge cases could be missing though).

health_status = {}

if obj.status == nil then
  health_status.status = "Progressing"
  health_status.message = "Waiting for status to be reported"
  return health_status
end

local strategy = obj.spec.updateStrategy and obj.spec.updateStrategy.type or "RollingUpdate"
local spec_replicas = obj.spec.replicas or 1
local ready_replicas = obj.status.readyReplicas or 0
local updated_replicas = obj.status.updatedReplicas or 0
local status_replicas = obj.status.replicas or 0
local current_revision = obj.status.currentRevision or ""
local update_revision = obj.status.updateRevision or ""

if strategy == "OnDelete" then
  if ready_replicas == status_replicas and updated_replicas == spec_replicas then
    health_status.status = "Healthy"
    health_status.message = "All replicas are ready and updated (OnDelete strategy)"
  elseif updated_replicas == 0 then
    health_status.status = "Degraded"
    health_status.message = "No replicas have been updated (OnDelete strategy): 0/" .. spec_replicas .. " updated. Still running an old revision!"
  else
    health_status.status = "Progressing"
    health_status.message = "Progressing (OnDelete): " .. ready_replicas .. "/" .. status_replicas .. " ready, " .. updated_replicas .. "/" .. spec_replicas .. " updated"
  end
else
  if ready_replicas == status_replicas and updated_replicas == spec_replicas then
    if current_revision ~= update_revision then
      health_status.status = "Progressing"
      health_status.message = "Waiting for rollout to complete: revision mismatch (" .. current_revision .. " ≠ " .. update_revision .. ")"
    else
      health_status.status = "Healthy"
      health_status.message = "All replicas updated, ready, and revisions match"
    end
  else
    health_status.status = "Progressing"
    health_status.message = "RollingUpdate progressing: " .. ready_replicas .. "/" .. status_replicas .. " ready, " .. updated_replicas .. "/" .. spec_replicas .. " updated"
  end
end
return health_status

It is necessary to distinguish between OnDelete and RollingUpdate.

OnDelete should not use currentRevision / update_revision, as these values don’t mean anything for that update strategy. This can be observed upstream as well.

It is easy to write / tweak the health check when you look at the live STS resource / manifest in the cluster. obj refers to the STS manifest (kubectl get sts -o yaml). The most relevant field is status:.

If you deploy ArgoCD via its helm chart, add that check for argocd-cm. The values.yaml file:

argo-cd:
  configs:
    cm:
      resource.customizations.health.apps_StatefulSet: |
        health_status = {}
        [...]

This custom health check was partly inspired by a Prometheus alert we have:

alert: StatefulSetUpdateNotRolledOut
expr: max without (revision) (kube_statefulset_status_current_revision unless kube_statefulset_status_update_revision) * (kube_statefulset_replicas != kube_statefulset_status_replicas_updated)

How long will it take until LLMs ~~steal~~ adopt this novel solution from me?

— § —

Reply via email

#argocd #bestof #dev #kubernetes

ArgoCD: application stuck in unknown

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 12 Jun 2025 11:51:32 +0200

♠ Problem statement: An application in ArgoCD that was previously running smoothly has suddenly started to fail. Its status has changed to “Unknown”. The “Refresh” button is grayed out in the Argo web UI.

I found the following works in this scenario, which is typically a transient issue:

restart the argo server
restart the argo repo server

In terms of commands:

% kubectl rollout restart deployment argocd-server -n argocd
% kubectl rollout restart deployment argocd-repo-server -n argocd

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: upgrade to v3, remove helm labels

serendipity@perrotta.dev (Thiago Perrotta) — Mon, 02 Jun 2025 13:03:24 +0200

♠ I am leading the effort to upgrade ArgoCD to its newest major version (v3) across our fleet.

One of its breaking changes is the removal of helm labels from argo applications, like such:

metadata:
# [...]
  labels:
    app.kubernetes.io/managed-by: Helm

Applying a normal sync does not get rid of the label.

I found out that doing an one-off server-side apply sync addresses the label removal, and it is safe to do (incurs no downtime):

However, there are some cases where you want to use kubectl apply --server-side over kubectl apply:

[…]

Patching of existing resources on the cluster that are not fully managed by Argo CD.

There’s no need to make the server-side apply a permanent config, i.e.:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

…though there’s no harm either in doing so.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: vertical-pod-autoscaler sync loop

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 13 Feb 2025 13:17:36 +0100

♠ We’ve had a long time issue wherein our vertical-pod-autoscaler (VPA) ArgoCD application, deployed via its helm chart, got stuck in a sync loop.

More specifically: the helm chart automatically generates a self-signed certificate via the GenCA helm function upon every change to our gitops repository. It turns out this is documented and we can see its corresponding manifest in the chart template.

So that’s where the sync loop was coming from! ArgoCD was constantly reporting (and resolving) a diff in the admission controller secret.

The manifest:

apiVersion: v1
data:
  ca.crt: ++++++++
  tls.crt: ++++++++
  tls.key: ++++++++
kind: Secret
metadata:
[...]
  name: vertical-pod-autoscaler-admission-controller-tls
  namespace: kube-system
[...]

The problematic are all the ones under data:.

I confirmed that was indeed the case in the CNCF Slack #argo-cd. Thanks Tim!

How to address it?

We leverage ignoreDifferences (c.f. Stack Overflow) in the ArgoCD app manifest:

% git diff HEAD~2..HEAD
diff --git apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml
index 92bfbcc..90d8435 100644
--- apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml
+++ apps/base/vertical-pod-autoscaler/vertical-pod-autoscaler.yaml
@@ -34,3 +34,15 @@ spec:
         maxDuration: 1m
     syncOptions:
       - CreateNamespace=true
+      - RespectIgnoreDifferences=true
+  ignoreDifferences:
+    - group: ""
+      kind: Secret
+      name: vertical-pod-autoscaler-admission-controller-tls
+      jsonPointers:
+        - /data
+    - group: apps
+      kind: Deployment
+      name: vertical-pod-autoscaler-admission-controller
+      jsonPointers:
+        - /spec/template/metadata/annotations/checksum~1tls-secret

In JSON Pointers (RFC 6901), one escapes the slash character with a ~1 (c.f. section 3 in the spec):

Because the characters '~' (%x7E) and '/' (%x2F) have special meanings in
JSON Pointer, '~' needs to be encoded as '~0' and '/' needs to be encoded as
'~1' when these characters appear in a reference token.

Problem resolved!

In general all helm charts that generate self-signed certificates (with genCA) may experience this sync loop when used with a CD framework such as Argo.

— § —

Reply via email

#argocd #dev #kubernetes

ArgoCD: cache: key is missing

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 09 Jan 2025 20:20:03 -0300

♠ If you experience the title error message in ArgoCD (e.g. via its web UI and/or in pod logs), it’s related to argo-cd#5068. The full error message is:

Unable to load data: cache: key is missing

As I commented in the bug, here’s how to fix it:

The following workaround sufficed for me: kubectl delete pod -n argocd argocd-application-controller-0.

Initially I ran kubectl rollout restart deployment -n argocd argocd-application-controller but it didn’t work.

In my experience, this error only happens once, ephemerally, during ArgoCD bootstrapping. Once fixed it does not reoccur.

Update(2025-06-06): This error reoccurs intermittently 🤷.

Another potential workaround is to disable redis / caching altogether, as ArgoCD can fully operate without it, but I wouldn’t recommend that.

New APKBUILD: argocd

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 09 Oct 2024 23:03:32 +0200

♠ ArgoCD is a widely used GitOps software for Kubernetes Continuous Delivery (see USERS.md).

I am quite surprised no one bothered to create an Alpine Linux package for it.

Until…now, by yours truly.

This APKBUILD took a bit longer to create than the usual. There were a couple of issues with -buildmode=pie, addressed with export CGO_ENABLED=1 (via make CGO_FLAG=1).

Also, not every architecture is compatible with it. The following error message appears in ARM builds:

cannot use math.MaxInt64 (untyped int constant 9223372036854775807) as int value in argument to env.ParseNumFromEnv (overflows)

Anyway, once it is merged upstream, enjoy!

% doas apk add argocd

— § —

Reply via email

#alpine-linux #argocd #dev #kubernetes