Aws on ¬ just serendipity 🍀

Terraformer

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 19 Nov 2025 10:22:18 -0300

CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code

In other words: “reverse terraform”.

I was quite excited to add this project to my tool belt, as it can save a lot of time with scaffolding and terraform import commands.

Then Claude Code, once again, surprised me.

It turns out there’s absolutely no need to adopt terraformer if you have a superb agent available and a few cents/dollars to spare.

Claude can figure out which CLI arguments to pass to aws, az, etc. to list all relevant cloud resources. Then it can generate / scaffold a basic set of Terraform files to manage them¹. And then it can craft an one-off shell or python script to import them all in terraform.

Easy. Quick. Painless.

The interesting part is that I had initially asked Claude to explicitly use terraformer to do so, but it turned out to be more complicated / less efficient than having the agent perform the task directly.

I am sorry if you were expecting a terraformer tutorial.

You’ll need to refactor these later for maintainability. ↩︎

— § —

Reply via email

#aws #dev #terraform

New script: AWS China MFA login

serendipity@perrotta.dev (Thiago Perrotta) — Sat, 04 Oct 2025 11:34:58 +0200

♠ Previously.

aws_china_mfa:

A script to authenticate to AWS China using MFA and export temporary session credentials.

The idea is simple. To log into AWS, we need the following environment variables to be defined:

export AWS_PROFILE='china'
export AWS_ACCESS_KEY_ID='AKIAWHEATLICIOUSPANCAK'
export AWS_SECRET_ACCESS_KEY='wSyrupyDeliciousSecretKeyForBreakfastDelight42'
export AWS_SESSION_TOKEN='FwoGZXIvYXdzEBaaDCakesYrUpSWeetToKenArEDelIcIoUs...'

AWS_PROFILE is static and can be hard-coded. It’s also possible to supply it to the aws CLI via --profile.

The other three are fetched upon a successful authentication against AWS, using MFA (multi-factor authentication) with an OTP.

The usage experience of the script is like this:

% ./aws_china_mfa.sh
Note: Script is being executed. To apply credentials, run:
  eval "$(./aws_china_mfa.sh china)"

Using AWS profile: china
Enter the MFA token code for your AWS China account: 123456
Retrieving MFA device ARN...
Requesting session token...

  export AWS_PROFILE='china'
  export AWS_ACCESS_KEY_ID='AKIAWHEATLICIOUSPANCAK'
  export AWS_SECRET_ACCESS_KEY='wSyrupyDeliciousSecretKeyForBreakfastDelight42'
  export AWS_SESSION_TOKEN='FwoGZXIvYXdzEBaaDCakesYrUpSWeetToKenArEDelIcIoUs...'

✓ Successfully authenticated to AWS China

Copy and paste the export commands above to apply credentials.

The script prompts the user for an OTP; that’s the only input.

There are three ways to execute it:

source it: the variables are then exported directly
eval it: ditto
execute it: in this mode it’s not possible to export the variables. Instead, they are simply printed to stdout, and there’s a message instructing the user to manually copy and paste the exports

Later on I added integration with 1Password, via --op-account and --op-item, so that the OTP can be retrieved directly from it, without any user intervention or interruption – besides using biometrics (touch ID) to confirm the operation.

— § —

Reply via email

#aws #dev #security

Terraform: rename resources

serendipity@perrotta.dev (Thiago Perrotta) — Wed, 19 Mar 2025 11:40:43 +0100

♠ Sometimes I need to make a simple refactoring rename change to a terraform resource that has already been applied. For example:

resource "aws_api_gateway_request_validator" "delete" {
  name                        = "Validate body, query string parameters, and headers"
  rest_api_id                 = aws_api_gateway_rest_api.this.id
  validate_request_body       = true
  validate_request_parameters = true
}

resource "aws_api_gateway_request_validator" "this" {
  name                        = "Validate body, query string parameters, and headers"
  rest_api_id                 = aws_api_gateway_rest_api.this.id
  validate_request_body       = true
  validate_request_parameters = true
}

The simplest way to do so is to terraform apply. However, that incurs destruction + re-creation of the resource, which would incur downtime in case it’s already deployed in prod.

The best way is % terraform state mv aws_api_gateway_request_validator.{delete,this}. In-place move, simple and easy.

When doing it at scale, running multiple terraform state mv commands could be cumbersome.

There’s a second way, using the moved block in HCL:

moved {
    from = aws_api_gateway_request_validator.delete
    to = aws_api_gateway_request_validator.this
}

Note: terraform docs call this operation “move”, instead of “rename”.

— § —

Reply via email

#aws #dev #terraform

★ skopeo: operate container images and registries

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 17 Dec 2024 17:49:31 -0300

♠ When working with docker and private image registries, a common workflow is to copy images from one private registry in the cloud to another. This can be done with skopeo.

This post includes some common recipes for it.

Usage 1) Default / Root to Staging #

From the default / root account registry to the staging registry:

skopeo sync \
  --src-creds "AWS:$(aws ecr get-login-password --region {region} --profile default)" \
  --dest-creds "AWS:$(aws ecr get-login-password --region {region} --profile staging)" \
  --override-os linux --override-arch amd64 \
  --src docker --dest docker \
  {account_id_root}.dkr.ecr.{region}.amazonaws.com/{org}/{repository}:{tag} \
  {account_id_staging}.dkr.ecr.{region}.amazonaws.com/{org}

Example values¹:

repository: argocd-gitops-tools
tag (version): 1.0.1
region: us-east-1

Usage 2) Public to MFA #

From a public registry to a private registry that uses MFA (multi-factor authentication).

First, it’s necessary to get MFA credentials. Source the script below (china.mfa.sh):

#!/usr/bin/env bash
# shellcheck disable=SC2155

export AWS_PROFILE=china

echo -n "Enter the MFA token code for your AWS China account: " && read -r token

mfa_arn="$(aws iam --profile "$AWS_PROFILE" get-user --output text --query User.Arn | sed 's/:user\//:mfa\//')"
credentials="$(aws --profile "$AWS_PROFILE" sts get-session-token --serial-number "$mfa_arn" --token-code "$token" --duration-seconds 86400)"

echo "Got credentials: $credentials"

export AWS_ACCESS_KEY_ID=$(echo "$credentials" | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo "$credentials" | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo "$credentials" | jq -r '.Credentials.SessionToken')

[[ -n "$AWS_SESSION_TOKEN" ] && echo "Success!"

Important: It is necessary to do source china-mfa.sh. Doing ./china-mfa.sh will not work.

Then we can proceed with the image sync:

skopeo sync \
  --dest-creds "AWS:$(aws ecr get-login-password --region cn-north-1)" \
  --override-os linux --override-arch amd64 \
  --src docker \
  --dest docker \
  quay.io/argoproj/argocd:v2.12.6 \
  {account_id_mfa}.dkr.ecr.cn-north-1.amazonaws.com.cn/quay.io/argoproj

Notes:

A public registry does not need authentication, hence there’s no --src-creds.
--dest-creds does not specify a --profile. Likewise, no AWS_PROFILE env var should be defined.
In this example, China (cn-north-1) is an AWS account with MFA enabled.

Potpourri #

It’s possible to pass --scoped to prefix images at destination using the full source image path as scope.
To sync multiple image architectures, pass -a or --all.

If syncing from/to Azure (ACR), use az acr login. Find the username in the Azure portal. For example:

az acr login -n {container registry name} --expose-token | jq -r '.accessToken' | skopeo login {registry} --password-stdin --username {username}

image = registry (includes the org name) + repository + tag ↩︎

— § —

Reply via email

#aws #bestof #dev #docker #kubernetes

Docker build: ignore cache

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 10 Dec 2024 19:45:42 -0300

♠ To build a docker image completely from scratch, without reusing cache layers on your system:

docker build --no-cache -f Dockerfile . -t {image name}:{image tag} --platform
linux/amd64

Pass an explicit platform in case e.g. you’re building on a Mac M1 (arm64) with the intent of running it on Linux in the cloud (amd64 typically).
The docker image full specification is: image = {registry}/{repository}:{tag}. For example, the registry could be an AWS ECR URL, whereas the repository is something like “bitnami/sealed-secrets”, and the tag typically follows semantic versioning.

— § —

Reply via email

#aws #dev #docker

Check if docker image exists in registry

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 25 Oct 2024 14:11:14 +0200

♠ Given a docker registry, image and tag, run the following command:

% docker manifest inspect 9876543210.foo.ecr.us-east-1.amazonaws.com/org/image:123456789

If it does not exist, you’ll see:

no such manifest: 9876543210.foo.ecr.us-east-1.amazonaws.com/org/image:123456789

Ensure you are authenticated and/or connected to the VPN in case of a private registry.

When using ECR (AWS’s registry), you can authenticate this way:

% which docker_login
docker_login () {
	aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 9876543210.foo.ecr.us-east-1.amazonaws.com
}

— § —

Reply via email

#aws #dev #docker

★ prodaccess

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 18 Oct 2024 14:57:27 +0200

♠ At Google we had a prodaccess command used to log in and authenticate in all sorts of internal corp systems¹ for 20 hours. It became a daily ritual to run prodaccess first thing in the morning when logging in upon unlocking our physical workstations².

During the pandemic, when heavily using our virtual workstations (dubbed cloudtops, think of an EC2 instance or a VPS) it was common to hook prodaccess into ssh. There was a prodcertstatus command that would tell you when prodaccess would expire. A common pattern was to modify your ssh startup command to automatically run prodaccess whenever prodcertstatus indicated that access would expire within 2-4 hours, depending on the user’s preference.

Given my trained muscle memory, I wanted to continue to use this alias even in environments other than Google:

[...]
prodaccess() {
	echo "--> VPN"
	vpn_login

	echo "--> Teleport"
	teleport_login

	echo "--> AWS"
	aws_login

    echo "--> Azure"
    azure_login

	case "$1" in
		-u|--update)
			echo "--> mr"
			# Update all well-known corp repositories
			(cd ~/Corp && mr update && mr run git world)

			# Package manager upgrade
			echo "--> sd-world"
			sd-world

			shift
			;;

		-a|--all)
			echo "--> Atlas Mongo"
			atlas_login

			shift
			;;
	esac
}
[...]

Notes:

mr is https://myrepos.branchable.com/ (“a tool to manage all your version control repositories”)³
sd-world was introduced in a previous post: it performs a full system upgrade
--all holds services I only need / use from time to time.
shift is not really needed, but IMHO it’s a good practice (and another ingrained habit of mine), especially when inside for loops
git world is an alias in my ~/.gitconfig that runs git fetch --all && git remote prune origin

Every morning I invoke prodaccess -u when starting my work day. A few password prompts and touch IDs later and I do not need to worry about logging in anymore during the middle of the day, which reduces overall context switching and stress.

Until someone in CorpEng decided it was a good idea to deprecate it in favour of a new gcert command, destroying many SWE-years of developed muscle memory. ↩︎
Ah, the good ol’ times. ↩︎
I should write a post about it sometime. ↩︎

— § —

Reply via email

#aws #bestof #dev

AWS S3: search by suffix

serendipity@perrotta.dev (Thiago Perrotta) — Thu, 05 Sep 2024 17:09:38 +0200

♠ Given an AWS S3 bucket, how to recursively list all objects within it that match a given suffix?

The following example searches for all objects that end with thiagowfx.

aws s3 rm --profile {aws_profile} s3://example.com/my/path/ --recursive --dryrun --exclude '*' --include "*thiagowfx"

This is a hack with a dry-run deletion operation.

Alternatively, use aws s3 ls¹ plus grep / awk / sed:

aws s3 ls --profile {aws_profile} s3://example.com/my/path | awk -F ' ' '{print $4}' | grep 'thiagowfx$'

Why would you use rm in lieu of ls? Mostly because of the built-in --include / --exclude options. If you happen to have millions of objects in your S3 bucket, then you do not need to list them all.

https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html ↩︎

— § —

Reply via email

#aws #dev

Terraform: AWS deployment to random availability zones

serendipity@perrotta.dev (Thiago Perrotta) — Tue, 21 May 2024 14:31:03 +0200

♠ A common scenario: there’s a new deployment you would like to roll out to AWS. Let’s say you pick “us-east-1” as your cloud region. There are multiple availability zones within it:

us-east-1a
us-east-1b
us-east-1c
us-east-1d
us-east-1e
us-east-1f

Suppose you want to pick two of them for your service/app, and you don’t particularly care about which one. How to proceed?

Option #1: Hard-coding #

Pick two arbitrary zones and hard-code them.

variable "availability_zones" {
  type    = list(string)
  default = ["us-east-1a", "us-east-1b"]
}

resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.chartmuseum.id
  cidr_block        = element(var.private_subnets, count.index)
  availability_zone = element(var.availability_zones, count.index)
  count             = length(var.private_subnets)
}

Caveat: The paradox of choice, unnecessary decision fatigue.

Option #2: Pick the first two #

Use the AWS data source to dynamically find all zones, and pick the first two.

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.chartmuseum.id
  cidr_block        = element(var.private_subnets, count.index)
  availability_zone = element(data.aws_availability_zones.available.names, count.index)
  count             = length(var.private_subnets)
}

Note that terraform plan should display the full zone list.

Caveat: Heavily biased towards the first two zones.

Option #3: Random shuffling #

Pick two zones at random!

data "aws_availability_zones" "available" {
  state = "available"
}

resource "random_shuffle" "aws_availability_zone_names" {
  input        = data.aws_availability_zones.available.names
  result_count = 2
}

resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.chartmuseum.id
  cidr_block        = element(var.private_subnets, count.index)
  availability_zone = element(random_shuffle.aws_availability_zone_names.result, count.index)
  count             = length(var.private_subnets)
}

Winner: In my opinion, this is the most elegant approach.

random_shuffle will output the selected regions upon running terraform apply.

— § —

Reply via email

#aws #dev #terraform

Adding a healthcheck to chartmuseum in AWS Fargate

serendipity@perrotta.dev (Thiago Perrotta) — Fri, 17 May 2024 11:15:09 +0200

♠ Assume that you have a Chartmuseum container running in AWS Fargate.

Chartmuseum is a repository for helm charts. AWS Fargate is an Amazon service to run containers (“serverless”), being part of ECS (Elastic Container Service).

Problem statement: Add a container healthcheck to the chartmuseum task definition associated with the container.

The official docs suggest using curl:

["CMD-SHELL", "curl -f http://localhost/ || exit 1"]

For Chartmuseum specifically we’re interested in its /health endpoint, as per this reference:

["CMD-SHELL", "curl -f http://localhost/health || exit 1"]

But we’re using port 8080:

["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]

If you use this healthcheck for the official chartmuseum image (ghcr.io/helm/chartmuseum) it will fail, because the Alpine Linux environment it uses does not contain curl.

A straightforward fix is to use wget instead:

["CMD-SHELL", "wget -q --spider http://localhost:8080/health || exit 1"]

--spider is needed because we do not want to download anything, -q is optional and short for “quiet”.

The /health endpoint merely returns a simple JSON:

{"healthy":true}

References: https://stackoverflow.com/questions/47722898/how-can-i-make-a-docker-healthcheck-with-wget-instead-of-curl

— § —

Reply via email

#aws #dev #docker #kubernetes