Setup guide

Simplest way to enable the whole read/inventory half of the skill: describing instances, AMIs, launch templates, ASGs, ECS services and task definitions, Lambda configuration and aliases, log groups and CloudWatch metrics.

↳ Prefer composing the narrower AWS managed read policies instead: AmazonEC2ReadOnlyAccess, AmazonECS_FullAccess is NOT read-only so use a custom ecs:Describe*/ecs:List* policy, AWSLambda_ReadOnlyAccess, CloudWatchLogsReadOnlyAccess and ComputeOptimizerReadOnlyAccess.

Inventory instances, resolve Name tags, read instance type/state/AMI/AZ, and print the blast radius before any destructive call.

↳ Describe actions do not support resource-level permissions; scope with the ec2:Region condition key and, where supported, aws:ResourceTag on the mutating actions instead.

Launch EC2 instances from a resolved AMI with IMDSv2 required and encrypted gp3 root volumes.

↳ Condition on ec2:InstanceType, ec2:Subnet and aws:RequestTag/Environment, and require ec2:MetadataHttpTokens = required so IMDSv1 instances cannot be launched at all.

Decommission instances after an explicit, resource-naming confirmation; always exercised with DryRun first.

↳ Restrict to instances carrying a specific tag using a Condition on aws:ResourceTag/Environment, and exclude production entirely from the automation principal.

Bake tagged AMIs from a running instance before an ASG roll, so a rollback image always exists.

↳ Pair with ec2:CreateTags restricted via aws:RequestTag, and keep ec2:DeregisterImage on a separate, more tightly held policy.

Roll a new launch template version through an Auto Scaling group with MinHealthyPercentage and AutoRollback set.

↳ Scope the Resource element to the specific ASG ARNs, and grant autoscaling:RollbackInstanceRefresh alongside it so a bad roll can always be reverted.

Point a service at a new task definition revision, force a new deployment, and roll back to the previously recorded revision.

↳ Scope Resource to arn:aws:ecs:<region>:<account>:service/<cluster>/<service> and add a Condition on ecs:cluster so a single cluster is reachable.

Register a new revision with only the container image changed, keeping every other field identical to the running revision.

↳ RegisterTaskDefinition does not support resource-level permissions, so pair it with a tightly scoped iam:PassRole and rely on ecs:UpdateService scoping to bound the impact.

Required to attach an instance profile in RunInstances and task/execution roles in RegisterTaskDefinition; without it both calls fail with AccessDeniedException.

↳ Always enumerate the exact role ARNs in Resource and add Condition {"StringEquals": {"iam:PassedToService": ["ec2.amazonaws.com", "ecs-tasks.amazonaws.com"]}}. A wildcard PassRole is a privilege-escalation path to any role in the account.

Ship a new deployment package and publish it as an immutable numbered version.

↳ Scope Resource to arn:aws:lambda:<region>:<account>:function:<name>; grant lambda:PublishVersion on the same resource so every deploy is versioned.

Shift traffic with AdditionalVersionWeights for a canary, promote to 100%, and roll back instantly by restoring the previous version.

↳ Scope Resource to the alias ARN (arn:aws:lambda:<region>:<account>:function:<name>:live) rather than the function, so only the intended alias can be repointed.

Tail and query ECS and Lambda log groups and run CloudWatch Logs Insights queries to verify a deploy actually works.

↳ Replace with an inline policy granting logs:FilterLogEvents, logs:GetLogEvents, logs:StartQuery and logs:GetQueryResults on arn:aws:logs:<region>:<account>:log-group:/aws/lambda/<name>:* and /ecs/<service>:*.

Fetch EC2, Lambda and ECS-on-Fargate right-sizing recommendations to back a resize proposal with data rather than intuition.

↳ Grant only the three GetEC2InstanceRecommendations / GetLambdaFunctionRecommendations / GetECSServiceRecommendations actions if the account also uses other Compute Optimizer surfaces.

Attached to the EC2 instance role (not the operator) so Session Manager can replace SSH for debugging without opening port 22.

↳ This goes on the instance profile. The operator separately needs ssm:StartSession scoped to instances carrying a specific tag.

Cause: Not a failure. This is the success signal for a `--dry-run` call: the parameters are valid and the principal is authorised.

Fix: Treat it as the green light. Re-run the identical command without `--dry-run` only after explicit confirmation naming the instance IDs and the account.

Cause: The principal lacks ec2:RunInstances, or a Condition (tag, instance type, subnet, region) or an SCP denies it. EC2 deliberately does not say which.

Fix: Re-run with `--dry-run` and decode the message: `aws sts decode-authorization-message --encoded-message <blob>` (needs sts:DecodeAuthorizationMessage). Then confirm with `aws iam simulate-principal-policy --action-names ec2:RunInstances`.

Cause: The task definition names an execution or task role that the deploying principal is not allowed to pass.

Fix: Add iam:PassRole to the deployer policy with Resource set to that exact role ARN and Condition {"StringEquals": {"iam:PassedToService": "ecs-tasks.amazonaws.com"}}.

Cause: Temporary STS or IAM Identity Center credentials aged out mid-session; typical for 1-hour assumed roles and SSO sessions.

Fix: Re-run `aws sso login --profile <name>`, or re-issue `aws sts assume-role` and re-export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. For long deploys, prefer a profile with role_arn so the CLI refreshes automatically.

Cause: The access key was deleted or deactivated, AWS_SESSION_TOKEN is missing for an ASIA-prefixed temporary key, or the credentials belong to a different partition (GovCloud/China).

Fix: Check `aws configure list` for where each value is resolved from. If the key ID starts with ASIA you must also export AWS_SESSION_TOKEN; if it starts with AKIA, confirm the key is still Active in IAM.

Cause: executionRoleArn was dropped from the task definition JSON — usually by rebuilding it by hand instead of round-tripping describe-task-definition.

Fix: Regenerate the JSON from `aws ecs describe-task-definition --query taskDefinition`, delete only the read-only fields, and change only the image. Never author the blob from scratch.

Cause: Tasks are failing to reach a steady state — image pull failure, failing container health check, target group health check failing, or no capacity to place a replacement task.

Fix: Read `aws ecs describe-services --query 'services[0].events[:10].message'` and `aws ecs describe-tasks --query 'tasks[].stoppedReason'`. Check maximumPercent gives placement headroom, then roll back to the recorded previous task definition.

Cause: A previous update-function-code or update-function-configuration has not finished; Lambda serialises updates per function.

Fix: Wait for the function to settle with `aws lambda wait function-updated-v2 --function-name my-fn` (or poll LastUpdateStatus until Successful) before issuing the next call. Never retry in a tight loop.

Cause: An earlier refresh on the same Auto Scaling group is still running or was never cancelled.

Fix: Inspect it with `aws autoscaling describe-instance-refreshes --auto-scaling-group-name <asg>`; wait for Successful, or stop it with `aws autoscaling cancel-instance-refresh --auto-scaling-group-name <asg>` before starting a new one.

Cause: The per-family On-Demand vCPU service quota for the region is exhausted.

Fix: Check the quota with `aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A`, request an increase, pick a smaller instance type, or launch into a different region or family.

Cause: AWS has no free capacity for that instance type in that AZ right now. It is a capacity condition, not a permission or quota problem.

Fix: Retry in another AZ or subnet, choose a different instance type or size, or use an ASG with multiple instance types and subnets so it can place elsewhere automatically.

Cause: No profile, environment variables, or instance/container role was resolvable — often AWS_PROFILE points at a profile that does not exist, or the shell lost the exported variables.

Fix: Run `aws configure list` to see which source each value comes from, confirm ~/.aws/config contains the profile, and re-export AWS_PROFILE or the key variables. In CI, confirm the OIDC role assumption step actually ran.