Setup guide

Single-attachment grant that covers every read call this skill makes across CloudWatch, Logs, X-Ray, CloudTrail and the resource APIs used to resolve ARNs.

↳ Prefer the three service-scoped read policies below; ReadOnlyAccess also grants data reads such as s3:GetObject and dynamodb:GetItem that this skill never needs.

Grants cloudwatch:GetMetricData, cloudwatch:DescribeAlarms, cloudwatch:DescribeAlarmHistory and the CloudWatch Logs read/query actions used in steps 2 and 3.

↳ Replace with an inline policy granting only cloudwatch:GetMetricData, cloudwatch:ListMetrics, cloudwatch:DescribeAlarms, cloudwatch:DescribeAlarmHistory, logs:DescribeLogGroups, logs:StartQuery, logs:GetQueryResults and logs:StopQuery.

Grants xray:GetServiceGraph, xray:GetTraceSummaries and xray:BatchGetTraces for the trace localisation step.

↳ Skip entirely if the workload has no X-Ray or ADOT instrumentation.

Grants cloudtrail:LookupEvents, cloudtrail:DescribeTrails, cloudtrail:GetTrailStatus and cloudtrail:GetEventSelectors for the change-forensics step.

↳ An inline policy with just cloudtrail:LookupEvents and cloudtrail:GetTrailStatus covers the 90-day management-event path.

Starts each CloudWatch Logs Insights query; paired with logs:GetQueryResults and logs:StopQuery for the async poll-and-release loop.

↳ Scope the Resource to the specific log groups, e.g. arn:aws:logs:us-east-1:123456789012:log-group:/aws/ecs/prod-api:*

Runs the batched metric + metric-math queries (error rate, p99 latency, FILL) that quantify blast radius.

↳ GetMetricData does not support resource-level scoping; constrain with a cloudwatch:namespace condition key instead.

Lists control-plane changes in the incident window to find the deploy, scaling action or security-group edit that preceded the metric inflection.

↳ Read-only and cannot be resource-scoped; pair with cloudtrail:GetTrailStatus so an empty result can be distinguished from a stopped trail.

Runs SQL over a CloudTrail Lake event data store when the evidence is older than the 90-day LookupEvents window or needs aggregation.

↳ Scope Resource to the specific event data store ARN; omit entirely if CloudTrail Lake is not in use.

Checks for an open AWS-side service event before the agent spends time blaming the customer's own code.

↳ Requires a Business or Enterprise Support plan; drop it and skip step 1's Health check on Basic/Developer.

Creates or repairs the alarm that should have caught this incident. Gated behind explicit user confirmation because alarms page humans.

↳ Scope Resource to arn:aws:cloudwatch:<region>:<account>:alarm:<prefix>* so the skill cannot overwrite unrelated alarms.

Publishes the incident dashboard that pins the metrics used during triage.

↳ Dashboards are account-global; scope by dashboard name prefix in the Resource element.

Sets retention on log groups found with retentionInDays: null, which is both a cost and a forensics gap.

↳ Scope to the specific log-group ARNs; this action can shorten retention and destroy evidence, so confirm before use.

Lets the agent's base principal assume the read-only observability role in the target (often cross-account) workload account.

↳ Scope to the single role ARN and require an ExternalId condition for cross-account use.

Cause: The role can describe log groups but was never granted the Logs Insights query actions, or the policy Resource does not cover this log group.

Fix: Add logs:StartQuery, logs:GetQueryResults and logs:StopQuery with Resource arn:aws:logs:<region>:<account>:log-group:<name>:* (note the trailing :*).

Cause: The IAM Identity Center session or assumed-role session reached its duration limit mid-investigation.

Fix: Re-run `aws sso login --profile <profile>` (or re-assume the role) and retry; do not fall back to long-lived AKIA keys.

Cause: AWS_ACCESS_KEY_ID starts with ASIA but AWS_SESSION_TOKEN is missing, or the access key was deleted.

Fix: Export AWS_SESSION_TOKEN alongside the ASIA key, or create a fresh key; `unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN` before using AWS_PROFILE.

Cause: The account is at its cap of concurrent CloudWatch Logs Insights queries, usually from abandoned queries that were never stopped.

Fix: Run `aws logs describe-queries --status Running` and `aws logs stop-query --query-id <id>` for stale ids, then serialize queries instead of fanning out.

Cause: Invalid Logs Insights syntax - a leading pipe, an unescaped slash inside a regex, or a field that does not exist in the group.

Fix: Check available fields with `aws logs get-log-group-fields --log-group-name <name>` and validate the query string before re-running.

Cause: Log-group name typo, or the CLI is pointed at a different region from the one the workload logs into.

Fix: Confirm with `aws logs describe-log-groups --log-group-name-prefix <prefix> --region <region>` and set AWS_REGION explicitly.

Cause: The requested period is too fine for the age of the data - CloudWatch keeps 1-minute data for 15 days, 5-minute for 63 days, 1-hour for 455 days.

Fix: Increase --period to 300 or 3600, or narrow the start/end window; prefer get-metric-data, which surfaces the same limits more clearly.

Cause: More than one entry was passed to --lookup-attributes, or an unsupported AttributeKey was used.

Fix: Pass exactly one attribute (EventName, EventSource, ReadOnly, Username, ResourceType, ResourceName, EventId or AccessKeyId) and filter the rest client-side.

Cause: The AWS Health API is only available on Business, Enterprise On-Ramp and Enterprise Support plans.

Fix: Skip the Health check and rely on the public Service Health Dashboard, or upgrade the support plan.

Cause: Too many X-Ray or CloudWatch API calls per second, common when paginating traces in a tight loop.

Fix: Set AWS_RETRY_MODE=adaptive (or boto3 Config(retries={'mode':'adaptive','max_attempts':10})) and add a short sleep between pages.

Cause: The --dry-run permission check failed: the principal lacks ec2:RebootInstances on that instance.

Fix: Grant the action or use a different mitigation. Remember that DryRunOperation - not a silent success - is the signal that the real call would have worked.

Cause: The target role's trust policy does not list the calling principal, or an ExternalId condition is unmet.

Fix: Update the role trust policy (`aws iam update-assume-role-policy`) and pass --external-id on the assume-role call.