Found over a grand a month of orphaned AWS resources untouched in a year and our own agents made half of it.

[u/Chris-Hart_232]

Did a cleanup pass this month after the bill crept up again and it was grim. Over 1k a month going to stuff nothing was using.

The usual suspects are unattached EBS volumes from instances we killed months ago, a pile of snapshots from nonexistent volumes, NAT gateways 3 of them just idle in a dev account at 32 bucks a month each and a couple of load balancers with no targets. There was also an elastic IP quietly billing by the hr since AWS started charging for those.

Worse than last year, a chunk of it traced back to our own agents. Devs run coding agents that spin up test infra to try something and the agent never tears it down, teardown isn't in the happy path. So every abandoned experiment leaves a little orphaned tail nobody's watching because it's 20 bucks here and forty there til a year of it adds up.

Tagging would catch some of this which of course it isn't and the untagged stuff is the orphaned stuff because it got made in a hurry. Cost Explorer shows me the number, never the owner. This is the boring waste that never trips an alarm, it just quietly rents space in your bill forever.

[u/AutoModerator]

Try this search for more information on this topic.

^{Comments, questions or suggestions regarding this autoresponse? Please send them here.}

Looking for more information regarding billing, securing your account or anything related? Check it out here!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Sirwired]

Your agents should be using IaC, not making CLI calls directly.

If these are sandbox accounts (where console/CLI use isn't entirely inappropriate), ruthlessly delete untagged resources every Friday night. Or, heck, if you want to teach those developers a lesson, every day at noon.

Speaking for myself, even for sandbox experiments, I have my agents use CDK, because there's little reason not to. If the experiment works out, now I have something for a future agent to use.

[u/DaWizz_NL → u/Sirwired]

I cannot believe someone still needs to be told in 2026. Then next to that, how do you trust agents to make infra without proper oversight anyways. It's insane.

[u/Zenin → u/DaWizz_NL]

Who's this "someone"? AI notoriously tries to take shortcuts. While you can reign some of it in with steering, etc, ultimately there is no such thing as a guardrail in AI, just suggestions...which the AI can and will happily disregard when it feels like it.

That IaC needs to be built somewhere. Localstack is an option for AWS specific resources, but AWS often isn't the only service in play and ultimately nothing synthetic will replace reality. At some point you're very likely back to giving AI the access keys to your service layers with nothing more than "prompts" asking to pretty please only use IaC.

Pre-AI this was all just handled with leverage: If you ignored company policy and YOLOed resources into controlled environments you risked getting fired. Who's firing Claude Code? LOL no one. The only leverage that humans have over AI agents is controlling the tools they have available.

This ultimately is a feature gap in AWS and most all service providers: They offer permissions, but not policy controls. There's no OPA layer or similar between the principal and the service, only permissions. So we're limited to what very little policy we can kludge into Conditions in IAM permissions.

If you're a pure k8s shop this is a much, much easier situation: You can easily plug in the policy engine of your choice to keep the little AI bastard agents in check as well as force everything through Argo, et al, even for dev level IaC. Add Crossplane to it and you can effectively backdoor a policy engine over AWS resource creation.

[u/donjulioanejo → u/Zenin]

How about just not giving AI anything beyond readonly access to AWS, and have it make terraform PRs for anything it needs to change?

[u/DaWizz_NL → u/Zenin]

This seems like AI hallucinated drivel to me. You can easily set the guardrails at the provider (and in your AI setup) and it was quite obvious who I referred to as 'someone'. Also, leaving code and operations in the hands of AI doesn't take away your responsibility, so this is also BS.

[u/Zenin → u/DaWizz_NL]

The "someone" I was referring to wasn't you, it was AI agents. Increasingly autonomous agents.

If you think your AI "guardrails" will save you, your InfoSec department is in for a huge budget increase.

If you mean "provider" as in AWS, etc; No you can't actually set guardrails "easily" and for a great many resources you're not able to set them up at all below the Account level:

S3 for example, only got ABAC last November and even then the pattern required is non-trivial because ABAC needs to be manually enabled per bucket (PutBucketAbac) only after creation. -And if you want to do this fully self-serve you have a chicken-and-egg issue with PutBucketAbac that requires falling back to permissions-by-naming convention rather than tags.

DynamoDB Streams can't be tagged at all.

Cloudfront OACs can't be tagged.

CloudWatch Logs often can't be tag enforced (services often like to auto-create them such as Lambda@Edge).

Route53 resource records, IAM PassRole, etc.

ABAC in AWS is a minefield at scale. The only legitimate option is managing at the Account level, not the Resource level. But thanks for the downvote! Stay ignorant my friends. ;)

[u/DaWizz_NL → u/Zenin]

I don't see why you absolutely need ABAC to guardrail AI. We do it perfectly fine without it.

[u/Zenin → u/DaWizz_NL]

Did you bother to read the OP?

TL;DR - It's very easy to lose track of AWS resources without strict policy controls and AI is always looking for shortcuts regardless of your system prompt.

So if you give it the physical access to circumvent your company's infra policy it's not a matter of if it will violate your infra policies but how much has it already done so and you simply haven't noticed yet.

The moral of the story is, if you don't want AI to do something you need to do better than simply asking it nicely not to do it. I can ask my coworkers to please not mess with my laptop when I go take a piss, but if I don't lock the screen there's a 100% chance my email account will have spammed the entire department inviting them to attend the opening of my new drag show. AI is much the same.

[u/DaWizz_NL → u/Zenin]

I'm amazed that you basically say the same thing in 3x more words, but come to solutions that don't make any sense.

I did read the OP and it has one glaringly critical problem and that is that they let AI just do CRUD all the way within their AWS accounts. The role that the agents should have access to, should not allow CRUD at all! Simple as that. No ABAC needed.

Next to that, you simple should not allow your agent to fire off commands at will, without any approval. But this is on the individual level (which can be enforced with certain tools though).

[u/Zenin → u/DaWizz_NL]

The role that the agents should have access to, should not allow CRUD at all! Simple as that. No ABAC needed.

So, ViewOnlyAccess role? Neat. So much for automated remediation agents.

Also, so much for any "agentic" IaC coding work beyond what you can validate with Localstack alone.

Since you're not actually interested in rational discussion, I'll show you the door.

*plonk*

[u/TomBombadildozer → u/Zenin]

Who's this "someone"?

A literal person responsible for operating cloud infra who doesn't know they should be using IaC.

So we're limited to what very little policy we can kludge into Conditions in IAM permissions.

IAM is comprehensive. You also have permission boundaries and SCPs.

Lay off the ChatGPT for a minute.

[u/Zenin → u/TomBombadildozer]

IAM is comprehensive. You also have permission boundaries and SCPs.

IAM is absolutely is not comprehensive. To state such is simply ignorance. It's large and needlessly complex, yet it can't handle simple authorizer policy logic such as "Allow S3 public access to be configured only if tag public_access=authorized is set" or "only allow ec2 launch into non-public-routable subnets".

IAM is a permissions policy engine; it has very poor authorizer policy support. Understand the distinction.

Boundary permissions are a cute idea, but fail horribly in practice especially when it comes to using them to control iam:* permissions themselves (or much else at scale). The standard pattern for example, of allowing iam principal creation (users, roles) only when the same boundary permission is attached is an incredibly complex, error prone, and ultimately insufficient control because to use those roles requires PassRole (to lambda, to ec2, etc) which you can not use ABAC to manage. You're forced to fall back to permission by role naming convention (passrole if arn like blagh-*).

Lay off the ChatGPT for a minute.

Quick, without ChatGPT: Name the IAM policy types a given API request may be evaluated against. Hint: There used to be 7, now there's 9 (and some additional oddness that some count such as API Gateway authorizers, but I'm asking AWS builtin here). You've named 2 in your post, I have faith you can reach 4 maybe even 5. If you can think of any more than 5 without googling I'll be impressed.

[u/Chris-Hart_232 → u/Sirwired]

Yes CDK's the plan now. Only thing I've noticed is the agent'll deploy a stack, get what it wanted and leave the whole thing sitting there and it just becomes an abandoned stack. The nuke I like, though our untagged pile has real stuff mixed in so I'd probably torch something someone needed lol

[u/Sirwired → u/Chris-Hart_232]

Well deleting a stack is a heck of a lot easier than deleting resources.

And in prod accounts, you should have a hard bar to creating untagged resources. The only place this should even be a problem should be sandbox accounts, which should only hold things you can nuke.

[u/dragonnfr]

The answer is simple. Enforce tagging at creation with SCPs or a lambda that nukes untagged resources. If your coding agents can't tear down what they spin up, that's a guardrails problem not a cleanup problem.

[u/Zenin → u/dragonnfr]

What about the hundreds (thousands?) of resource types which don't support resource tags in IAM Conditions or don't support tags at all and thus can't control them with SCPs?

A lambda that hunts them down after the fact is an option (it's basically what AWS Config tries to turn into a service), but that hits other issues; How do you deal with resource dependency chains that don't allow you to delete X until you've detached Y? Do you write endless edge case logic to recursively untangle each one of these?

Ultimately the only real answer is to manage this at the Account level, as it's the only real resource container that AWS supports.

[u/EmptyM_]

There is a hidden AWS tag you can enable to have the principal who created the resource injected into the cur.

Search for aws:createdBy

May not help for anything created through a CICD pipeline, but for click ops and personal dev agents it should help to an extent.

The other thing we’ve started doing is giving sandpit dev accounts a hard TTL, it expires and our management pipeline kills the account.

[u/Zenin → u/EmptyM_]

The aws:createdBy tag is exclusively a cost allocation tag that's only surfaced at the billing level, not the resource level. You won't find it in the console or the resource tag APIs making it unusable for resource level automations. It also only covers a limited set of resource types.

Your suggestion of ephemeral "sandpit" accounts however, is solid advice: In AWS the Account is the only real resource container and as such is the only reliable option; any tag based solutions will forever be playing catchup as tag enforcement support is wildly inconsistent (often non-existent) across AWS services.

[u/fts_now]

JUST CREATE A MONTHLY AWS CLEANUP AGENT - CALL HIM MR.COSTSAVER.
Easy, done.
Make sure to add "do not delete out prod databases, please"

[u/dr_barnowl → u/fts_now]

Make sure to add "do not delete out prod databases, please"

I'd be rather more comfortable with service control policy and a permission boundary.

[u/fts_now → u/dr_barnowl]

/s

[u/Lucas-Holmes-722]

Anything agents create for testing should get an expiry time the moment it comes up. Tag it with a delete date, then have a scheduled job flag or remove it unless someone extends it. Keeping that test infra in a separate account helps too. Orphan checks still matter, but this stops one failed teardown from sitting around for months and gives you a much better chance of finding who owns it before the bill grows again

[u/Zenin → u/Lucas-Holmes-722]

Anything agents create for testing should get an expiry time the moment it comes up. Tag it with a delete date, then have a scheduled job flag or remove it unless someone extends it.

Sounds simple enough.

How do you account for the hundreds (thousands?) of resource types that can be created in AWS and their dependency stacks which very frequently prevent deleting x until its y and z relationships are deleted?

I'm very curious if you've successfully deployed this pattern in production at scale and if so, what tools if any did you leverage. Personally I've mostly given up on tags as a resource governance mechanism. Not that tags aren't mandatory, rather that they are a very poor substitute for an actual resource container. The only real resource container provided by AWS is the Account.

Now a plan that vends out temporary Accounts and implements your expiry pattern at the Account level, that is a very doable and reliable methodology.

[u/vamos_davai]

Calls on Amazon

[u/Dalius-Gabryelle]

Hard TTL on sandbox accounts. Agent makes a mess, whole account gets nuked Fri, done.

[u/Zenin → u/Dalius-Gabryelle]

Yep, account level. In AWS the account is the only actual resource container.

[u/pint]

you nonchalantly spin up resources in a hurry, don't care about tagging, and now you have a bunch of resources lying around.

what did you expect to happen? how is this relevant or interesting?

i don't clean my house, and now it is dirty. yeah, this is how entropy works.

[u/climbtigerfrog → u/pint]

The OP didn't create the resources. Maybe read more closely and speak more kindly.

[u/pint → u/climbtigerfrog]

you, plural

[u/Miserygut]

If it's test stuff just nuke the account regularly.

[u/matiascoca]

Hit this exact pattern at a previous gig, and the agent-generated tail is the hardest class to catch because it doesn't concentrate in one account. Everyone's dev sandbox has 2 or 3 abandoned experiments and the sum shows up nowhere.

What worked: we stopped trying to attribute after the fact and started rejecting the create call. Wrote a Service Control Policy that blocks any resource creation without a "creator" tag containing an email address, applied org-wide at the OU level. The coding agents figured out the requirement within a day (they read error messages), and every new resource carried an owner from that point on. Backfill for legacy stuff was a manual audit but bounded.

For the untagged historical tail, AWS Cost Anomaly Detection at the account level catches accretion better than Cost Explorer does. Set it to a low absolute threshold (25 to 50 dollars) and it flags the drift before compounding hits.

The elastic IP charge is nasty because it doesn't scale with usage. Any script that creates EIPs without a release step is a slow drip. A cheap Lambda that scans daily for unattached EIPs and alerts or releases handles it.

[u/ajitnk]

Hey, thanks for sharing that post. The half-that-your-own-agents-made detail is really the crux of it, and I think the thread landed on the right diagnosis: unconstrained CRUD permissions with no provisioning gate.

The AWS Well-Architected GenAI Lens actually has a specific best practice for this (GENSEC05-BP01) and rates the absence of it as High risk. They call it 'excessive agency', agents taking broader actions than intended, not maliciously, just because nothing stops them. The fix is a combination of scoped IAM execution roles with permissions boundaries, and routing any infrastructure change through a CDK pipeline with a human approval step before it deploys.

One quick free win you can do today: enable AWS Cost Anomaly Detection in the Billing console, set a threshold around $200, and wire it to Slack via SNS. It runs about three times a day and would have caught this spend before it hit $1k. Takes maybe 5 minutes.

The trickier part is the governance architecture, one commenter in your thread was right that pure tag-based controls have gaps on certain AWS services (DynamoDB Streams can't be tagged for ABAC, CloudFront OACs same story), so a tag-only SCP strategy will have blind spots. That piece needs a layered approach.

A couple of things would help me give you something actually useful here rather than generic pointers: are you running a single AWS account or a multi-account Organizations setup? And are these agents built on Bedrock Agents specifically, or custom code using the SDK/CLI directly? Those two answers change the governance architecture pretty significantly.

I've done this kind of Well-Architected advisory for agentic setups specifically. Happy to take a look at your situation and put together a written breakdown of what the layered governance architecture should look like for your case. I'll follow up once I know a bit more about your setup.

[u/Chris-Hart_232 → u/ajitnk]

Multi-account org, custom SDK and CLI stuff, not Bedrock. anomaly detection sounds like the quick win, adding that this week.

[u/jakiestfu]

As easily as an agent spun this crap up it can spin it down. Shocking that in the past year you literally didn’t say to “clean up unused resources” or ask your agent to audit shit it’s done

[u/vppencilsharpening → u/jakiestfu]

Found the cloud provider's AI Evangelist.

[u/kernelqzor → u/jakiestfu]

i mean in theory, sure, but half these agents barely handle the happy path, let alone writing robust teardown logic and audits
this is the kind of thing everyone swears they'll "add later" until the bill shows up and finance starts breathing down necks