Tsvetan Ron Jason Federico

Agenda

• The Problem

• The Solution

• The Future

The Problem

• AWS temporary instances are started and forgotten

• AWS instances are started by an employee and then they leave the company

• Creates huge costs and makes us sell AVAX for costs.

Components

How bad is it?

• We identified 51 running instances across all regions

• Total cost accumulated so far: $78,691.76

• The top 5 most expensive instances account for over $43,201.44 combined — roughly 55% of the total spend

• Average cost per instance: $1,542.98

• Median cost per instance: $699.28

What we did

Blame Game
Who did it?

And the winner is...

Austin for his instance

(honorable mention: Darioush)

Slack!

Explain Output

Activity Metrics

• How long was the instance around?

  • Root EBS AttachTime (survives stop/start).

• When was the instance last started?

  • Query DescribeInstances.LaunchTime

• ​When was the last time it did real work?

  • Most recent hour with hourly peak CPU ≥ 5 % (30-day window).

Verdicts / 1

• Exempt: EKS / ASG / Spot → hidden

• Expired: ExpiresAt past →HIGH

• ExpiringSoon: ExpiresAt within 72 h → MED

• Inactive (vetoed by future ExpiresAt): time since last_active - 7 d LOW, 14 d MED, 30 d HIGH

Verdicts / 2

• Underutilized (vetoed by future ExpiresAt): p95 < 2 % → LOW - "95% of hours were at or below this CPU level, only the top 5% of hours were busier."

• LongLived: age ≥ 90 d → Low

• NonCompliant: tag missing → Low (High if ManagedBy=gasleak/* stripped)

Next Targets

• Dockerize the binary

• Hit all development targets

• Monitor all k8s deployments

• Full slack integration, identify slack user and notify them

• Regular "you spent XXXX" reports via slack

• Shaming reports "top AWS user this week was ..."

• Automatic scheduled shutoff via AWS API

Questions?