Triaging 4C Insight's AWS Cost Spikes

4C Insights builds and optimizes ad campaigns across closed ecosystems like Facebook, Twitter, and LinkedIn. Last summer we were lucky to catch an unexpected cost increase for 4C that was more than doubling their daily AWS bill. Our experience with 4C inspired us to build an automated cost anomaly detection system, allowing customers to immediately identify root causes of sudden cost increases so would never get blindsided by another unexpected bill.

This past summer our cloud billing analysts noticed that 4C’s daily costs had more than doubled from $150 to $350. We got in touch with them and the spike turned out to have been caused by a misconfigured developer deployment. Alerted to the situation, they engineered a fix that got their costs back to normal.

We were happy we managed to catch the spike before the excess costs spiraled out of control, especially since typically, no one would look at the AWS bill until the end of the month and the increase occurred on the 7th.

In the future we didn’t want to leave situations like this to chance, so we got to work building an automated cost anomaly detection system that could surface the key drivers of a cost increase the day it occurs. We now provide this service for all of our customers and so far it’s flagged anomalous cost increases for over 60% of our installed customer base.

In November, 4C again experienced an anomalous increase, but this time our automated system sprung into action, sending an alert on the day of the increase. The email alert includes a breakdown of the services, accounts and regions responsible for the cost spike to help them figure out what caused it.

Our infrastructure analysis page provided the 4C engineering team further tooling to break down and pin-point the exact EC2 instances driving the cost increase in even more detail.

With these new tools at their disposal, 4C was able to isolate the source of the cost increase and get their costs back down to normal within days. They sent us this screenshot to confirm the cost increase was addressed on their end.

Had the cost increase continued unchecked for the rest of the month, it could have cost them over $2000 in unnecessary fees.

With this system in place, the 4C finance team doesn’t have to worry about unexpectedly high AWS bills at the end of the month, and the dev team doesn’t have to waste time trying to parse through cost explorer every day.