Chasing the Dragon: The Myth of Rightsizing First
“How much tape are we using?” was the question a CFO asked me recently, along with the product manager, and our CTO.
It took a moment, we stared at each other, and then the CTO replied “Do you mean storage?”.
“Yes storage, we are debating paying dividends to our investors.” This conversation was taking place post acquisition of a SaaS based company by a consulting firm with legacy IT.
The SaaS company’s customers were in the top tier of online traffic and the team was parsing the daily website logs. The technical team had been hitting a wall as our customers wanted daily reporting of these across all their properties correlated to google analytics and other data, but the database and code structure were not able to parse on a cadence in time with the customer’s needs.
While in the middle of fixing this scaling issue, the CFO was asking for costs to be cut. Keep in mind this was a “one pizza” technical team and part of the acquisition was to onboard a lot of new customers.
For most savvy CTO’s and teams that run divisions, they can weigh the Total Cost of Ownership and then do a Scenario Analysis, which is a finance term for looking at multiple business scenarios and cash flows. A Scenario Analysis looks at the tradeoffs and impacts from choosing technology and a team. This decision framework can be applied to products, services, and any new technical initiatives.
Here are the variables to keep in mind for this exercise:
Scenario A
- We spend $45,000 (Developer time) to save $5,000, and don’t gain any new customers (we also don’t receive an R&D credit, as this is not new technology)
- Net $-40,000
Scenario B - We spend $45,000 for new technology so we can add new enterprise customers, obtain an R&D credit, and expand contracts to offset the $45k.
- 1st customer - $50k contract
- Net $5,000
The impact on business and not Utilization is the larger business scenario that the company and division should optimize for. Oftentimes, because engineers own the code and infrastructure, they will want to optimize for Utilization, as reducing cost is something that feels tangible. But cost reduction in the cloud has two components, Utilization and Rates which can be adjusted to impact Cost and Margin. Utilization is how much memory, CPU, space, and network usage the service uses. Rates are the price multiplier based on utilization or runtime of the service such as 1YR Std reserved hourly cost of 0.0843 for an AWS t4g.xlarge server.
Engineers should master this to uplevel, as it gets more complicated at scale.
If you want to uplevel as an individual engineer, the ability to provide a scenario analysis and be part of that loop means you get to see what happens at a team level, then manager level, and so-on.
How this plays out at larger companies
Enter enterprise companies: instead of a quick ask to the CEO or your lead for a bump in compute time (or time to rightsize), you will need to go through a planning and alignment process to figure out what type of work is happening and how you can fit in the rightsizing actions.
While deployment script changes can be rapid, quick code & infrastructure shifts are a thing of the past as there is an authorization and review matrix to make a change. A product development team might own the code, an SRE might own reliability, a security team might own review of service changes, and a FinOps owner might oversee plan purchasing.
Both Tech and DevOps companies have the same issues. For example, here is Gitlabs decision matrix just for financial decision making:
This creates Decision Paralysis, whereby team members are most likely not going to make the best decision for their division or the company, as they only have control over specific aspects of the changes. This leads to the toil of trying to optimize the problems brought up today vs. longer term issues, which are outlined in Google’s SRE handbook.
SRE Principle #2
“SREs must have time to make tomorrow better than today.” SRE - Workbook - Team Lifecyles
Chasing these changes as service usage increases becomes fairly complex, so there needs to be a balance between today and tomorrow. Just “Chasing the Dragon” of optimal utilization for every service becomes a toil, and “if toil becomes too burdensome, talented engineers will flee the team."
To focus your team on growth, reduce burnout, and also keep your margins is a balancing act between Utilization Optimization and Rate Optimization. While waiting is a “rolling meter” that keeps on ticking, you can consider adjusting the rates if you want to reinvest your team’s time and energy. To help with this process, you can utilize our application for rate optimization, which will enable your team with the needed time to properly align Rightsizing and Utilization with their schedule.
If you’re starting to scale-up your organization and the question of cost cutting vs. rightsizing is top of mind, reach out to me or the team with a quick note, or simply sign-up for our “early adopter” program, as we have some exciting optimization analysis tools in the pipeline!
