Cost assessments are sort of our bread-and-butter here at Duckbill, both in identifying ways to reduce costs and, more importantly, assessing an organization’s ability to manage its own costs. About two years ago, we decided to formalize our assessment methodology into a structure we call Cloud Cost Compass.

Since then, we’ve run this assessment on many dozens of organizations, with monthly AWS spend ranging from $100,000 per month up to several tens-of-millions per month. Today, we’re publishing our assessment methodology.

Two interesting takeaways

We learn two interesting things from our experience running this assessment on so many organizations.

First, high-performing organizations are not high-performing everywhere, and vice versa. The reality is that even in low-performing organizations, there are always pockets of high performance. In high-performing organizations, there are always pockets of weakness.

Second, organizations have a tendency to regress their performance during periods of business growth, creating a need to rethink their approach to managing costs. This makes sense, of course: innovation is inherently wasteful, and growth is often better than cutting expenses. As growth slows down, the need for efficiency comes back into play.

Criteria

Cloud Cost Compass contains six areas to assess. 

Compute OptimizationCompute Optimization is a measure of compute density, intentionality of choices concerning various kinds of available compute resources, and use of ephemeral compute resources.
Data OptimizationData Optimization measures how well an organization understands and leverages the available types of AWS data storage and understands and optimizes for the architectural and financial impacts of data transfer. 
CloudinessCloudiness is a measure of how an organization uses AWS, employs modern cloud practices, and level of continual improvement. Does the organization treat AWS simply as another data center by only leveraging the most basic primitives of AWS? Or does the organization make extensive use of the higher-order services? The more the latter is true, the more optimized cloud costs tend to be.
Resource Purchasing CommitmentsResource Purchasing Commitments measures the extent to which an organization takes advantage of the significant discounts AWS provides for customers who are willing to commit to a base standard of usage. This includes Reserved Instances, Savings Plans, and private pricing agreements (EDP/PPA).
Cost VisibilityMaking costs more visible across the organization is an important aspect in understanding and controlling those costs. Companies that make investments in visibility strategies fare better with understanding and managing AWS costs. This criteria also covers cost allocation.
Collaborative Cost ManagementMeasures the strength of the collaborative relationship between Finance and Engineering when it comes to AWS cloud costs, particularly decisions and understanding related to unit economics, forecasting, business seasonality, and each department’s ability to influence spend.

Rubric

We score an organization on a 1-5 scale for each area. We intentionally leave 2 and 4 empty to provide room for in-between scores. The rubric is subjective, as it’s designed to be used and interpreted by people with expertise in the subject matter rather than run by some automated system or a junior-level team member.

12345
Compute OptimizationNearly all of our infrastructure is on long-lived EC2 instances. We treat AWS as just another datacenter.Extensive use of AWS-managed services relevant to the environment’s use cases.

Extensive use of IaC tools and practices.

Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices. 

Established architectural review processes.
Instance types are intentionally chosen

Anything that can have a lights-out has one

Serverless is used as much as is practicable

High utilization of low-cost process architectures (eg Graviton)

Maximized compute density
Data OptimizationWe almost exclusively use EBS storage or EC2 instance-backed storage. We leverage S3 only for backup purposes.

No S3-IT, no EBS/S3 lifecycles. Basically, AWS defaults.

No understanding of data transfer or data lifecycle.
Consideration given toward use of various S3 storage classes

Efforts made toward understanding and classifying data usage patterns

Efforts made toward understanding data transfer patterns
Extensive/predominant use of lowest-cost S3 storage classes.

Demonstrated understanding of data usage patterns (instance store, EBS, S3, etc).

Data transfer costs are well-understood and architectural decisions are data-driven. 
CloudinessWe use only the most basic AWS primitives, like EC2, S3, EBS, and RDS.

Use of old-school orchestration systems (eg, OpenShift) or hypervisors (eg, VMware)
Moderate use of AWS-managed services relevant to the environment’s use cases.

Moderate levels of IaC tooling and practices
Moderate levels of knowledge across the Engineering org with respect to AWS/cloud design and architecture 
Extensive use of AWS managed services relevant to the environment’s use cases.

Extensive use of IaC tools and practices.

Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices.
Established architectural review processes.
Extensive use of AWS managed services relevant to the environment’s use cases.

Extensive use of IaC tools and practices.

Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices.
Established architectural review processes.
Resource Purchasing CommitmentsWe have not purchased a Reserved Instance or Savings Plan yet. Everything runs on-demand.

No private pricing contracts, though eligible for them.
Ad hoc purchasing of RIs/SPs for eligible services.

No assigned owner for contracts, but contracts are in place.

No assigned owner for RIs/SPs.

RI/SP coverage consistently >75% and utilization consistently >80%
RI/SP coverage consistently >90% and utilization consistently >90%

We have an assigned owner whose job is to forecast and maintain RI/SP purchases.

We have an assigned owner for the AWS bill and contract and have high confidence we have the best-for-us deal possible.
Cost Visibility I get a bill every month and I pay it (sometimes on-time!)One sole owner of AWS spend, but it’s at a VP/CTO level. No defined process for spend visibility/management.

Tagging strategy in place but inconsistently applied with low to moderate coverage.

Low-to-moderate ability to map spend to teams/products/services. Teams do not contribute to managing their spend in an active way.
We have a dedicated role/team for understanding spend.

We can accurately map spend to teams/products/services/projects with ease. These teams/products/projects are aware of their spend and take an active role in managing it.

Strong understanding of our cost model and how that influences our AWS private pricing and contractual commitments.
Collaborative Cost ManagementRelationship between engineering and finance is either non-existent or actively hostile.
Finance has no data on cloud spend beyond the AWS bill.
Finance has no influence on cloud spend. Engineering has no influence on budget.

Unit economics are undefined.

There is no cloud spend forecast.

No understanding of business seasonality as it applies to cloud cost.
Engineering and finance regularly communicate about cloud spend, primarily to share information.

Finance periodically gets data on cloud spend from engineering.

There is an established, regular way to set goals on cloud spend. It’s not collaborative, but it’s functional.

You have a unit economics story that somebody set, but it’s not clearly understood how you got to it or how accurate it is. It’s not really used outside finance.

You have an unreliable forecast. 

Basic understanding of business seasonality as applied to Engineering and Finance concerns.
Engineering and finance effectively communicate about cloud spend.

Finance has a self-service way of looking at cloud spend by cost center, customer, etc.

Finance and engineering actively collaborate to influence/set goals on cloud spend in response to broader organizational pressures.

You have a unit economics story, both eng and finance know how you got it, and either eng or finance can influence it. Unit costs play an active role in decision-making across the organization.

Engineering and finance actively collaborate to continually refine their cloud spend forecast.

Demonstrated ability to tie business seasonality with Engineering and Finance concerns.

Want us to assess your organization?

Just reach out and we’ll work to get you scheduled.

Group 3 Group 9 Asset 20 Asset 21