It can be all too easy to let Kubernetes-related cloud costs get out of hand—and for many enterprises, it is exactly what is happening. Software provisioning and access to expensive resources like GPUs are just a few of the factors that will increase budgets without a conscious effort to mitigate costs. And as enterprises continue to scale their use of Kubernetes, every little bug and cost inefficiency scales along.
The answer lies in visibility and ownership. Enterprises need to see where and how they are spending with enough granularity to implement change when needed, and they need to cultivate a culture of spend responsibility and accountability that affects both engineering and finance teams. In many cases, the mere act of informing engineering teams of their Kubernetes spend has a significant effect on more efficient spending. More thoughtful use of Kubernetes also results in more streamlined, productive and secure environments, in addition to cost savings.
Enterprises should understand that they have four methods for monitoring Kubernetes costs, with each option best suited for specific use cases:
- Limited cost monitoring. In this method, a centralized team or teams (often finance or devops) are responsible for receiving Kubernetes monthly billing and then handling unnecessary costs and any contributing issues. Organizations with small application engineering teams and less advanced environments are best suited for this method. Those with larger, multi-tenant environments need a more robust approach.
- Shows. The display method introduces detailed breakdowns of Kubernetes costs and cloud costs for each team in the organization. Each team is provided with this accurate spend data so they can better understand and more proactively manage their spend responsibilities. Showbacks are ideal for organizations with three or more application engineering teams and 20+ engineers.
- Refunds. Restorations are restorations with teeth. Here, teams must pay from their own budgets to cover the costs of Kubernetes and the cloud they build. This method is best suited for the same larger organizations as demos. However, for a chargeback approach to succeed, businesses must commit to a chargeback culture and agree that controlling these costs is a key shared goal they can achieve.
- Monitoring of limited costs. This approach requires teams to pay from their budgets if/when their resource costs disappear beyond set spending limits or, in some cases, pay from their budgets only for selected resources. As with chargebacks, the company culture needs to be on board for this method to thrive.
Whatever method an organization uses, Kubernetes cost controls will fail if their deployment is too abrupt, perceived as unfair, or poorly managed. To gain the trust, collaboration, and organization-wide buy-in you need to make your Kubernetes cost controls a success, follow these five best practices.
Build a payback strategy instead of trying to force it overnight.
Teams often experience shock at their first cost reviews and need time to understand why certain costs occur and how to change practices to reduce them. Putting them on the hook for the bill right away—before they’ve had time to think through and put together careful cost-cutting plans—will only lead to panic, bad decisions, and a build-up of resentment from team leaders. Starting with limited cost monitoring or demos allows teams to ease responsibility for costs and provides fair warning of incoming bills.
Make cost allocation fair and transparent.
Teams need full confidence in the cost metrics they are responsible for. However, without careful curation, costs in a Kubernetes distributed system are not that low. To create buy-in, use transparent cost allocation models that ensure these metrics are repeatable, auditable and verified. Also, be sure to provide teams with actionable data and clarify what role they play in controlling overspend.
Be careful with the allocation of idle resources, which typically fall to the team making cluster-level provisioning decisions. System-wide and shared resources also require careful allocation.
Assigning costs by namespace is a particularly powerful method of delineating cost responsibilities. Ideally, allocate costs based on maximum resource requests and team usage, but only if they have control over those settings (which makes it fair). Similarly, find fair approaches to handling expensive one-off jobs such as research projects.
Make the ownership of each resource crystal clear.
Using an access controller and an “escalation approach” can clarify the owner of each resource. The escalation approach consists of defining the owner label at the deployment, namespace, and cluster levels, thereby establishing an escalation path in the event of problems. To enforce these labels, use the Open Policy Agent or access controller webhook.
Review your spending data every week.
Scheduled weekly data reviews allow teams to flag overspend and eliminate future waste, while avoiding sticker shock when monthly bills come due. Automated alerts should also alert if resource usage becomes excessive or unusual and requires attention to avoid overspending.
Focus on culture change.
For enterprises trying to reduce Kubernetes costs as they grow, achieving a culture that values savings and respects existing cost management approaches is the real hurdle. The technical methods behind this cost control are not difficult to implement and follow—if all teams are motivated to do so. Make sure costs are clear, fair, transparent and enforceable, then give teams the tools they need to succeed and the culture will come.
In most cases, enterprises that implement a culture where teams actively regulate their own Kubernetes spend can expect cost savings of 30% or more, along with additional performance and security gains. Distributing responsibility for the costs of a Kubernetes distributed system is a worthwhile exercise and easier to instill sooner rather than later.
Rob Farage co-founded the Kubecost.
The New Tech Forum provides a venue for exploring and discussing emerging enterprise technologies in unprecedented depth and breadth. The selection is subjective, based on our selection of technologies that we believe are important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing materials for publication and reserves the right to edit all submitted content. Send all inquiries to email@example.com.
Copyright © 2022 IDG Communications, Inc.