Our Kubernetes bill was $50K/month. Eight months later, it's $8K. Here's exactly how we did it.

The Cost Breakdown Problem#

Most organizations don't know where their Kubernetes money goes:

Our Initial Spend (Monthly):
├─ Compute Nodes: $25K (60%)
├─ Persistent Storage: $12K (20%)
├─ NAT Gateways: $8K (12%)
└─ Misc (monitoring, logs): $5K (8%)
Total: $50K

Key insight: 60% of spend was on unused compute.

Strategy 1: Right-Sizing Nodes#

The Problem#

Most teams over-provision because:

• Uncertain about peak load
• Fear of performance issues
• Default values too large

The Solution#

Step 1: Measure actual usage

# Get current node sizes
kubectl get nodes -o wide

# Check pod resource requests
kubectl top pods -A

# Analyze over 2 weeks for patterns

Step 2: Implement resource requests/limits

apiVersion: v1
kind: Pod
metadata:
  name: api-server
spec:
  containers:
  - name: api
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Step 3: Use smaller node types

Before:
├─ 10 × n2-standard-8 (8 vCPU, 32GB) = $4K/mo each

After:
├─ 20 × e2-standard-2 (2 vCPU, 8GB) = $0.6K/mo each
└─ Total: $12K vs $40K

Result: 70% compute cost reduction

Strategy 2: Spot/Preemptible Instances#

How It Works#

Cloud providers sell unused capacity at 70-90% discount:

• AWS: Spot Instances
• GCP: Preemptible VMs
• Azure: Spot VMs

Tradeoff: Can be terminated with 24-hour notice

Implementation#

apiVersion: v1
kind: NodePool
metadata:
  name: preemptible
spec:
  nodeConfig:
    preemptible: true
    machineType: n2-standard-4
  
  autoscaling:
    minNodeCount: 5
    maxNodeCount: 50

Recommended mix:

• 30% on-demand (critical workloads)
• 70% preemptible (batch, non-critical)

Cost Impact#

Savings with 70% preemptible:
On-demand node cost: $1,000/mo
Preemptible cost: $300/mo (70% discount)

Mixed cost: 0.3 × $1K + 0.7 × $300 = $510/mo per node
Savings: 49% per node

Strategy 3: Autoscaling Intelligence#

Horizontal Pod Autoscaling (HPA)#

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 2
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

Vertical Pod Autoscaling (VPA)#

Automatically adjust resource requests:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "auto"
  resourcePolicy:
    containerPolicies:
    - containerName: api
      maxAllowed:
        cpu: "2"
        memory: "1Gi"
      minAllowed:
        cpu: "50m"
        memory: "128Mi"

Result: Prevents over/under-provisioning

Strategy 4: Storage Optimization#

The Problem#

Persistent volumes accumulate: 50GB database, 100GB logs, 200GB backups = $12K/month

The Solution#

# Implement storage tiers
StorageClass:
  hot:      # SSD, fast - $0.30/GB/month
  warm:     # Standard, medium - $0.10/GB/month  
  cold:     # Archive, slow - $0.02/GB/month

# Auto-tier based on age
kind: Lifecycle
metadata:
  name: storage-lifecycle
spec:
  rules:
  - action: migrate
    target_class: warm
    days_old: 30
  - action: migrate
    target_class: cold
    days_old: 90
  - action: delete
    days_old: 365

Database Optimization#

-- Before: Uncompressed table
SELECT pg_total_relation_size('events') / (1024^3) as GB;
-- Result: 450GB

-- After: With compression
ALTER TABLE events SET (
  fillfactor = 70
);
REINDEX TABLE events;
CLUSTER events USING events_idx;

-- Result: 120GB
-- Cost reduction: $180/month

Result: 80% storage cost reduction

Strategy 5: Network Optimization#

NAT Gateway Costs#

Problem: 
├─ Data processed: 100TB/month
├─ Cost per GB: $0.045
└─ Total: $4,500/month

Solution:
├─ VPC peering instead of NAT
├─ CloudFront cache for static
└─ New cost: $500/month

Implementation#

apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  type: ClusterIP  # No NAT gateway needed
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080

---
# Use Ingress + CloudFront instead of LoadBalancer
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api
            port:
              number: 80

Result: 80% network cost reduction

Strategy 6: Reserved Capacity#

For predictable baseline load, Reserved Instances provide 25-30% savings:

Comparison:
On-demand n2-standard-4: $0.30/hour = $220/mo
3-year commitment: $0.20/hour = $150/mo
Monthly savings: $70

For 50 nodes:
$70 × 50 = $3,500/month saved

Recommendation:

• 30-40% of nodes on reserved instances
• Remaining on spot/on-demand

The Complete Optimization Checklist#

Month 1: Quick Wins#

• Remove unattached disks
• Delete unused VMs
• Terminate terminated pods
• Savings: ~15%

Month 2: Right-Sizing#

• Implement resource requests/limits
• Analyze and resize node types
• Savings: ~35%

Month 3: Automation#

• Enable HPA/VPA
• Implement spot instances (30% mix)
• Storage tiering
• Savings: ~50%

Month 4-6: Advanced#

• Reserved instances for baseline
• Network optimization
• Database optimization
• Savings: ~80%

Monitoring and Alerts#

Track your progress:

# Prometheus rules
groups:
- name: cost_monitoring
  rules:
  - alert: KubernetesClusterCostHigh
    expr: sum(container_memory_usage_bytes) / (1024^3) > 500
    annotations:
      summary: "High memory utilization: {{ $value }}GB"
      
  - alert: UnusedPersistentVolume
    expr: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes > 0.8
    annotations:
      summary: "High PV utilization: {{ $value }}"

ROI Calculation#

Our Optimization Journey:

Initial: $50K/month × 12 = $600K/year

Optimization Costs:
├─ Engineering time: $50K
├─ Tools (Kubecost, etc): $10K
└─ Total: $60K

Savings (Year 1):
├─ $50K → $8K per month
├─ Monthly savings: $42K
└─ Annual savings: $504K

Year 1 ROI: ($504K - $60K) / $60K = 740%
Year 2+: Full $504K savings with no additional cost

Tools That Helped#

• Kubecost: Visibility into Kubernetes spending ($500/mo)
• Karpenter: Advanced autoscaling (free/open-source)
• Lens: Visual cluster management (free)
• Prometheus: Metrics collection (free)

Lessons Learned#

1. Visibility First: You can't optimize what you can't measure
2. Right-sizing > Reserved Instances: Better ROI
3. Spot Instances are Game-Changing: 70% savings for right workloads
4. Storage is Hidden: Often 20-30% of total spend
5. Network Adds Up: $0.045/GB × 100TB = $4,500

The Surprising Finding#

The biggest issue wasn't technical—it was organizational. Teams didn't know about resources because:

• No cost allocation by team
• No incentives to optimize
• Complex billing

Solution: Show each team their costs. Optimization becomes everyone's problem.

Conclusion#

Kubernetes cost optimization isn't about cutting corners—it's about:

• Right-sizing resources
• Using appropriate instance types
• Automating where possible
• Measuring and monitoring

Result: Better performance at lower cost.

Start measuring today.

---

References#

• Google Cloud Cost Optimization
• Kubecost Documentation
• Karpenter
• AWS Compute Optimizer