W
Winnoventures
Back to BlogDevOps

Kubernetes at Scale: Lessons from Managing 500+ Microservices

W
Winnoventures DevOps TeamMay 5, 202515 min read

Background

We manage Kubernetes clusters across multiple cloud providers for our enterprise clients. Here are the hard-won lessons from running 500+ microservices in production.

Resource Management

Always set resource requests and limits. Without them, your cluster will be unpredictable:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "500m"

Horizontal Pod Autoscaler

HPA based on CPU alone is insufficient. We use custom metrics from Prometheus:

  • Queue depth for message-processing services
  • Request latency (p99) for API services
  • Active connections for WebSocket services

    • Namespace Strategy

    We use a strict namespace-per-team model:

  • team-frontend, team-backend, team-data
  • Resource quotas per namespace
  • RBAC per namespace

    • This prevents noisy-neighbor problems and makes cost attribution easy.

    Monitoring Stack

    Our observability stack:

  • Prometheus + Grafana for metrics
  • Loki for logs
  • Tempo for distributed tracing
  • Alertmanager for paging

    • Top Mistakes to Avoid

  • 1Not setting PodDisruptionBudgets — causes downtime during node drains
  • 2Ignoring etcd health — a sick etcd brings down the entire cluster
  • 3Skipping network policies — every service can talk to every other service by default
  • 4No pod anti-affinity rules — all replicas end up on the same node

    • Conclusion

    Kubernetes rewards investment in operations. Automate everything, document your runbooks, and build a culture of observability.

    KubernetesDevOpsCloudInfrastructure
    W

    Winnoventures DevOps Team

    Infrastructure Engineering · Winnoventures

    Expert insights from the Winnoventures engineering team — sharing what we learn building real products for global clients.