GOMEMLIMIT and the soft GC pacing change every Go service should set

The problem

A Go service in a Kubernetes pod with a 2 GB memory limit. The Go runtime targets a heap somewhere below 2 GB based on the GOGC setting (default 100 — meaning grow heap to 2× live data before next GC).

If live data is 1.2 GB, Go targets a 2.4 GB heap. K8s kills the pod at 2.0 GB. OOM. Pod restart. Cascade.

The fix

// In init or main
debug.SetMemoryLimit(1800 * 1024 * 1024)  // 1.8 GB, slightly under the cgroup limit

Or via env var: GOMEMLIMIT=1800MiB.

The runtime treats this as a soft limit. As the heap approaches the limit, GC runs more aggressively to keep below it. The pod stays under the cgroup limit; no OOM.

Why “soft”

The runtime can’t promise to stay below the limit if live data plus stack plus goroutines genuinely exceed it. The soft semantics:

Trigger GC more often as you approach the limit.
Spend more CPU on GC if needed.
If the limit is genuinely unachievable, exceed it (and let the OOM killer do its thing).

This is the right shape. Hard limits in a GC’d runtime would either thrash forever or give up too early.

The Kubernetes wiring

Match GOMEMLIMIT to the cgroup limit minus ~10% headroom:

containers:
- name: my-go-service
  resources:
    limits:
      memory: "2Gi"
  env:
  - name: GOMEMLIMIT
    valueFrom:
      resourceFieldRef:
        resource: limits.memory
        divisor: 1
  # OR set it explicitly: value: "1800MiB"

The resourceFieldRef lets you derive GOMEMLIMIT from the cgroup limit automatically.

Side effect: less wasteful behaviour

A side effect of setting GOMEMLIMIT is that the runtime doesn’t grow the heap as much during normal operation. For services with bursty allocation patterns, this means:

Less resident memory in steady state.
More frequent GC (CPU cost).
Lower OOM risk.

For most production services, the CPU cost is well worth the OOM avoidance. For latency-critical paths, profile both GOGC and GOMEMLIMIT to balance throughput vs predictability.

Genie’s settings

GOGC=100        (default — heap grows to 2× live data)
GOMEMLIMIT=1800MiB  (matches the K8s 2 GiB limit minus headroom)
GOMAXPROCS=4    (matches the CPU limit; otherwise Go assumes the whole node)

The three knobs every Go service in K8s should set. Together they make Go’s resource consumption match the cgroup’s view of what’s available.

What I see teams miss

Setting GOMEMLIMIT without setting GOMAXPROCS is a common half-fix. The runtime knows about the memory limit but thinks it has all CPU cores; it spawns goroutines and tunes GC for a much bigger machine than it actually has. CPU starvation and weird latencies follow.

Set all three. Update them together when you change pod resources. The configuration drift between “what K8s thinks” and “what Go thinks” is a class of bug that disappears.

Default for any new Go service: GOMEMLIMIT from cgroup, GOMAXPROCS from cgroup, GOGC=100 until you have a reason to change it. The settings are boring; the OOM kills disappear.