---
name: performance
description: Use for performance, profiling, latency, throughput, allocation, caching, and hot paths.
---

# Performance

## Iron Law

`MEASURE BEFORE OPTIMISING. MEASURE AGAIN BEFORE KEEPING THE CHANGE.`

## When to Use

- Diagnosing slowness, optimizing latency/throughput/allocation,
  reading profiles, designing benchmarks, investigating p99/p99.9, or
  deciding whether a performance change is worth it.
- Adding, reviewing, tuning, or debugging caches (application,
  database, Redis/Memcached, CDN, browser, edge), including stale
  data, stampedes, hot keys, and miss latency.

## When NOT to Use

- Concurrency correctness without measured slowness; use
  `async-systems`.
- Database query safety without profiling context; use `database`.
- HTTP API cache semantics unrelated to storage or performance; use
  `api`.

## Core Ideas

1. Name the target metric before changing code.
2. Use a realistic workload and identical before/after conditions.
3. Optimize the measured bottleneck, not the suspicious-looking code.
4. Tail latency matters; averages hide user pain.
5. CPU, off-CPU, memory, allocation, I/O, lock contention, and
   network wait are different problems.
6. Micro-benchmarks prove local mechanics, not end-to-end wins.
7. Keep complexity only when the measured gain justifies it.

For caches:

8. No invalidation story, no cache.
9. Keys encode freshness, tenant, permissions, locale, and version
   where those affect the value.
10. Prefer event/key-based expiration for correctness; use TTL as a
    safety net, not the primary truth.
11. Every hot key needs stampede protection.
12. Layered caches need explicit ownership so stale data cannot hide
    in an outer layer.
13. Never put secrets or raw PII in keys or values unless the cache is
    treated as sensitive storage.

## Workflow

1. Define the metric: p99 latency, throughput, CPU time, allocation
   rate, memory, or error budget impact. Capture baseline with
   production-shaped data and concurrency.
2. Profile to find the dominant bottleneck. If caching is considered,
   state the value being cached, source of truth, invalidation trigger,
   TTL/jitter, stampede policy, stale tolerance, and cache metrics.
3. Make one change.
4. Re-measure under the same conditions. Check adjacent regressions:
   memory, error rate, tail latency, CPU, maintainability.

## Verification

- [ ] Target metric is named and user/business relevance is clear.
- [ ] Baseline and after measurements use the same workload and
      environment; raw results or profile artifacts are saved.
- [ ] Only one performance change is measured per commit.
- [ ] Off-CPU and allocation behavior were considered where relevant.
- [ ] Load generator avoids coordinated omission for latency work.
- [ ] Adjacent metrics did not regress enough to erase the win.
- [ ] Added complexity is justified by measured improvement.
- [ ] Cache invalidation is specified: owner/trigger/stale tolerance
      documented; TTL has jitter where many entries can expire
      together; negative caching is intentional and bounded.
- [ ] Cache keys include all inputs that change the value and exclude
      secrets/raw PII unless the cache is treated as sensitive storage.
- [ ] Hot keys are protected by singleflight, locking, probabilistic
      early refresh, or equivalent.
- [ ] Cache metrics cover hit rate, miss latency, eviction, memory,
      and refresh errors; tests cover stale data and invalidation, not
      only the warm-cache happy path.

## Handoffs

- Use `database` for query plans, indexes, and migration risk.
- Use `observability` for production validation and continuous
  profiling.
- Use `security` when cache keys/values can include tenant data,
  secrets, authorization context, or personal data.
- Use `error-handling` when cached data participates in remote-call
  retry or fallback behavior.

## References

- Flame graphs: <https://www.brendangregg.com/flamegraphs.html>
- USE method: <https://www.brendangregg.com/usemethod.html>
- Coordinated omission:
  <https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE/m/BfDekfBEs_sJ>
- RFC 5861 stale-while-revalidate:
  <https://datatracker.ietf.org/doc/html/rfc5861>
- Cache stampede prevention:
  <https://cseweb.ucsd.edu/~avattani/papers/cache_stampede.pdf>