---
name: redis
description: "Use when using Redis (or Valkey/ElastiCache/Upstash/Dragonfly/Memorystore — any Redis-protocol store) as a cache, queue, rate limiter, or distributed lock and you need it CORRECT, not just connected: stampede-proof caching, locks that don't release someone else's hold, race-free rate limits, jobs that survive a worker crash. Triggers: 'cache stampede', 'thundering herd', 'distributed lock', 'SET NX PX', 'Redlock', 'fencing token', 'rate limiter race', 'INCR then EXPIRE', 'BullMQ stalled jobs', 'XREADGROUP / XAUTOCLAIM', 'maxmemory-policy', 'KEYS * in prod', 'lock released by the wrong owner', 'limitador de peticiones', 'bloqueig distribuït', 'la cau s'invalida malament'. NOT durable SQL queues with SELECT FOR UPDATE SKIP LOCKED (that is postgresdb), NOT vector similarity search over embeddings (that is vector-db)."
tags: [redis, valkey, cache, rate-limiting, distributed-locks, queues, bullmq, streams, lua, data-infra]
recommends: [postgresdb, vector-db, clickhouse-analytics, scaling, nodejs, fly-io]
profiles: []
origin: risco
---

# Redis — cache, lock, rate limiter, queue done correctly

Redis is four primitives wearing one server: a **cache**, a **distributed lock**, a **rate
limiter**, and a **job queue**. Each has a correctness contract that has nothing to do with
whether `SET` returns `OK`. You almost certainly already "have Redis working" — a value goes
in, a value comes out. What you do not yet have is a cache that won't stampede your database, a
lock that won't release someone else's hold, a rate limiter that survives concurrent requests,
and a queue whose jobs aren't silently lost when a worker dies. This skill is engine- and
pattern-level, **client-SDK-agnostic**: examples are `redis-cli` and Lua, so they hold across
node-redis, ioredis, redis-py, go-redis, and Lettuce. Every pattern applies unchanged to Valkey
and any Redis-protocol store, since Valkey is a BSD fork of Redis 7.2 and stays protocol-compatible.

## When to use / When NOT

**Use when you are:**

- Adding or reviewing a **cache**: TTL strategy, cache-aside vs write-through, key naming,
  stampede/thundering-herd prevention, negative caching, invalidation.
- Building a **distributed lock**: "only one worker does X", cron de-duplication, leader-ish
  mutual exclusion — the `SET NX PX` + Lua-release + fencing-token contract, and the Redlock call.
- Building a **rate limiter**: per-user/IP/API-key throttling (fixed window, sliding-window log,
  sliding-window counter, token bucket) and why naive `INCR`+`EXPIRE` is a race.
- Running a **queue / background jobs on Redis**: Lists vs Streams vs a library (BullMQ, Sidekiq,
  RQ, Celery), at-least-once delivery, acks, stalled-job recovery, idempotency.
- Deciding **eviction & persistence**: `maxmemory-policy`, RDB vs AOF, cache mode vs database mode.
- Reviewing Redis code for foot-guns: `KEYS *`, unbounded keys, non-atomic read-modify-write,
  missing TTL, a lock released by the wrong owner.

**NOT for (route instead):**

- Durable SQL queue (`SELECT … FOR UPDATE SKIP LOCKED`), engine-level SQL caching →
  [`../postgresdb/SKILL.md`](../postgresdb/SKILL.md). Boundary: if the *durable store* is the
  queue, that's postgresdb; if *Redis* is the queue, you're here.
- Vector similarity / semantic search as the primary job →
  [`../vector-db/SKILL.md`](../vector-db/SKILL.md). Redis 8 vector sets are only *noted* here.
- Analytical / clickstream event store → [`../clickhouse-analytics/SKILL.md`](../clickhouse-analytics/SKILL.md).
- System-level capacity / load strategy, and app/CDN/HTTP-layer caching (Cache-Control, ISR, edge) →
  [`../scaling/SKILL.md`](../scaling/SKILL.md). This skill owns the Redis-specific data contract underneath.
- Provisioning a managed Redis (cluster sizing, dashboards) →
  [`../fly-io/SKILL.md`](../fly-io/SKILL.md) / [`../aws-essentials/SKILL.md`](../aws-essentials/SKILL.md).
  This skill gives the client-side contract, not console clicks.
- Language-runtime async/job concepts unrelated to Redis → [`../nodejs/SKILL.md`](../nodejs/SKILL.md)
  or your language skill.

## Non-negotiables

1. **Every cache key has a TTL.** No TTL in a cache = a memory leak that eventually evicts or 500s.
2. **Never `KEYS *` in production — use `SCAN`.** `KEYS` is O(N) and blocks the single thread;
   one bad pattern freezes every client.
3. **Read-modify-write must be atomic.** App-side `GET` then `SET`/`INCR` races under concurrency.
   Use one Lua `EVAL`, a single atomic command, or `WATCH`/`MULTI`.
4. **A lock = random token + `PX` + Lua compare-and-delete.** Plain `DEL` releases whoever holds it
   now (maybe not you); a lock with no `PX` is held forever if the owner crashes.
5. **Pick `maxmemory-policy` deliberately.** The default `noeviction` makes a full cache fail every
   write — correct for a database, fatal for a cache.
6. **A queue must ack and recover.** `RPOP` straight into a worker loses the job if the worker dies
   mid-task; use a processing list or Streams consumer groups.
7. **Cap unbounded structures.** Lists grow forever without `LTRIM`; streams without `MAXLEN ~`.
8. **One Lua = one short atomic step.** Redis is single-threaded; a long script or huge `MULTI/EXEC`
   blocks *all* clients, not just yours.
9. **Namespace keys** as `svc:entity:id` (e.g. `cart:user:42`) so scans, eviction, and humans can reason.
10. **Know cache vs database durability before choosing persistence.** Cache = lose-it-and-rebuild;
    database = RDB/AOF and `noeviction`. Decide which Redis is *before* you tune it.

## Decision table — which primitive, which structure

| Need | Structure | Canonical commands | The one gotcha |
| --- | --- | --- | --- |
| Cache | string / hash + TTL | `SET k v EX 300` / `GETEX` / `HSET`+`EXPIRE` | stampede on expiry |
| Lock | string + random token | `SET k tok NX PX 30000` + Lua compare-and-`DEL` | wrong-owner release |
| Rate limit | string / zset / hash | one atomic `EVAL` (window/log/bucket) | lost `EXPIRE` → immortal counter |
| Queue (simple) | list | `LMOVE src proc LEFT RIGHT` / `BRPOPLPUSH` | job loss on crash without a processing copy |
| Queue (reliable) | stream | `XADD` / `XREADGROUP` / `XACK` / `XAUTOCLAIM` | unacked PEL grows unbounded |

## Caching

**Namespace + TTL with jitter.** A fixed TTL on a batch of keys written together synchronizes their
expiry — they all die in the same second and stampede the origin at once. Add jitter.

```bash
# Bad: 1000 keys warmed in a loop with the same TTL all expire together
SET product:42 "$json" EX 300
# Good: spread expiry with per-key jitter (300s ± up to 60s)
SET product:42 "$json" EX $(( 300 + RANDOM % 60 ))
```

**Cache-aside (read path).** *Why:* it is the default for read-heavy data; the trap is forgetting
the TTL and forgetting to cache the miss.

```text
# Bad: cache only hits, never the miss → every request for an absent id hits the DB
val = GET product:42
if val == nil: val = db.fetch(42); SET product:42 val EX 300   # absent rows still hammer the DB

# Good: cache-aside with negative caching
val = GET product:42
if val == "__MISS__": return null                              # cached miss, no DB call
if val == nil:
    val = db.fetch(42)
    if val == null: SET product:42 "__MISS__" EX 30            # short negative TTL
    else:           SET product:42 val EX 300
```

**Stampede prevention.** When a hot key expires, N concurrent requests all miss and hammer the
origin. Two mitigations:

- **Recompute lock** — one request rebuilds, the rest briefly serve stale or wait. *The lock TTL
  must exceed worst-case origin latency*, or two requests rebuild anyway.

  ```bash
  # On miss, try to claim the rebuild; only the winner recomputes
  SET product:42:lock 1 NX PX 5000   # 5s > slowest DB rebuild
  # winner: rebuild, SET product:42, then DEL product:42:lock
  # losers: serve last-known value or retry after a short sleep
  ```

- **Probabilistic early expiration (XFetch)** — refresh *before* TTL with rising probability so one
  request renews early while others still hit a warm key. Math + write-through/read-through/
  write-behind in [`references/caching.md`](references/caching.md).

**Invalidation:** on write, either delete the key (next read rebuilds) or write-through. Never rely on
manual invalidation *instead of* a TTL — you will miss a path and serve stale forever.

## Distributed locks

```bash
# Bad: GET-then-DEL races — between your GET and DEL the lock can expire and a NEW owner takes it,
#      and your DEL frees THEIR lock.
GET resource:lock        # == my token?
DEL resource:lock        # ← may delete someone else's lock

# Good: claim with a random token + PX, release only if the token is still mine (atomic Lua).
SET resource:lock "$TOKEN" NX PX 30000     # NX = only if absent; PX = auto-expire so a crash frees it
```

Release **must** be a compare-and-delete in one Lua step:

```lua
-- KEYS[1] = lock key, ARGV[1] = my token. Returns 1 if I held and released it, else 0.
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end
```

**TTL vs work duration.** If the protected work can outlast `PX`, the lock expires mid-task and a
second worker starts. Either size `PX` above the worst case, or run a watchdog that renews the TTL
(again via a token-checked Lua `PEXPIRE`) while work continues — never blindly `PEXPIRE`.

**Fencing tokens.** A GC pause or a long syscall can freeze the lock holder *past* expiry; another
worker acquires, then the paused one wakes and writes — both think they hold the lock. The only fix
is a **fencing token**: a monotonically increasing number issued with the lock and checked by the
protected resource, which rejects any write carrying a stale token. No Redis lock alone provides this.

> **Redlock decision.** Single-instance `SET NX PX` is fine when occasional double-execution is
> *tolerable* (idempotent work, best-effort cron de-dup). For *hard* mutual exclusion, a single
> Redis lock is not enough and multi-instance Redlock is contested (it assumes bounded clock drift
> and no long pauses) — **fence the resource or use a lease/consensus system**. Full Redlock
> walk-through and critique in [`references/locks-and-rate-limiting.md`](references/locks-and-rate-limiting.md).

## Rate limiting

```text
# Bad: GET/INCR then EXPIRE as two steps. If the process dies (or loses the race) between INCR and
#      EXPIRE, the key has NO TTL and counts forever — the user is throttled permanently.
n = INCR rl:user:42
if n == 1: EXPIRE rl:user:42 60     # ← may never run
```

Fixed window, done atomically in one `EVAL`:

```lua
-- KEYS[1] = counter, ARGV[1] = limit, ARGV[2] = window seconds. Returns 1 = allow, 0 = deny.
local n = redis.call('INCR', KEYS[1])
if n == 1 then redis.call('EXPIRE', KEYS[1], ARGV[2]) end   -- TTL set atomically with the first hit
if n > tonumber(ARGV[1]) then return 0 end
return 1
```

Fixed window allows up to 2× the limit across a boundary. For smoother limits:

- **Sliding-window log** — a sorted set scored by timestamp, pruned and counted in one Lua:
  `ZREMRANGEBYSCORE` (drop entries older than the window) → `ZCARD` (count) → `ZADD` (record now) →
  `EXPIRE`. Exact, but O(requests) memory per key.
- **Token bucket** — a hash holding `{tokens, last_refill}`; refill by elapsed time and decrement,
  all in one Lua. Allows bursts up to bucket size, then a steady rate. Smallest memory.

Choose by accuracy-vs-memory; full Lua for all four in
[`references/locks-and-rate-limiting.md`](references/locks-and-rate-limiting.md).

## Queues

| Option | When | Acks / recovery |
| --- | --- | --- |
| **List** | simplest at-least-once, low volume | manual: processing list + requeue stalled |
| **Stream** | reliable jobs, consumer groups, native acks | `XACK` + `XAUTOCLAIM` for stalled/pending |
| **Library** (BullMQ/Sidekiq/RQ/Celery) | you want retries, scheduling, a dashboard | built-in; don't hand-roll |

**Reliable list:** never `RPOP` straight into the worker — a crash mid-task loses the job. Move it to
a per-worker processing list atomically, then ack by removing it.

```bash
job=$(redis-cli LMOVE jobs jobs:proc:w1 LEFT RIGHT)   # atomic: pop from jobs, push to processing
# ... do work (idempotently) ...
redis-cli LREM jobs:proc:w1 1 "$job"                  # ack: remove from processing
# a reaper requeues anything left in jobs:proc:* after a worker dies
```

**Streams (preferred for reliable jobs):** consumer groups give per-message acks and a Pending
Entries List for recovery.

```bash
redis-cli XGROUP CREATE jobs g1 '$' MKSTREAM
redis-cli XREADGROUP GROUP g1 worker1 COUNT 10 BLOCK 5000 STREAMS jobs '>'
# ... process ...
redis-cli XACK jobs g1 "$id"                          # ack a done message
redis-cli XAUTOCLAIM jobs g1 worker2 60000 0          # reclaim messages idle > 60s (stalled worker)
```

Delivery is **at-least-once**: a message can be reprocessed after a crash, so make handlers
idempotent (dedupe on a job id / idempotency key). Reach for **BullMQ** (TypeScript; stalled-job lock
renewal via `lockDuration`/`lockRenewTime`), **Sidekiq** (Ruby), or **RQ**/**Celery** (Python) when
you want retries, delays, and a dashboard — don't reinvent them. Full implementations, DLQ, and the
library comparison in [`references/queues.md`](references/queues.md).

## Eviction & persistence

| `maxmemory-policy` | Behavior | Use for |
| --- | --- | --- |
| `noeviction` (default) | writes fail when full | Redis-as-database (durable data you can't drop) |
| `allkeys-lru` / `allkeys-lfu` | evict any key by recency / frequency | a pure cache (every key is disposable) |
| `volatile-lru` / `volatile-ttl` / `volatile-lfu` | evict **only keys that have a TTL** | mixed: durable keys without TTL stay |

A "cache" running `noeviction` with no TTLs is a memory leak that starts 500ing writes when full —
the single most common Redis incident. Set `maxmemory` *and* an `allkeys-*` policy for a cache.

**Persistence:** RDB = periodic point-in-time snapshots (fast restart, can lose the last interval);
AOF = append every write (durable, slower, larger). A pure cache often needs *neither* — losing it
just rebuilds from the origin. Decide cache-mode vs database-mode first, then persistence follows.

## Anti-patterns → STOP

| Rationalization | What actually happens | Do instead |
| --- | --- | --- |
| "`INCR` then `EXPIRE` is fine" | a lost `EXPIRE` leaves a TTL-less counter → immortal throttle | one atomic `EVAL` (set TTL when `n == 1`) |
| "I'll `GET` then `DEL` the lock" | races; frees a *different* owner's lock | random token + Lua compare-and-delete |
| "`KEYS user:*` to find my keys" | O(N), blocks the single thread, freezes all clients | `SCAN MATCH user:* COUNT 100` |
| "No TTL, I'll invalidate manually" | you miss a path → stale forever + unbounded memory | TTL on every cache key, always |
| "Redlock means the lock is safe" | a GC pause double-acquires; clock-drift assumptions | fence the resource, or accept double-run on idempotent work |
| "`RPOP` the job into the worker" | worker dies mid-task → job vanishes | `LMOVE` to a processing list, or Streams + `XACK` |
| "Big `MULTI` / long Lua = throughput" | single thread blocks every client for the duration | keep each atomic step short; chunk the work |
| "Fixed-window limiter is exact" | allows ~2× the limit across the window boundary | sliding-window log or token bucket if exactness matters |

## Quick reference

```bash
# Cache:      SET k v EX 300 | GETEX k EX 300 | DEL k | SCAN 0 MATCH 'product:*' COUNT 100
# Lock:       SET k tok NX PX 30000  (release via the Lua compare-and-delete above)
# Rate limit: EVAL "<lua>" 1 rl:user:42 100 60   (limit 100 / 60s)
# Queue:      LMOVE jobs jobs:proc LEFT RIGHT | LREM jobs:proc 1 "$job"
#             XADD jobs '*' field v | XREADGROUP ... | XACK | XAUTOCLAIM

# Diagnostics (read-only):
redis-cli INFO memory          # used_memory, maxmemory, evicted_keys
redis-cli --bigkeys            # find the keys eating your memory
redis-cli SLOWLOG GET 10       # slowest recent commands
redis-cli OBJECT FREQ mykey    # access frequency (needs allkeys-lfu)
redis-cli MEMORY USAGE mykey   # bytes for one key
redis-cli SCAN 0 MATCH 'sess:*' COUNT 100   # never KEYS in prod
```

`scripts/verify.sh` scans your project for the high-confidence foot-guns (`KEYS *` in source,
lock release without a token compare, `INCR` with no `EXPIRE`, `maxmemory` with no policy). It is
read-only, never connects to a server, and exits 0 when no Redis usage is found.

## Project grounding

Record this project's Redis decisions in `02-DOCS/wiki/stack/redis.md` (recorded, not gated — same
convention `postgresdb` uses): which primitive(s) you run, the `maxmemory-policy`, persistence
(RDB/AOF/none), the client library, and any lock/rate-limit Lua you depend on. Future agents read
this before touching the cache.

## See Also

- [`references/caching.md`](references/caching.md) — read/write-through, write-behind, XFetch math, negative caching, hot/big keys.
- [`references/locks-and-rate-limiting.md`](references/locks-and-rate-limiting.md) — full Redlock + critique + fencing, watchdog renewal, all four rate-limit algorithms with Lua.
- [`references/queues.md`](references/queues.md) — full reliable-list + Streams implementations, `XAUTOCLAIM` recovery, DLQ, BullMQ/Sidekiq/RQ/Celery detail, idempotency.
- Siblings: [`../postgresdb/SKILL.md`](../postgresdb/SKILL.md) · [`../vector-db/SKILL.md`](../vector-db/SKILL.md) · [`../scaling/SKILL.md`](../scaling/SKILL.md) · [`../clickhouse-analytics/SKILL.md`](../clickhouse-analytics/SKILL.md).
