{"_id":"@altaf007/queuectl","name":"@altaf007/queuectl","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@altaf007/queuectl","version":"1.0.0","description":"CLI-based background job queue system","bin":{"queuectl":"bin/queuectl.js"},"directories":{"doc":"docs"},"scripts":{"test":"node test/run-all.js"},"keywords":[],"author":"","license":"ISC","type":"commonjs","dependencies":{"better-sqlite3":"^12.11.1","commander":"^15.0.0"},"gitHead":"72f7e1d767f718c14e803c6da9db143a9b89dc6d","_id":"@altaf007/queuectl@1.0.0","_nodeVersion":"24.18.0","_npmVersion":"11.16.0","dist":{"integrity":"sha512-MpiHUfID/ALw1axXNehRm1Bo9NJqOSMf9jYOSiURW0mE4SMrLpUSvsNHXgc2XE/JW37iTGYYYHJEC4dzs5HNfQ==","shasum":"69ae27713754ae0577bfb42d1733091a2aad7646","tarball":"https://registry.npmjs.org/@altaf007/queuectl/-/queuectl-1.0.0.tgz","fileCount":19,"unpackedSize":91707,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCID5Mic3Taqyq2jp5N4KW8DwfIAnyLDdSARVU+lOkXFqjAiAYMphBD8oTu2CTkyvRtTWEvua3eckesXFcIZULGhQpVA=="}]},"_npmUser":{"name":"altaf007","email":"altafraja01076@gmail.com"},"maintainers":[{"name":"altaf007","email":"altafraja01076@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/queuectl_1.0.0_1783868519676_0.6462622607141641"},"_hasShrinkwrap":false}},"time":{"created":"2026-07-12T15:01:59.444Z","1.0.0":"2026-07-12T15:01:59.841Z","modified":"2026-07-12T15:02:00.120Z"},"maintainers":[{"name":"altaf007","email":"altafraja01076@gmail.com"}],"description":"CLI-based background job queue system","keywords":[],"license":"ISC","readme":"# queuectl — CLI-Based Background Job Queue System\n\nA minimal, production-grade job queue system that runs entirely via the command line. Workers execute shell commands as background jobs with automatic retries, exponential backoff, a Dead Letter Queue (DLQ), and persistent SQLite storage.\n\nBuilt for a backend internship assignment. Tech stack: Node.js, better-sqlite3, commander.\n\n---\n\n## Setup\n\n```bash\ngit clone <repo-url> queuectl\ncd queuectl\nnpm install\n```\n\nRun commands directly:\n\n```bash\nnode bin/queuectl.js --help\n```\n\nOr link globally for `queuectl` on your PATH:\n\n```bash\nnpm link\nqueuectl --help\n```\n\n---\n\n## Usage\n\n### Enqueue a Job\n\n```bash\nnode bin/queuectl.js enqueue '{\"id\":\"demo\",\"command\":\"echo hello world\"}'\n```\n\nOutput (to stderr):\n\n```\nJob demo enqueued (pending)\n```\n\n### Start Workers\n\nStart a single worker in the foreground (blocks until stopped):\n\n```bash\nnode bin/queuectl.js worker start\n```\n\nStart multiple concurrent poll loops in one process:\n\n```bash\nnode bin/queuectl.js worker start --count 3\n```\n\n### Stop Workers\n\nFrom a **separate terminal** (no shared memory):\n\n```bash\nnode bin/queuectl.js worker stop\n```\n\nOutput:\n\n```\nSent SIGTERM to worker 63175\n```\n\n### Check Status\n\n```bash\nnode bin/queuectl.js status\n```\n\n```\npending      0\nprocessing   0\ncompleted    2\nfailed       0\ndead         1\nactive workers 0\n```\n\n### List Jobs\n\nHuman-readable table:\n\n```bash\nnode bin/queuectl.js list --state completed\n```\n\n```\nj1               echo a                   completed    0  /3   2026-07-11T21:11:08\nj2               echo b                   completed    0  /3   2026-07-11T21:11:08\n```\n\nJSON output (stdout-only, pipes cleanly through `jq`):\n\n```bash\nnode bin/queuectl.js list --state pending --json | jq .\n```\n\n```json\n[\n  {\n    \"id\": \"j3\",\n    \"command\": \"sleep 10\",\n    \"state\": \"pending\",\n    \"attempts\": 1,\n    \"max_retries\": 3,\n    \"created_at\": \"2026-07-11T21:11:08.732Z\"\n  }\n]\n```\n\n### Dead Letter Queue\n\nList dead jobs:\n\n```bash\nnode bin/queuectl.js dlq list\n```\n\n```\ndlq-demo  attempts=2/2  error=\"exit code 1\"  updated=2026-07-11T20:57:58.316Z\n```\n\nRetry a dead job (resets `attempts` to 0):\n\n```bash\nnode bin/queuectl.js dlq retry dlq-demo\n```\n\n```\nJob dlq-demo retried (pending)\n```\n\n### Configuration\n\n```bash\nnode bin/queuectl.js config set max-retries 5\nnode bin/queuectl.js config set backoff-base 2\nnode bin/queuectl.js config list\n```\n\n```\nmax_retries=5\nbackoff_base=2\nstale_timeout_seconds=15\npoll_interval_ms=2000\n```\n\nConfig keys accept hyphens or underscores interchangeably (e.g. `max-retries` ↔ `max_retries`).\n\n---\n\n## Publishing to npm\n\nTo distribute `queuectl` as a globally installable command-line tool via the npm registry:\n\n### 1. Package Configuration\nEnsure your `package.json` specifies the binary name and path under the `\"bin\"` field, and includes only the necessary directory files:\n```json\n{\n  \"name\": \"queuectl\", // Note: you might need to use a scoped name like \"@username/queuectl\" if the name is taken\n  \"version\": \"1.0.0\",\n  \"bin\": {\n    \"queuectl\": \"bin/queuectl.js\"\n  },\n  \"files\": [\n    \"bin\",\n    \"src\"\n  ]\n}\n```\n\n### 2. Login and Publish\n1. Register for an npm account on [npmjs.com](https://www.npmjs.com/).\n2. Log in from your terminal:\n   ```bash\n   npm login\n   ```\n3. Publish your package:\n   ```bash\n   # If using a standard name:\n   npm publish\n   \n   # If using a scoped name (e.g., @username/queuectl):\n   npm publish --access public\n   ```\n\n### 3. Install and Run\nOnce published, users can install it globally from anywhere:\n```bash\nnpm install -g queuectl\nqueuectl --help\n```\n\n---\n\n## Architecture\n\n### Directory Structure\n\n```\nqueuectl/\n├── bin/queuectl.js      CLI entrypoint (commander)\n├── src/\n│   ├── db.js            SQLite connection + schema\n│   ├── config.js        Key-value config backed by config table\n│   ├── queue.js         Queue operations (enqueue, DLQ, list, status)\n│   ├── worker.js        Worker poll loop, claim, exec, sweep\n│   └── commands/        Reserved for future command modules\n├── test/                Test suite (6 tests, run via npm test)\n├── .queuectl/           Runtime data (auto-created)\n│   ├── queuectl.db      SQLite database\n│   └── workers/         PID files for worker discovery\n├── .gitignore\n├── package.json\n├── README.md\n└── DECISIONS.md\n```\n\n### Job Lifecycle\n\n```\n                  ┌──────────┐\n                  │  pending │◄────────────────────────────┐\n                  └────┬─────┘                              │\n                       │ claim (atomic UPDATE)              │\n                       ▼                                    │\n                  ┌──────────┐                              │\n                  │processing│                              │\n                  └────┬─────┘                              │\n                    ┌──┴──┐                                 │\n                    ▼      ▼                                │\n              ┌────────┐ ┌──────┐                           │\n              │completed│ │failed│──retry (backoff)─────────┘\n              └────────┘ └──┬───┘        ◄── alive again\n                            │ exhausted\n                            ▼\n                       ┌──────┐\n                       │ dead │ (DLQ)\n                       └──────┘\n```\n\nA crashed (SIGKILL'd) worker leaves a job stuck in `processing`. The stale-job sweep (`sweepStaleJobs`) resets it to `pending` after `stale_timeout_seconds` without incrementing `attempts`.\n\n### Persistence\n\nSQLite via better-sqlite3, WAL mode for concurrent reads. Two tables:\n\n- **`jobs`** — `id`, `command`, `state`, `attempts`, `max_retries`, `next_attempt_at`, `claimed_at`, `worker_pid`, `last_error`, timestamps\n- **`config`** — `key`, `value`\n\nThe database file lives at `.queuectl/queuectl.db` and survives process restarts.\n\n### Worker Concurrency\n\n- `--count N` spawns N concurrent async poll loops **within a single process** (one signal handler, one PID registration)\n- **Real cross-process safety** (multiple `worker start` terminals) is guaranteed by the atomic claim query, which SQLite serializes via file-level write locks\n- Workers must be stopped gracefully via `worker stop`; SIGKILL is handled by crash recovery\n\n### Graceful Shutdown\n\nOn SIGTERM or SIGINT (Ctrl+C):\n\n1. Worker sets a `shuttingDown` flag\n2. Current in-flight `exec` child continues to completion\n3. Worker deregisters its PID file\n4. Poll loop detects flag and calls `process.exit(0)`\n\nNo new job is claimed after the flag is set.\n\n### Stale-Job Recovery (Crash)\n\n`sweepStaleJobs()` runs on every worker startup and every poll iteration. It finds jobs in `processing` where `claimed_at` is older than `stale_timeout_seconds` and resets them to `pending`. `attempts` is NOT incremented — a crashed worker is not a failed command.\n\n---\n\n## Assumptions & Trade-offs\n\n### SQLite over plain JSON\n\nSQLite provides atomic transactions, concurrent access (WAL mode), and built-in indexing — all essential for correct multi-worker behavior. JSON file storage would require implementing our own locking and atomic writes. The cost: a native binary dependency (`better-sqlite3`).\n\n### Duplicate ID on enqueue: Reject\n\nIf two enqueues use the same `id`, the second is rejected with an error. Silently overwriting could destroy a pending job the user didn't intend to replace. Since job IDs are user-chosen, the user can pick a different ID. This is at-most-once semantics on create.\n\n### Config changes: per-job vs live\n\n- **`max_retries`**: captured per-job at enqueue time. A job created under the old config keeps its own limit. Changing the config doesn't retroactively affect already-enqueued jobs.\n- **`backoff_base`**: read live from config at each retry. This is a global timing policy, not a per-job contract. Changing it affects the next scheduled retry of every in-flight job.\n\n### Stale-job sweep doesn't increment attempts\n\nA crashed worker (SIGKILL) didn't run the command — we literally don't know if it would have succeeded or failed. Incrementing `attempts` would unfairly consume the retry budget. The job is re-queued as if never picked up.\n\n### `dlq retry` resets attempts to 0\n\nA manual retry from the DLQ is an explicit human decision to \"try this job from scratch.\" Leaving `attempts` at the exhausted value would immediately re-exhaust it on the next failure, making the retry pointless.\n\n### PID recycling risk\n\nPID files (`worker stop` → `kill(pid, SIGTERM)`) have a fundamental limitation: if a worker crashes and its PID is reassigned by the OS to an unrelated process, `worker stop` would signal the wrong process. Acceptable for a local development tool. Production systems use socket-based process supervision (e.g. systemd).\n\n---\n\n## Testing\n\n```bash\nnpm test\n```\n\nRuns 6 tests sequentially:\n\n| Test | What it covers |\n|---|---|\n| `unit-queue.js` | Enqueue validation, claim atomicity, state transitions, DLQ retry |\n| `e2e-basic.js` | A successful job completes with attempts=0 |\n| `e2e-retry-dlq.js` | Failing job retries with backoff, reaches DLQ, `dlq retry` works |\n| `e2e-concurrency.js` | 3 real separate OS processes process 30 jobs exactly once |\n| `e2e-crash-recovery.js` | `kill -9` mid-job → new worker recovers within 30s |\n| `e2e-restart.js` | Job data survives full process restart |\n\n### Manual Verification\n\n**Scenario 1 — Basic job:**\n\n```bash\nnode bin/queuectl.js enqueue '{\"id\":\"test1\",\"command\":\"echo hello\"}'\nnode bin/queuectl.js worker start\n# Ctrl+C after job completes\nnode bin/queuectl.js list --state completed\n```\n\n**Scenario 2 — Retry + DLQ:**\n\n```bash\nnode bin/queuectl.js enqueue '{\"id\":\"test2\",\"command\":\"exit 1\",\"max_retries\":2}'\nnode bin/queuectl.js worker start\n# After ~10s Ctrl+C\nnode bin/queuectl.js dlq list\n# Should show 2/2 attempts\nnode bin/queuectl.js dlq retry test2\n```\n\n**Scenario 3 — Parallel workers (separate terminals):**\n\n```\nTerminal A: node bin/queuectl.js worker start\nTerminal B: node bin/queuectl.js worker start\nTerminal C: node bin/queuectl.js enqueue '{\"id\":\"p1\",\"command\":\"sleep 2\"}'\n            node bin/queuectl.js enqueue '{\"id\":\"p2\",\"command\":\"sleep 2\"}'\n# Both process concurrently without overlap\n```\n\n**Scenario 4 — Crash recovery:**\n\n```bash\n# Terminal A\nnode bin/queuectl.js enqueue '{\"id\":\"crash1\",\"command\":\"sleep 20\"}'\nnode bin/queuectl.js worker start\n# Terminal B: kill -9 <PID of terminal A's worker>\n# Terminal A again\nnode bin/queuectl.js worker start\n# Job completes within ~22s\n```\n\n**Scenario 5 — Persistence:**\n\n```bash\nnode bin/queuectl.js enqueue '{\"id\":\"p1\",\"command\":\"echo survived\"}'\n# Kill the entire process\nnode bin/queuectl.js worker start  # Job is still there, gets processed\n```\n\n---\n\n## Demo Recording\n\n<!-- TODO: Upload demo video and insert link here -->\n\n---\n\n## License\n\nMIT\n","readmeFilename":"README.md","_rev":"1-dbe2bbddafa4c331d656280b6ef1916b"}