{"_id":"@aoede/tamper","name":"@aoede/tamper","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@aoede/tamper","version":"1.0.0","description":"ESM encoder/decoder for Tamper - a compact format for bulk categorical datasets","type":"module","keywords":["tamper","compression","categorical","tabular","encoding","bitmap","rle","esm"],"author":{"name":"John Smith | Aoede","email":"tamper@aoede.uk.net","url":"https://www.aoede.uk.net"},"license":"Apache-2.0","repository":{"type":"git","url":"git+https://github.com/aoede3/tamper.git"},"bugs":{"url":"https://github.com/aoede3/tamper/issues"},"homepage":"https://github.com/aoede3/tamper#readme","engines":{"node":">=18.0.0"},"exports":{"./encoder":{"types":"./encoders/js/index.ts","default":"./dist/encoders/js/index.js"},"./encoder/browser":{"types":"./encoders/js/env/browser.ts","default":"./dist/encoders/js/env/browser.js"},"./decoder":{"types":"./clients/js/src/tamper.ts","default":"./dist/clients/js/src/tamper.js"}},"scripts":{"build":"tsup","ci":"npm run build && npm run typecheck && npm run test","test:decoders":"tsx ./scripts/compare-decoders.ts","test:encoders":"tsx ./scripts/compare-encoders.ts","test:legacy":"tsx ./scripts/compare-decoders.ts && tsx ./scripts/compare-encoders.ts","test":"vitest run","test:unit":"vitest run tests/unit","test:integration":"vitest run tests/integration","test:coverage":"vitest run --coverage","test:ui":"vitest --ui --coverage","test:watch":"vitest","inspect":"tsx ./scripts/inspect-pack.ts","example":"tsx ./scripts/size-comparison.ts","typecheck":"tsc --noEmit","format":"prettier --write .","check-format":"prettier --check .","check-exports":"attw --pack . --ignore-rules=cjs-resolves-to-esm","prepublishOnly":"npm run ci","local-release":"npm run ci && changeset version && changeset publish"},"devDependencies":{"@changesets/cli":"^2.27.0","@types/node":"^20.12.7","@vitest/coverage-v8":"^2.1.8","@vitest/ui":"^2.1.9","prettier":"^3.3.3","tsup":"^8.0.0","tsx":"^4.21.0","typescript":"^5.5.4","vitest":"^2.1.8"},"gitHead":"59d832b6f26fd1432433ef0e973ecb7550162b18","_id":"@aoede/tamper@1.0.0","_nodeVersion":"25.3.0","_npmVersion":"11.6.2","dist":{"integrity":"sha512-yIUmCfjVmkErtXX+rrRomAUYIkrJEkOkwiqhVbkjIpjqBYC3xdXcXciFY29HTz0ur465ubAeV1l3RFVz1fAupA==","shasum":"3355f40efbd1abab3995911c84ed2063a584c5a2","tarball":"https://registry.npmjs.org/@aoede/tamper/-/tamper-1.0.0.tgz","fileCount":31,"unpackedSize":249394,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCv6DKgEgwUmbMcM+9dBy9SWNgTiyN13lSduAWYSHHjXQIhAN0PgNPCCy8tuj6d3jlAUmnARJbTWy9ZPLm7ZV3gUE6z"}]},"_npmUser":{"name":"aoede","email":"npm@aoede.uk.net"},"directories":{},"maintainers":[{"name":"aoede","email":"npm@aoede.uk.net"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tamper_1.0.0_1770234379745_0.20962349095910904"},"_hasShrinkwrap":false}},"time":{"created":"2026-02-04T19:46:19.583Z","1.0.0":"2026-02-04T19:46:19.945Z","modified":"2026-02-04T19:46:20.154Z"},"maintainers":[{"name":"aoede","email":"npm@aoede.uk.net"}],"description":"ESM encoder/decoder for Tamper - a compact format for bulk categorical datasets","homepage":"https://github.com/aoede3/tamper#readme","keywords":["tamper","compression","categorical","tabular","encoding","bitmap","rle","esm"],"repository":{"type":"git","url":"git+https://github.com/aoede3/tamper.git"},"author":{"name":"John Smith | Aoede","email":"tamper@aoede.uk.net","url":"https://www.aoede.uk.net"},"bugs":{"url":"https://github.com/aoede3/tamper/issues"},"license":"Apache-2.0","readme":"# Tamper (ESM)\n\n**ESM encoder/decoder for Tamper - a compact format for bulk categorical datasets.**\n\nThis repository contains an ESM-native implementation of the Tamper encoder and decoder format originally developed at the New York Times, plus strict parity tooling to ensure identical output to the frozen legacy implementation.\n\nThis project is an independent ESM implementation of the Tamper format. It does not define a new format and is not affiliated with the original NYT repository.\n\nTamper is a **column-oriented packer for tabular categorical data** (low-cardinality enums, booleans, bucketed integers) where JSON + compression becomes inefficient.\n\n## References\n\n- [Original NYT Tamper repository](https://github.com/nytimes/tamper)\n- [NYT Tamper project documentation](https://nytimes.github.io/tamper/)\n- [Original NYT announcement / background](https://archive.nytimes.com/open.blogs.nytimes.com/2014/04/16/introducing-pourover-and-tamper/)\n- [unsetbit/tamp community encoder implementation](https://github.com/unsetbit/tamp)\n\n---\n\n## When to use this\n\nTamper is a good fit when your data is:\n\n- **Tabular** (many rows with the same attributes)\n- **Categorical-heavy** (enums, booleans, small integers)\n- **Bulk** (transferred or stored as snapshots)\n- **Read-mostly / immutable**\n- Required to match **legacy Tamper output exactly**\n\nUse cases:\n\n- Analytics extracts for dashboards\n- Lookup / reference tables\n- ML-style categorical feature matrices shipped to JS or WASM\n\n## When _not_ to use this\n\nDo **not** use Tamper for:\n\n- Nested or hierarchical objects\n- General APIs or CRUD payloads\n- Arbitrary graphs\n- Free-form documents or HTML\n\nIf your data is not mostly categorical and tabular, JSON + Brotli/Zstd or a schema-based format (e.g. Protobuf, Arrow) will likely be a better fit.\n\n---\n\n## Overview\n\n**Tamper** is a data serialisation protocol originally developed at the New York Times to efficiently transfer large categorical datasets from server to browser.\n\nThis repository provides a **modern ESM implementation** of the original CommonJS codebase, with:\n\n- identical encoded output\n- identical decoded results\n- strict, automated parity checks against the frozen legacy implementation\n\n---\n\n## Core encoding approach\n\nTamper packs categorical columns using bitwise encodings, automatically selecting the most efficient strategy per attribute:\n\n- **Integer packing** - sparse or bounded integer values\n- **Bitmap packing** - dense categorical values\n- **Existence packing** - tracks presence using run-length encoding\n\nThese strategies are chosen automatically by the encoder based on observed data characteristics.\n\n---\n\n## Performance\n\nTamper achieves significant compression for categorical tabular data:\n\n- **Sparse datasets**: 10-15x compression (e.g., 500 events across 10K IDs)\n- **Dense multi-value attributes**: 20-30x compression (bitmap encoding)\n- **Very sparse datasets**: 4-5x compression at scale (existence encoding with RLE)\n\nThe compression ratio improves with dataset size due to fixed header overhead. See real examples with the size comparison script:\n\n```bash\nnpm run example\n```\n\nThis script demonstrates four scenarios showing Tamper vs plain JSON size, compression ratios, and the impact of:\n\n- Existence encoding for sparse data\n- Integer encoding for categorical values\n- Bitmap encoding for multi-value attributes\n- Fixed overhead on small vs large datasets\n\n**Note:** These compression ratios are before any transport-level compression. Tamper packs can be further compressed with gzip/brotli for additional gains, often achieving better overall compression than gzip/brotli on plain JSON (due to Tamper's elimination of field name repetition and use of bit-packed encodings).\n\n---\n\n## Repository structure\n\n```\n├── clients/js/src/         # ESM decoder (browser-side)\n├── encoders/js/\n│   ├── core/               # Environment-agnostic encoder logic\n│   └── env/                # Node.js & browser adapters\n├── legacy/                 # Frozen legacy implementation (reference only)\n├── vendor/bitsy/           # Vendored bitset library (no npm deps)\n├── scripts/                # Parity verification tools\n└── test/                   # Test datasets & canonical outputs\n```\n\n---\n\n## Requirements\n\n- Node.js (ESM-capable; tested with current LTS)\n- npm (for installing dev tooling)\n- Encoder runtime uses a local `vendor/bitsy` shim (no network installs)\n\nInstall dev dependencies for TSX-driven scripts:\n\n```bash\nnpm install\n```\n\n---\n\n## Usage\n\n### Decoder (ESM)\n\nExports:\n\n- `createTamper()` - decoder factory\n- `Tamper` - decoder methods\n- default export - alias of `createTamper`\n\n```js\nimport createTamper from \"./clients/js/src/tamper.ts\";\nimport fs from \"node:fs/promises\";\n\nconst tamper = createTamper();\nconst pack = JSON.parse(await fs.readFile(\"pack.json\", \"utf8\"));\nconst items = tamper.unpackData(pack);\n```\n\n---\n\n### Encoder (ESM)\n\nEntry points:\n\n- Node / standard ESM: `encoders/js/index.ts`\n- Browser / edge: compose core + environment adapter\n\nExports:\n\n- `createPackSet`, `PackSet`\n- `Pack`, `IntegerPack`, `BitmapPack`, `ExistencePack`\n\n```js\nimport { createPackSet } from \"./encoders/js/index.ts\";\n\nconst tamp = createPackSet();\n// configure attributes + pack data...\nconst json = tamp.toJSON();\n```\n\nBrowser / edge example:\n\n```js\nimport createEncoder from \"./encoders/js/core/createEncoder.ts\";\nimport browserEnv from \"./encoders/js/env/browser.ts\";\n\nconst { createPackSet } = createEncoder(browserEnv);\n\nconst tamp = createPackSet();\n// configure attributes + pack data...\nconst json = tamp.toJSON();\n```\n\n---\n\n## Parity verification (strict)\n\nDecoder parity compares decoded output from the legacy and ESM implementations:\n\n```bash\ntsx scripts/compare-decoders.ts\n```\n\nEncoder parity builds packs from test datasets and compares full JSON output against canonical fixtures:\n\n```bash\ntsx scripts/compare-encoders.ts\n```\n\nThe ESM implementation's parity is verified by ensuring all canonical fixtures match byte-for-byte.\n\n---\n\n## Notes\n\n- Encoder output is tuned to exactly match canonical JSON fixtures (including legacy fields such as `max_guid` and existence metadata).\n- The legacy implementation is retained **only** for parity verification and reference; it is not used at runtime.\n- The browser encoder uses `Uint8Array` and `DataView` and does not depend on Node.js `Buffer`.\n\n---\n\n## Expected output\n\n```text\nPASS large.json\nPASS run.json\nPASS run2.json\nPASS small.json\nPASS small2.json\nPASS sparse.json\nPASS spstart.json\n\nAll 7 file(s) passed parity checks.\n...\nAll 7 file(s) passed encoder parity checks.\n```\n","readmeFilename":"README.md","_rev":"1-66d270457687e17bf3db886437d010c6"}