{"_id":"@cohenerickson/mtbl","name":"@cohenerickson/mtbl","dist-tags":{"latest":"0.1.0"},"versions":{"0.1.0":{"name":"@cohenerickson/mtbl","version":"0.1.0","description":"Pure-TS streaming reader for mtbl (Sorted String Table) files. Memory-efficient enough for files in the hundreds of GB.","license":"MIT","repository":{"type":"git","url":"git+https://github.com/cohenerickson/mtbl.git"},"homepage":"https://github.com/cohenerickson/mtbl#readme","bugs":{"url":"https://github.com/cohenerickson/mtbl/issues"},"type":"module","main":"./dist/src/index.js","types":"./dist/src/index.d.ts","engines":{"node":">=18.0.0"},"scripts":{"build":"tsc","watch":"tsc --watch","test":"tsc && node --test --import tsx test/reader.test.ts","prepublishOnly":"npm run build"},"exports":{".":{"types":"./dist/src/index.d.ts","import":"./dist/src/index.js"}},"publishConfig":{"access":"public"},"devDependencies":{"@types/node":"^22.10.0","tsx":"^4.19.0","typescript":"^5.7.0"},"peerDependencies":{"snappy":"*","lz4-napi":"*"},"peerDependenciesMeta":{"snappy":{"optional":true},"lz4-napi":{"optional":true}},"_id":"@cohenerickson/mtbl@0.1.0","gitHead":"2751ecfc37574029a3bf9dcf851eb5997c65e04b","_nodeVersion":"22.19.0","_npmVersion":"10.9.3","dist":{"integrity":"sha512-fIBwz0hDF/TjbDRpNzeDaFuCjgBYQUFN36NyZ7dpXXgG3h3VhowxoC3GmG3UbclD39RVsLd7sdRpNqTV2onxdg==","shasum":"c8428f6b80b72eac64f6a4e7f1b2c9b1c771c0a7","tarball":"https://registry.npmjs.org/@cohenerickson/mtbl/-/mtbl-0.1.0.tgz","fileCount":23,"unpackedSize":76648,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIA4e6S8cZyqT0Jti0mMd5WeqmuaTKRR+eIrvTyO5TG5VAiB7WpnkrwLy4Hxyzz9ho/1IUkIcArNW1IuF+kC2iWE88Q=="}]},"_npmUser":{"name":"cohenerickson","email":"cohenerickson@gmail.com"},"directories":{},"maintainers":[{"name":"cohenerickson","email":"cohenerickson@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/mtbl_0.1.0_1777589344654_0.28520647230383545"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-30T22:49:04.547Z","0.1.0":"2026-04-30T22:49:04.897Z","modified":"2026-04-30T22:49:05.137Z"},"maintainers":[{"name":"cohenerickson","email":"cohenerickson@gmail.com"}],"description":"Pure-TS streaming reader for mtbl (Sorted String Table) files. Memory-efficient enough for files in the hundreds of GB.","homepage":"https://github.com/cohenerickson/mtbl#readme","repository":{"type":"git","url":"git+https://github.com/cohenerickson/mtbl.git"},"bugs":{"url":"https://github.com/cohenerickson/mtbl/issues"},"license":"MIT","readme":"# MTBL\n\nPure-TypeScript streaming reader for [mtbl](https://github.com/farsightsec/mtbl) (Sorted String Table) files. Memory-efficient enough for files in the hundreds of GB.\n\n> **Vibe-coded.** This library was drafted in a long AI-assisted session. The architecture is solid and the tests pass, but real-world edge cases may surface. Fresh eyes and contributions are very welcome — please open issues or PRs.\n\n## Requirements\n\nNode.js 18+. zstd compression requires Node 22.15+.\n\n## Install\n\n```bash\nnpm install @cohenerickson/mtbl\n```\n\nFor lz4 or snappy compressed files, also install the relevant peer dep (see [Compression support](#compression-support)).\n\n## Usage\n\n```ts\nimport { MTBLReader } from \"mtbl\";\n\nconst reader = new MTBLReader(\"./images.mtbl\");\nawait reader.ready;\n\n// Point lookup\nconst value = await reader.get(\"some-key\"); // Buffer | null\n\n// Check existence without fetching the value\nconst exists = await reader.has(\"some-key\"); // boolean\n\n// Bulk lookup (sorts keys internally for data block reuse)\nconst map = await reader.getMany([\"k1\", \"k2\", \"k3\"]);\n\n// Iteration — one block in memory at a time regardless of file size\nfor await (const { key, value } of reader.getPrefix(\"photos/2024/\")) {\n  // key and value are Buffers, safe to retain across iterations\n}\n\nawait reader.close();\n```\n\n> **Warning:** Do not accumulate all entries into an array on large files. `Array.fromAsync(reader.iterate())` will OOM on any file that doesn't fit in memory. Consume the async iterator without buffering.\n\n### Full API\n\n```ts\nclass MTBLReader {\n  constructor(path: string, options?: MTBLReaderOptions);\n\n  /** Resolves when the file is open and the index is loaded. */\n  ready: Promise<void>;\n\n  /** File metadata from the on-disk trailer. */\n  metadata(): Promise<MTBLMetadata>;\n\n  // Point access\n  get(key: KeyInput): Promise<Buffer | null>;\n  has(key: KeyInput): Promise<boolean>;\n  getMany(keys: readonly KeyInput[]): Promise<Map<Buffer, Buffer>>;\n\n  // Iteration\n  iterate(options?: IterateOptions): AsyncGenerator<MTBLEntry>;\n  keys(options?: IterateOptions): AsyncGenerator<Buffer>;\n  values(options?: IterateOptions): AsyncGenerator<Buffer>;\n  getPrefix(prefix: KeyInput): AsyncGenerator<MTBLEntry>;\n  getRange(start: KeyInput, end: KeyInput): AsyncGenerator<MTBLEntry>;\n  [Symbol.asyncIterator](): AsyncGenerator<MTBLEntry>;\n\n  // Bounds\n  firstKey(): Promise<Buffer | null>;\n  lastKey(): Promise<Buffer | null>;\n\n  close(): Promise<void>;\n}\n```\n\n`KeyInput` is `Buffer | Uint8Array | string` (strings are UTF-8 encoded). All returned keys and values are `Buffer`.\n\n#### `IterateOptions`\n\n```ts\ninterface IterateOptions {\n  /** Inclusive lower bound. Starts at the first key >= start. */\n  start?: KeyInput;\n  /** Exclusive upper bound. Stops before the first key >= end. */\n  end?: KeyInput;\n  /** Only yield entries whose key begins with this prefix. */\n  prefix?: KeyInput;\n}\n```\n\nIf `prefix` is given alongside `start`/`end`, the bounds are intersected.\n\n#### `MTBLMetadata`\n\n```ts\ninterface MTBLMetadata {\n  version: \"v1\" | \"v2\";\n  compression: \"none\" | \"zlib\" | \"lz4\" | \"lz4hc\" | \"zstd\" | \"snappy\";\n  compressionAlgorithm: number;\n  entryCount: number | bigint;\n  dataBlockCount: number | bigint;\n  bytesDataBlocks: number | bigint;  // compressed\n  bytesIndexBlock: number | bigint;\n  bytesKeys: number | bigint;        // uncompressed\n  bytesValues: number | bigint;      // uncompressed\n  fileSize: number;\n}\n```\n\nValues are `bigint` when they exceed `Number.MAX_SAFE_INTEGER`.\n\n## Compression support\n\n| Algorithm | Decoder                        | Status      |\n| --------- | ------------------------------ | ----------- |\n| `none`    | pass-through                   | built-in    |\n| `zlib`    | `node:zlib`                    | built-in    |\n| `zstd`    | `node:zlib` (`zstdDecompress`) | Node 22.15+ |\n| `lz4`     | `lz4-napi` peer dep            | optional    |\n| `lz4hc`   | `lz4-napi` peer dep            | optional    |\n| `snappy`  | `snappy` peer dep              | optional    |\n\nIf your file uses lz4 or snappy, install the relevant peer dep:\n\n```bash\nnpm install lz4-napi    # for lz4 / lz4hc\nnpm install snappy      # for snappy\n```\n\n## Architecture\n\nThe reader is structured in layers:\n\n- `file-handle.ts` — thin wrapper around `fs.promises.open` providing positional reads. The only module that touches the filesystem.\n- `varint.ts` — varint and fixed-width LE integer decoders.\n- `trailer.ts` — parses the 512-byte trailer at the end of the file.\n- `compression.ts` — dispatches block decompression by algorithm.\n- `block.ts` — parses an individual decompressed block (the LevelDB-style prefix-compressed layout with restart-point binary search). This is the workhorse module.\n- `framed-block.ts` — reads a single block envelope from disk (`[varint length][u32 crc][payload]`), decompresses, and constructs a `Block`.\n- `index-block.ts` — special handling for the index block, which maps keys to data block offsets.\n- `reader.ts` — the public `MTBLReader` class. Pulls all the layers together.\n\n### Memory model\n\nFor a 500 GB file with default 8 KB data blocks, there are roughly 65 million data blocks. The on-disk index is typically under 1% of file size (~2.5–5 GB for a 500 GB file).\n\nThe current implementation loads the full index block into memory at open time, then reads exactly one data block per `get()` call from disk. As long as iteration consumers process entries as they arrive, per-operation memory stays flat: one ~8 KB compressed block + one decompressed block (~30–50 KB) at a time.\n\nIf the index is too large for memory, the path forward is a sparse or on-demand index inside `index-block.ts` — only `findBlockForKey` would need to change.\n\n## Format reference\n\nImplementation derived from reading the C source directly: [farsightsec/mtbl](https://github.com/farsightsec/mtbl) (specifically `reader.c`, `block.c`, `metadata.c`, `varint.c`, `compression.c`).\n\nThe format is a LevelDB-derived SSTable:\n\n- File ends with a fixed 512-byte trailer (magic `MTBL` for V2, fields are little-endian u64s).\n- Trailer points to an index block, which is a regular block whose values are varint64 offsets to data blocks.\n- Each block is framed on disk as `[varint64 length][u32 crc32c][payload]` (V2). V1 used a fixed u32 length instead of a varint.\n- Block payloads use LevelDB-style prefix-compressed entries (`[varint shared][varint non_shared][varint value_length][suffix][value]`) with a restart array at the end for binary search.\n\n## Contributing\n\nIssues and PRs are welcome. If you have a real mtbl file written by the reference C library (`mtbl_create` or similar), adding it as a test fixture would be particularly valuable — the test suite currently only exercises files written by the TypeScript fixture writer in `test/fixture.ts`.\n","readmeFilename":"README.md","_rev":"1-8086d0a16d5b01b1dd2c29d2cc96fee1"}