{"_id":"@caifrazier/atls-sdk","name":"@caifrazier/atls-sdk","dist-tags":{"rc":"1.0.0-rc.1","latest":"1.0.0-rc.1"},"versions":{"1.0.0-rc.1":{"name":"@caifrazier/atls-sdk","version":"1.0.0-rc.1","description":"TypeScript SDK for reading and validating Atlas (.atls) archives","author":{"name":"Cai Frazier"},"license":"Apache-2.0","type":"module","keywords":["atlas","crawler","seo","accessibility","sdk","archive","validation"],"repository":{"type":"git","url":"git+https://github.com/CaiFrazier/atls.git","directory":"packages/atls-sdk"},"bugs":{"url":"https://github.com/CaiFrazier/atls/issues"},"homepage":"https://github.com/CaiFrazier/atls/tree/main/packages/atls-sdk#readme","main":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.js"}},"publishConfig":{"access":"public"},"dependencies":{"@mongodb-js/zstd":"^2.0.1","ajv":"^8.17.1","ajv-formats":"^3.0.1","fzstd":"0.1.1","yauzl":"^3.1.3","@caifrazier/atls-spec":"1.0.0-rc.1"},"devDependencies":{"@types/archiver":"^6.0.3","@types/node":"^20.0.0","@types/yauzl":"^2.10.3","archiver":"^7.0.1","typescript":"^5.6.3"},"scripts":{"build":"tsc -b","dev":"tsc -b --watch","test":"VITEST_POOL=forks VITEST_MIN_WORKERS=1 VITEST_MAX_WORKERS=1 vitest run --pool=forks --minWorkers=1 --maxWorkers=1","lint":"eslint .","clean":"rimraf dist tsconfig.tsbuildinfo"},"_id":"@caifrazier/atls-sdk@1.0.0-rc.1","_integrity":"sha512-vX6rNb2F9cleBnTo9LsOfU4YadvzNQCBpklaT6iTnrQrNlcOScIPk+2zzWbEfzr54UYRg7wcV+6JSGZhXeKFAQ==","_resolved":"/private/var/folders/vw/qq7ngvrs48717pvqb7n89r640000gn/T/b999fbdf9dfcf136f3e4a572b3956881/caifrazier-atls-sdk-1.0.0-rc.1.tgz","_from":"file:caifrazier-atls-sdk-1.0.0-rc.1.tgz","_nodeVersion":"25.4.0","_npmVersion":"11.7.0","dist":{"integrity":"sha512-vX6rNb2F9cleBnTo9LsOfU4YadvzNQCBpklaT6iTnrQrNlcOScIPk+2zzWbEfzr54UYRg7wcV+6JSGZhXeKFAQ==","shasum":"859c84c959a94469bc31b03e485a464271840146","tarball":"https://registry.npmjs.org/@caifrazier/atls-sdk/-/atls-sdk-1.0.0-rc.1.tgz","fileCount":43,"unpackedSize":220908,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIDjVD0ZMwueR8FXYckC5BOQ4/P1nTckWgVH+BT1L6FrOAiBUz/7kA9fKk6xz1BwVu3eFE23YkntD9qxkeBS3TW7Ojw=="}]},"_npmUser":{"name":"scottfultz","email":"scott.fultz@caifrazier.com"},"directories":{},"maintainers":[{"name":"scottfultz","email":"scott.fultz@caifrazier.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/atls-sdk_1.0.0-rc.1_1783153841160_0.7197400860132659"},"_hasShrinkwrap":false}},"time":{"created":"2026-07-04T08:30:40.998Z","1.0.0-rc.1":"2026-07-04T08:30:41.301Z","modified":"2026-07-04T08:30:41.561Z"},"maintainers":[{"name":"scottfultz","email":"scott.fultz@caifrazier.com"}],"description":"TypeScript SDK for reading and validating Atlas (.atls) archives","homepage":"https://github.com/CaiFrazier/atls/tree/main/packages/atls-sdk#readme","keywords":["atlas","crawler","seo","accessibility","sdk","archive","validation"],"repository":{"type":"git","url":"git+https://github.com/CaiFrazier/atls.git","directory":"packages/atls-sdk"},"author":{"name":"Cai Frazier"},"bugs":{"url":"https://github.com/CaiFrazier/atls/issues"},"license":"Apache-2.0","readme":"# @caifrazier/atls-sdk\n\nTypeScript SDK for reading and validating Atlas `.atls` archives.\n\n## API\n\n> **Full API reference:** [`docs/API.md`](./docs/API.md) — every method, option, and return type.\n> The Atlas archive **format** itself is specified in\n> [`@cf/atlas-spec`](../atlas-spec/SPECIFICATION.md).\n\nPrimary exports:\n\n- `openAtlas(atlsPath, options?)`\n- `select(atlsPath, options)`\n- `validate(atlsPath, options)`\n- `diffManifests` / `formatManifestDiff`, `buildCrossModeIndex`\n- Low-level: `readManifest`, `readSummary`, `iterateParts`, `readBlob`, `openZipHandle`\n- `MalformedRecordError`\n- `AtlasResourceLimitError`\n\n## Quick Example\n\n```ts\nimport { openAtlas } from '@caifrazier/atls-sdk';\n\nconst atlas = await openAtlas('./crawl.atls');\n\nconsole.log(atlas.manifest.atlasVersion);\nconsole.log([...atlas.datasets]);\n\nfor await (const page of atlas.readers.pageMetadata()) {\n  console.log(page.url);\n}\n\natlas.close();\n```\n\n## Read Limits\n\nAll archive read APIs apply bounded ZIP, zstd, and JSONL limits by default. Oversized entries,\ndataset parts, blobs, root JSON files, and JSONL lines throw `AtlasResourceLimitError`.\n\nTrusted local workflows can raise limits explicitly:\n\n```ts\nconst atlas = await openAtlas(\"./large-local-crawl.atls\", {\n  limits: {\n    maxPartDecompressedBytes: 1024 * 1024 * 1024,\n    maxBlobDecompressedBytes: 512 * 1024 * 1024,\n  },\n});\n```\n\n## Reader Surface\n\n`openAtlas()` returns an `AtlasReader` with:\n\n- `manifest`\n- `summary`\n- `datasets`\n- `packs`\n- `stats` — `{ corruptedRecordCount }` (incremented when `strict: false`)\n- `hasDataset(name)` — returns `true` if the dataset is present in this archive\n- `getBlob(ref)` — fetch a content-addressed blob by its `bodyBlobRef` string\n- `close()`\n- `readers.*` — typed async iterables per dataset (see table below)\n\nSource: `src/index.ts`\n\n### `readers` methods\n\n| Method | Return type | Notes |\n|---|---|---|\n| `navigation()` | `AsyncIterable<Record>` | Link graph edges |\n| `pageMetadata()` | `AsyncIterable<FlatPageMetadata>` | On-page SEO signals |\n| `content()` | `AsyncIterable<Record>` | Full page text and headings |\n| `structuredData()` | `AsyncIterable<Record>` | JSON-LD / microdata / OpenGraph |\n| `technology()` | `AsyncIterable<Record>` | Detected tech stack per page |\n| `renderState()` | `AsyncIterable<Record>` | Full-mode render state (`full` only) |\n| `security()` | `AsyncIterable<Record>` | Security headers and HTTPS posture |\n| `performance()` | `AsyncIterable<Record>` | Performance timing (`prerender`/`full`) |\n| `pipelineFailures()` | `AsyncIterable<Record>` | Hard crawl failures |\n| `pipelineAlerts()` | `AsyncIterable<Record>` | Non-fatal warnings |\n| `assets()` | `AsyncIterable<AssetRecord>` | Images, scripts, fonts, stylesheets |\n| `events()` | `AsyncIterable<EventRecord>` | Crawl lifecycle events |\n| `accessibility()` | `AsyncIterable<AccessibilityRecord>` | WCAG 2.2 audit results |\n| `responses()` | `AsyncIterable<ResponseRecordV1>` | HTTP response metadata |\n| `resources()` | `AsyncIterable<ResourceRecordV1>` | Resource timing and network requests |\n| `dataset<T>(name)` | `AsyncIterable<T>` | Generic accessor for any dataset by key |\n\nDeprecated aliases (still functional, do not use in new code):\n\n| Method | Canonical equivalent |\n|---|---|\n| `pages()` | `pageMetadata()` |\n| `edges()` | `navigation()` |\n| `errors()` | `pipelineFailures()` |\n| `failedPages()` | `pipelineFailures()` |\n\n## Error Policy for Malformed Records\n\nBy default, `openAtlas` operates in **strict mode** (`strict: true`). If a\ndataset iterator encounters a record that cannot be parsed (invalid JSON or\nschema validation failure), it throws a `MalformedRecordError` with the\n`dataset` name and a human-readable `reason`.\n\nTo opt into lenient processing, pass `{ strict: false }`. Malformed records are\nskipped and counted in `atlas.stats.corruptedRecordCount`, so callers always\nknow how many records were lost.\n\n```ts\nimport { openAtlas, MalformedRecordError } from '@caifrazier/atls-sdk';\n\n// Strict (default) — throws on first bad record\ntry {\n  const atlas = await openAtlas('./crawl.atls');\n  for await (const page of atlas.readers.pageMetadata()) {\n    console.log(page.url);\n  }\n} catch (err) {\n  if (err instanceof MalformedRecordError) {\n    console.error(`Bad record in ${err.dataset}: ${err.reason}`);\n  }\n}\n\n// Lenient — skip bad records, inspect count afterwards\nconst atlas = await openAtlas('./crawl.atls', { strict: false });\nfor await (const page of atlas.readers.pageMetadata()) {\n  console.log(page.url);\n}\nif (atlas.stats.corruptedRecordCount > 0) {\n  console.warn(`Skipped ${atlas.stats.corruptedRecordCount} malformed records`);\n}\natlas.close();\n```\n\n## Validation\n\n```ts\nimport { validate } from '@caifrazier/atls-sdk';\n\nconst result = await validate('./crawl.atls', {\n  checkIntegrity: true,\n  checkBrokenLinks: true,\n  checkManifest: true,\n  strict: false,\n});\n\nconsole.log(result.valid, result.stats.errorsFound);\n```\n\nSource: `src/validate.ts`\n\n## `select` — Filtered Dataset Streaming\n\nFor one-off filtered reads without opening a full `AtlasReader`:\n\n```ts\nimport { select } from '@caifrazier/atls-sdk';\n\nfor await (const record of select('./crawl.atls', {\n  dataset: 'accessibility',\n  where: (a: any) => a.missingAltCount > 0,\n  fields: ['pageUrl', 'missingAltCount'],\n  limit: 50,\n})) {\n  console.log(record);\n}\n```\n\n`SelectOptions`: `dataset` (required), `where` (filter predicate), `fields` (dot-path projection), `limit`.\n\n## Advanced / Utilities\n\n### Diff two archives\n\n```ts\nimport { openAtlas, diffManifests, formatManifestDiff } from '@caifrazier/atls-sdk';\n\nconst [a, b] = await Promise.all([openAtlas('./before.atls'), openAtlas('./after.atls')]);\nconst diff = diffManifests(a.manifest, b.manifest);\nconsole.log(formatManifestDiff(diff));   // formatted table\n// or: JSON.stringify(diff)              // ManifestDiff object\na.close(); b.close();\n```\n\nExports: `diffManifests`, `formatManifestDiff`, types `ManifestDiff`, `DatasetDelta`, `DiffManifestsOptions`.\n\n### Cross-mode index\n\n```ts\nimport { buildCrossModeIndex } from '@caifrazier/atls-sdk';\n```\n\nBuilds a `CrossModeIndex` that aligns records across archives produced in different render modes. Exports: `buildCrossModeIndex`, types `CrossModeIndex`, `CrossModeEntry`.\n\n### Low-level helpers\n\nThese bypass `openAtlas` and operate directly on the zip:\n\n| Export | Description |\n|---|---|\n| `readManifest(zipHandle)` | Read and parse `manifest.json` |\n| `readSummary(zipHandle)` | Read and parse `summary.json` |\n| `iterateParts(zipHandle, dataset)` | Iterate raw NDJSON lines from compressed dataset parts |\n| `readBlob(zipHandle, ref)` | Fetch a blob by content-addressed ref string |\n| `openZipHandle(path)` | Open a zip for repeated reads (must call `.close()`) |\n| `ZipHandle` | Type for the handle returned by `openZipHandle` |\n\n## Development\n\n```bash\npnpm --filter @caifrazier/atls-sdk build\npnpm --filter @caifrazier/atls-sdk test\npnpm --filter @caifrazier/atls-sdk lint\n```\n\nArchived legacy docs are available at `legacy/v1/`.\n\n---\n\n_Last verified against codebase: 2026-06-04._\n","readmeFilename":"README.md","_rev":"1-487963a94572d7311885dae263c1d764"}