{"_id":"@backloghq/termlog-s3","name":"@backloghq/termlog-s3","dist-tags":{"latest":"0.1.0"},"versions":{"0.1.0":{"name":"@backloghq/termlog-s3","version":"0.1.0","description":"Amazon S3 storage backend for @backloghq/termlog.","main":"dist/index.js","types":"dist/index.d.ts","type":"module","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"scripts":{"build":"tsc","dev":"tsc --watch","lint":"eslint src/ tests/","test":"vitest run","test:watch":"vitest","test:coverage":"vitest run --coverage","test:integration":"S3_INTEGRATION=1 vitest run tests/s3-integration.test.ts tests/termlog-integration.test.ts","prepublishOnly":"npm run build"},"keywords":["full-text-search","termlog","s3","storage-backend","aws"],"author":{"name":"mbocevski"},"license":"MIT","sideEffects":false,"repository":{"type":"git","url":"git+https://github.com/backloghq/termlog-s3.git"},"bugs":{"url":"https://github.com/backloghq/termlog-s3/issues"},"homepage":"https://github.com/backloghq/termlog-s3#readme","engines":{"node":">=22"},"dependencies":{"@aws-sdk/client-s3":"^3.800.0"},"peerDependencies":{"@backloghq/termlog":">=0.1.0"},"devDependencies":{"@backloghq/termlog":"^0.1.0","@eslint/js":"^10.0.0","@types/node":"^25.0.0","@vitest/coverage-v8":"^4.1.2","eslint":"^10.0.0","typescript":"~6.0.2","typescript-eslint":"^8.58.1","vitest":"^4.1.2"},"gitHead":"26e1353f2f878aa82d58679eaa4685bb5e7905dc","_id":"@backloghq/termlog-s3@0.1.0","_nodeVersion":"25.9.0","_npmVersion":"11.13.0","dist":{"integrity":"sha512-Ef/C//fFBQrOnnmylhr1ioInvKUYasSOsytF2kznB4z6r6qzH9rSD9TrKqTpbhKjcfS1KoTaFTXPbuhsn/opJw==","shasum":"7b634c4e1daa0b384d79bcc0e990033ee9121768","tarball":"https://registry.npmjs.org/@backloghq/termlog-s3/-/termlog-s3-0.1.0.tgz","fileCount":16,"unpackedSize":22234,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCkY9jNfcG2Obw2iQtimPK5l16GLTv0dYX1qOUMlebxhQIhAKqJjyNC/o5rER2O2lR21QV2am1juDfA9Ki9itb+KZc7"}]},"_npmUser":{"name":"fenrirbaest","email":"marko.bocevski@gmail.com"},"directories":{},"maintainers":[{"name":"fenrirbaest","email":"marko.bocevski@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/termlog-s3_0.1.0_1777965592045_0.1869765761583606"},"_hasShrinkwrap":false}},"time":{"created":"2026-05-05T07:19:51.925Z","0.1.0":"2026-05-05T07:19:52.192Z","modified":"2026-05-05T07:19:52.450Z"},"maintainers":[{"name":"fenrirbaest","email":"marko.bocevski@gmail.com"}],"description":"Amazon S3 storage backend for @backloghq/termlog.","homepage":"https://github.com/backloghq/termlog-s3#readme","keywords":["full-text-search","termlog","s3","storage-backend","aws"],"repository":{"type":"git","url":"git+https://github.com/backloghq/termlog-s3.git"},"author":{"name":"mbocevski"},"bugs":{"url":"https://github.com/backloghq/termlog-s3/issues"},"license":"MIT","readme":"# termlog-s3\n\nAmazon S3 storage backend for [@backloghq/termlog](https://github.com/backloghq/termlog). Store termlog segment data in S3-compatible object stores (AWS S3, MinIO, Cloudflare R2, LocalStack).\n\n## Install\n\n```bash\nnpm install @backloghq/termlog @backloghq/termlog-s3\n```\n\n## Usage\n\n```typescript\nimport { TermLog } from \"@backloghq/termlog\";\nimport { S3Backend } from \"@backloghq/termlog-s3\";\nimport { S3Client } from \"@aws-sdk/client-s3\";\n\nconst backend = new S3Backend({\n  client: new S3Client({ region: \"us-east-1\" }),\n  bucket: \"my-bucket\",\n  prefix: \"my-index/\",\n});\n\nconst index = await TermLog.open({ dir: \"my-index\", backend });\nawait index.add(\"doc-1\", \"hello world\");\nawait index.add(\"doc-2\", \"hello termlog\");\n\nconst results = await index.search(\"hello\");\nawait index.close();\n```\n\n### MinIO / LocalStack\n\n```typescript\nconst backend = new S3Backend({\n  client: new S3Client({\n    region: \"us-east-1\",\n    endpoint: \"http://localhost:9000\",\n    forcePathStyle: true,\n  }),\n  bucket: \"my-bucket\",\n  prefix: \"my-index/\",\n});\n```\n\n## Options\n\n```typescript\nnew S3Backend({\n  client: myS3Client,   // S3Client instance (required)\n  bucket: \"my-bucket\",  // S3 bucket name (required)\n  prefix: \"my-index/\",  // Key prefix — scope all objects under this prefix (optional)\n});\n```\n\n## Single-writer constraint\n\nS3 provides no distributed lock. You must ensure **at most one writer** per `(bucket, prefix)` combination at any given time. Multiple concurrent readers are safe.\n\n## Multipart upload\n\n`createWriteStream` uses S3 multipart upload (required for streaming segment writes). Parts are buffered at 5 MiB (S3 minimum part size). If the stream completes with zero bytes, the adapter falls back to a `PutObject` call (S3 rejects `CompleteMultipartUpload` with empty Parts).\n\n**Required IAM permissions** for the multipart path: `s3:CreateMultipartUpload`, `s3:UploadPart`, `s3:CompleteMultipartUpload`, `s3:AbortMultipartUpload`.\n\n## IAM Permissions\n\nMinimum required permissions:\n\n```json\n{\n  \"Effect\": \"Allow\",\n  \"Action\": [\n    \"s3:GetObject\",\n    \"s3:PutObject\",\n    \"s3:DeleteObject\",\n    \"s3:ListBucket\",\n    \"s3:CreateMultipartUpload\",\n    \"s3:UploadPart\",\n    \"s3:CompleteMultipartUpload\",\n    \"s3:AbortMultipartUpload\"\n  ],\n  \"Resource\": [\n    \"arn:aws:s3:::my-bucket\",\n    \"arn:aws:s3:::my-bucket/my-index/*\"\n  ]\n}\n```\n\n## S3 lifecycle rules (recommended)\n\nAdd a lifecycle rule to abort incomplete multipart uploads after 1 day to avoid storage charges from crashed writers:\n\n```json\n{\n  \"Rules\": [{\n    \"ID\": \"abort-incomplete-multipart\",\n    \"Status\": \"Enabled\",\n    \"Filter\": { \"Prefix\": \"my-index/\" },\n    \"AbortIncompleteMultipartUpload\": { \"DaysAfterInitiation\": 1 }\n  }]\n}\n```\n\n## Development\n\n```bash\nnpm run build          # Compile TypeScript\nnpm run lint           # ESLint\nnpm test               # Run tests (in-memory mock S3, no AWS needed)\nnpm run test:coverage  # Tests with coverage\nnpm run test:integration  # Real S3 tests (requires S3_INTEGRATION=1 + credentials)\n```\n\n### Integration tests in CI\n\nThe CI `integration` job runs the full integration suite (including the 1500-object pagination stress test) against a MinIO container on every push — no AWS credentials, no cost. It runs after the unit-test matrix passes (`needs: test`).\n\n### Local integration testing\n\nTo run against MinIO locally:\n\n```bash\ndocker run -d --name minio -p 9000:9000 \\\n  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \\\n  minio/minio server /data\n\nAWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \\\n  aws --endpoint-url http://localhost:9000 s3 mb s3://termlog-test\n\nS3_INTEGRATION=1 S3_INTEGRATION_SLOW=1 \\\n  S3_TEST_BUCKET=termlog-test S3_TEST_ENDPOINT=http://localhost:9000 \\\n  npm run test:integration\n```\n\nTo run against real AWS S3 (opt-in, costs money):\n\n```bash\nS3_INTEGRATION=1 S3_TEST_BUCKET=my-bucket S3_TEST_REGION=us-east-1 \\\n  AWS_PROFILE=my-profile npm run test:integration\n```\n\n### Integration test env vars\n\n| Var | Required | Description |\n|---|---|---|\n| `S3_INTEGRATION=1` | yes | Enables real-S3 tests (otherwise all skipped) |\n| `S3_TEST_BUCKET` | yes | Bucket name |\n| `S3_TEST_REGION` | no | AWS region (default: `us-east-1`) |\n| `S3_TEST_ENDPOINT` | no | Custom endpoint for MinIO / LocalStack |\n| `AWS_PROFILE` | no | AWS credential profile |\n| `S3_INTEGRATION_SLOW=1` | no | Enables pagination stress test (1500 PutObject + 1500 DeleteObject — ~5 s on MinIO, ~60 s + cost on real AWS S3) |\n\n## License\n\nMIT\n","readmeFilename":"README.md","_rev":"1-acf1391b34d17af4fa25daab91571b66"}