{"_id":"@bluesky-social/sync","_rev":"2-9a78e4a9ac2a643a85ea87e5c6932d1a","name":"@bluesky-social/sync","dist-tags":{"latest":"0.1.35"},"versions":{"0.1.29":{"name":"@bluesky-social/sync","version":"0.1.29","keywords":["atproto","sync","firehose","relay"],"license":"MIT","_id":"@bluesky-social/sync@0.1.29","maintainers":[{"name":"openweb3.io","email":"mtsocialdao@gmail.com"},{"name":"web3km","email":"web3km@proton.me"}],"homepage":"https://atproto.com","bugs":{"url":"https://github.com/bluesky-social/atproto/issues"},"dist":{"shasum":"5c4d5adcbf71d444efc60569cfd5762b861424a9","tarball":"https://registry.npmjs.org/@bluesky-social/sync/-/sync-0.1.29.tgz","fileCount":55,"integrity":"sha512-SIAA4PDm/BCkoqb6kBJ2VLmJs9ThhRXuZniPrgo8phuiTtV4NYIQoQo5yBx5hy0J6R4hpVlcEcR4vLHjPoEgRQ==","signatures":[{"sig":"MEQCIBGfLQHw8kql66s/LxOeoWqxYHDqzsrbGikEr7LH8f2LAiBD0kcaS9r4IJHCHHXH7//I8WwWF5yPVRtWEwX6LwX8+Q==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":131203},"main":"dist/index.js","_from":"file:bluesky-social-sync-0.1.29.tgz","types":"dist/index.d.ts","engines":{"node":">=18.7.0"},"scripts":{"test":"../dev-infra/with-test-redis-and-db.sh jest","build":"tsc --build tsconfig.build.json"},"_npmUser":{"name":"web3km","email":"web3km@proton.me"},"_resolved":"/private/var/folders/cb/7sst87dj7jd40nc7b7q7h76r0000gn/T/ef6ed3e3dfbdab4f405f7dc4464e5fe0/bluesky-social-sync-0.1.29.tgz","_integrity":"sha512-SIAA4PDm/BCkoqb6kBJ2VLmJs9ThhRXuZniPrgo8phuiTtV4NYIQoQo5yBx5hy0J6R4hpVlcEcR4vLHjPoEgRQ==","repository":{"url":"git+https://github.com/bluesky-social/atproto.git","type":"git","directory":"packages/sync"},"_npmVersion":"10.7.0","description":"atproto sync library","directories":{},"_nodeVersion":"20.15.0","dependencies":{"ws":"^8.12.0","p-queue":"^6.6.2","multiformats":"^9.9.0","@bluesky-social/repo":"^0.8.5","@bluesky-social/common":"^0.4.11","@bluesky-social/syntax":"^0.4.0","@bluesky-social/lexicon":"^0.4.12","@bluesky-social/identity":"^0.4.8","@bluesky-social/xrpc-server":"^0.9.0"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^28.1.2","@types/ws":"^8.5.4","typescript":"^5.6.3"},"_npmOperationalInternal":{"tmp":"tmp/sync_0.1.29_1753693256001_0.08597979994660276","host":"s3://npm-registry-packages-npm-production"}},"0.1.35":{"name":"@bluesky-social/sync","version":"0.1.35","license":"MIT","description":"atproto sync library","keywords":["atproto","sync","firehose","relay"],"homepage":"https://atproto.com","repository":{"type":"git","url":"git+https://github.com/bluesky-social/atproto.git","directory":"packages/sync"},"engines":{"node":">=18.7.0"},"main":"dist/index.js","types":"dist/index.d.ts","dependencies":{"multiformats":"^9.9.0","p-queue":"^6.6.2","ws":"^8.18.3","@bluesky-social/common":"^0.4.12","@bluesky-social/identity":"^0.4.9","@bluesky-social/lexicon":"^0.5.1","@bluesky-social/syntax":"^0.4.1","@bluesky-social/repo":"^0.8.10","@bluesky-social/xrpc-server":"^0.9.5"},"devDependencies":{"@types/ws":"^8.18.1","jest":"^28.1.3","typescript":"^5.9.3"},"scripts":{"build":"tsc --build tsconfig.build.json","test":"../dev-infra/with-test-redis-and-db.sh jest"},"_id":"@bluesky-social/sync@0.1.35","bugs":{"url":"https://github.com/bluesky-social/atproto/issues"},"_integrity":"sha512-tvKQxqm0ntmVQXLFwEijZNyMDDhwxDOYGSpnr3AiPiK7dRot1uqmJPlNlKtvW1xucs1XOsWp8upAy51d4t7BOA==","_resolved":"/private/var/folders/cb/7sst87dj7jd40nc7b7q7h76r0000gn/T/b55e4d3b695ef1414e87b7f26fa3dc13/bluesky-social-sync-0.1.35.tgz","_from":"file:bluesky-social-sync-0.1.35.tgz","_nodeVersion":"20.15.0","_npmVersion":"10.7.0","dist":{"integrity":"sha512-tvKQxqm0ntmVQXLFwEijZNyMDDhwxDOYGSpnr3AiPiK7dRot1uqmJPlNlKtvW1xucs1XOsWp8upAy51d4t7BOA==","shasum":"613f89ac047e15773173dede265e16d34f429b0f","tarball":"https://registry.npmjs.org/@bluesky-social/sync/-/sync-0.1.35.tgz","fileCount":55,"unpackedSize":165594,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQCDL2hLnv/inC9ElCoR/bRxxk8SNJbQ3JgJi/l2IPBj3AIgAqs/NcZyx5sqt6v/1QgH+aJ6Jhuqdt/aaeEQ/v9/6uo="}]},"_npmUser":{"name":"web3km","email":"web3km@proton.me"},"directories":{},"maintainers":[{"name":"openweb3.io","email":"mtsocialdao@gmail.com"},{"name":"web3km","email":"web3km@proton.me"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/sync_0.1.35_1761039232687_0.47553432994858813"},"_hasShrinkwrap":false}},"time":{"created":"2025-07-28T09:00:55.937Z","modified":"2025-10-21T09:33:53.081Z","0.1.29":"2025-07-28T09:00:56.185Z","0.1.35":"2025-10-21T09:33:52.873Z"},"bugs":{"url":"https://github.com/bluesky-social/atproto/issues"},"license":"MIT","homepage":"https://atproto.com","keywords":["atproto","sync","firehose","relay"],"repository":{"type":"git","url":"git+https://github.com/bluesky-social/atproto.git","directory":"packages/sync"},"description":"atproto sync library","maintainers":[{"name":"openweb3.io","email":"mtsocialdao@gmail.com"},{"name":"web3km","email":"web3km@proton.me"}],"readme":"# @bluesky-social/sync: atproto sync tools\n\nTypeScript library for syncing data from the [atproto](https://atproto.com) network. Currently only supports firehose (relay) subscriptions\n\n[![NPM](https://img.shields.io/npm/v/@bluesky-social/sync)](https://www.npmjs.com/package/@bluesky-social/sync)\n[![Github CI Status](https://github.com/bluesky-social/atproto/actions/workflows/repo.yaml/badge.svg)](https://github.com/bluesky-social/atproto/actions/workflows/repo.yaml)\n\n## Usage\n\nThe firehose class will spin up a websocket connection to `com.atproto.sync.subscribeRepos` on a given repo host (by default the Relay run by Bluesky).\n\nEach event will be parsed, authenticated, and then passed on to the supplied `handleEvt` which can handle indexing.\n\nOn `Commit` events, the firehose will verify signatures and repo proofs to ensure that the event is authentic. This can be disabled with the `unauthenticatedCommits` flag. Similarly on `Identity` events, the firehose will fetch the latest DID document for the repo and do bidirectional verification on the associated handle. This can be disabled with the `unauthenticatedHandles` flag.\n\nEvents of a certain type can be excluded using the `excludeIdentity`/`excludeAccount`/`excludeCommit` flags. And repo writes can be filtered down to specific collections using `filterCollections`. By default, all events are parsed and passed through to the handler. Note that this filtered currently happens client-side, though it is likely we will introduce server-side methods for doing so in the future.\n\nNon-fatal errors that are encountered will be passed to the required `onError` handler. In most cases, these can just be logged.\n\nWhen using the firehose class, events are processed serially. Each event must be finished being handled before the next one is parsed and authenticated.\n\n```ts\nimport { Firehose } from '@bluesky-social/sync'\nimport { IdResolver } from '@bluesky-social/identity'\n\nconst idResolver = new IdResolver()\nconst firehose = new Firehose({\n  idResolver,\n  service: 'wss://bsky.network',\n  handleEvt: async (evt) => {\n    if (evt.event === 'identity') {\n      // ...\n    } else if (evt.event === 'account') {\n      // ...\n    } else if (evt.event === 'create') {\n      // ...\n    } else if (evt.event === 'update') {\n      // ...\n    } else if (evt.event === 'delete') {\n      // ...\n    }\n  },\n  onError: (err) => {\n    console.error(err)\n  },\n  filterCollections: ['com.myexample.app'],\n})\nfirehose.start()\n\n// on service shutdown\nawait firehose.destroy()\n```\n\nFor more robust indexing pipelines, it's recommended to use the supplied `MemoryRunner` class. This provides an in-memory partitioned queue. As events from a given repo must be processed in order, this allows events to be processed concurrently while still processing events from any given repo serially.\n\nThe `MemoryRunner` also tracks an internal cursor based on the last finished consecutive work. This ensures that no events are dropped, although it does mean that some events may occassionally be replayed (if the websocket drops and reconnects) and therefore it's recommended that any indexing logic is idempotent. An optional `setCursor` parameter may be supplied to the `MemoryRunner` which can be used to persistently store the most recently processed cursor.\n\n```ts\nimport { Firehose, MemoryRunner } from '@bluesky-social/sync'\nimport { IdResolver } from '@bluesky-social/identity'\n\nconst idResolver = new IdResolver()\nconst runner = new MemoryRunner({\n  setCursor: (cursor) => {\n    // persist cursor\n  },\n})\nconst firehose = new Firehose({\n  idResolver,\n  runner,\n  service: 'wss://bsky.network',\n  handleEvt: async (evt) => {\n    // ...\n  },\n  onError: (err) => {\n    console.error(err)\n  },\n})\nfirehose.start()\n\n// on service shutdown\nawait firehose.destroy()\nawait runner.destroy()\n```\n\n## License\n\nThis project is dual-licensed under MIT and Apache 2.0 terms:\n\n- MIT license ([LICENSE-MIT.txt](https://github.com/bluesky-social/atproto/blob/main/LICENSE-MIT.txt) or http://opensource.org/licenses/MIT)\n- Apache License, Version 2.0, ([LICENSE-APACHE.txt](https://github.com/bluesky-social/atproto/blob/main/LICENSE-APACHE.txt) or http://www.apache.org/licenses/LICENSE-2.0)\n\nDownstream projects and end users may chose either license individually, or both together, at their discretion. The motivation for this dual-licensing is the additional software patent assurance provided by Apache 2.0.\n","readmeFilename":"README.md"}