\n\nThis library allows building custom [Sui indexers](https://docs.sui.io/guides/developer/advanced/custom-indexer?ref=blog.sui.io) in JavaScript. It reads checkpoint files from both remote and local sources and processes them efficiently.\n\nIt also allows building a local leveldb from Sui's formal snapshots, which you can then process to initialize your indexer without the need to read checkpoints history for past epochs\n\n> **Note:** This is a work in progress and might not be suited for production usage.\n\n## 🚀 Getting Started\n\nFirst, install the library:\n\n```sh\nnpm install @aresrpg/sui-checkpoint-reader\n```\n\nHere's a basic example of how to use the library:\n\n```js\nimport { read_checkpoints } from '@aresrpg/sui-checkpoint-reader/reader'\n\nasync function get_remote_checkpoint(num) {\n const response = await fetch(`https://checkpoints.testnet.sui.io/${num}.chk`)\n const buffer = await response.arrayBuffer()\n return buffer\n}\n\nasync function process_checkpoint(checkpoint, index) {\n console.log('[indexer] process_checkpoint:', index)\n}\n\nconst known_types = {}\n\nawait read_checkpoints({\n from: 1, // Start processing from checkpoint 1\n to: Infinity, // Continue processing indefinitely\n get_remote_checkpoint, // Function to fetch checkpoint data\n concurrent_downloads: 25, // Number of concurrent downloads allowed\n known_types, // BCS types you want to parse\n checkpoints_folder: '', // Local folder for checkpoint files (nesting is supported at 1 level)\n cleanup_checkpoints: false, // Clean up processed checkpoint files\n process_checkpoint, // Function to process each checkpoint\n local_files_only: false, // if true, the reader will only use checkpoints exported by the node\n})\n```\n\nAnd here is how to get your leveldb from the formal snapshot\n\n```js\nimport { download_and_store_objects, read_snapshot_objects, get_db } from '@aresrpg/sui-checkpoint-reader/snapshot'\n\nconst DB_FOLDER = './sui-formal-objects'\nconst db = get_db(DB_FOLDER)\n\n// this will start download the formal snapshot of the specified epoch\n// and save any objects recognized by your known types into leveldb (key: id, value: object)\nawait download_and_store_objects({\n network: 'testnet',\n epoch: 440, // the epoch of the snapshot\n known_types, // same types as for checkpoint reading\n save_objects: false, // if you want to also save the .obj files locally under obj_folder/epoch_X/*.obj\n start_bucket = 1, // optional start files 1_1.obj\n start_part = 1,\n db,\n obj_folder = './obj_files' // if `save_objects` is true, it'll use this folder to save obj files\n concurrent_downloads = 1, // how many obj files to download in one go\n})\n\n\n// and here you can iterate easily on your leveldb, note that you can use anything else to read those\nfor await (const object of read_snapshot_objects(db)) {\n // process all objects here, for example use db.get(object.contents.kiosk_id)\n // to save something like { object, object_kiosk } in your storage of choice\n console.dir(object, { depth: Infinity })\n}\n```\n\n## ⚙️ How It Works\n\nThe reader supports two modes: **remote-only** and **hybrid** (more powerful) mode.\n\n### Hybrid Mode (with `checkpoints_folder` specified)\n\n1. **Initial Check**: The reader will first check existing checkpoint files and determine what needs to be downloaded to fill the gap between `from` and the lowest present checkpoint file.\n2. **Gap Filling**: If `from` is lower than the lowest checkpoint file, missing checkpoints will be downloaded up to the lower checkpoint file.\n3. **Sequential Processing**: The reader will process checkpoints sequentially as they become available.\n4. **Stopping Remote Reader**: Once all checkpoints up to the lowest file (or the `to` parameter) are downloaded, the remote reader will stop.\n5. **Local File Watching**: Local checkpoint files will be watched and processed as they become available.\n\n### Remote-Only Mode (without `checkpoints_folder`)\n\nIn this mode, the reader will rely solely on remote checkpoint files:\n\n- **Fetching Checkpoints**: Checkpoints are fetched from the specified remote source.\n- **Sequential Processing**: Checkpoints are processed in sequential order as they are downloaded.\n- **Efficient Downloads**: The reader handles concurrent downloads up to the specified limit.\n\n## Example Usage for Remote-Only Mode\n\n```js\nimport { read_checkpoints } from '@aresrpg/sui-checkpoint-reader'\n\nasync function get_remote_checkpoint(num) {\n const response = await fetch(`https://checkpoints.testnet.sui.io/${num}.chk`)\n const buffer = await response.arrayBuffer()\n return buffer\n}\n\nasync function process_checkpoint(checkpoint, index) {\n console.log('[indexer] process_checkpoint:', index)\n}\n\nconst known_types = {}\n\nawait read_checkpoints({\n from: 1,\n get_remote_checkpoint,\n concurrent_downloads: 25,\n known_types,\n process_checkpoint,\n})\n```\n\n## 🧩 BCS Types\n\nTo parse specific objects within the checkpoints, you can provide the BCS definitions directly.\n\n### Generating BCS Types\n\nYou can generate the necessary types using the https://github.com/Sceat/sui-bcs. Here is an example of how to generate them:\n\n```bash\nnpx sui-bcs gen --package --file --network \n```\n\n> The `network_name` defaults to `testnet` if not provided. Not that you can also generate 0x2 types as needed\n\n### Example Usage\n\nHere's an example of how you can define and use known BCS types:\n\n```js\n// npx sui-bcs gen --package 0x68e06aee7966648c7b3258f02d5c7da2cf9b035727d6e9167c90e662f20e0406 --file aresrpg-bcs.js --network testnet\nimport { read_checkpoints } from '@aresrpg/sui-checkpoint-reader'\n\nimport aresrpg_bcs from './aresrpg-bcs.js'\n\nconst ARESRPG =\n '0x68e06aee7966648c7b3258f02d5c7da2cf9b035727d6e9167c90e662f20e0406'\n\nconst known_types = {\n [ARESRPG]: aresrpg_bcs,\n}\n```\n\nThis configuration allows the checkpoint reader to correctly parse and process the objects within the checkpoints.\n\n> Note that 0x1 & 0x2 are built in\n\n## 📖 API Reference\n\n### `read_checkpoints(options)`\n\n**Options:**\n\n- `from`: Start processing from this checkpoint number (default: `1`).\n- `to`: Stop processing once this checkpoint number is reached (default: `Infinity`).\n- `get_remote_checkpoint`: Function to fetch a checkpoint array buffer from a number.\n- `concurrent_downloads`: Number of concurrent downloads allowed while catching up (default: `25`).\n- `known_types`: Generated BCS for types you want to parse.\n- `checkpoints_folder`: Local folder where your Sui node is dumping the checkpoint files (default: `''`).\n- `cleanup_checkpoints`: Option to delete already processed checkpoint files (default: `false`).\n- `process_checkpoint`: Function to process a checkpoint.\n\n### `download_and_store_objects(options)`\n\n**Options:**\n\n- `network`: Network to download the snapshot from (e.g., `'testnet'`).\n- `epoch`: Epoch of the snapshot.\n- `known_types`: Generated BCS for types you want to parse.\n- `save_objects`: If true, saves `.obj` files locally under `obj_folder/epoch_X/\\*.obj` (default: `false`).\n- `start_bucket`: Optional start file (e.g., `1_1.obj`) (default: `1`).\n- `start_part`: Part to start from (default: `1`).\n- `db`: Classic level instance.\n- `obj_folder`: Folder to store the `.obj` files\n- `concurrent_downloads`: Concurrent obj files download\n\n### `read_snapshot_objects(db)`\n\n- `db`: The Classic level db instance\n\nReturns:\n\nAn async generator yielding [db, object] pairs for processing.\n\n## Contributing 🛠️\n\nContributions are welcome! Please open an issue or submit a pull request with any improvements or bug fixes.\n\n## License 📄\n\nThis project is licensed under the ISC License.\n","readmeFilename":"README.md"}
![](\"https://user-images.githubusercontent.com/11330271/208825167-77d7bc78-17d0-4f33-ad35-d108b6fac730.gif\")
![\"fully](\"https://img.shields.io/badge/Made%20with-Javascript-%23f7df1e?style=for-the-badge\"){kind=icon}
\n ![\"powered](\"https://img.shields.io/badge/Powered%20By-Dark%20Magic-blueviolet?style=for-the-badge\"){kind=icon}
\n ![\"Chat\"/](\"https://img.shields.io/discord/265104803531587584.svg?logo=discord&style=for-the-badge\")
\n ![\"NPM](\"https://img.shields.io/npm/v/%40aresrpg%2Fsui-checkpoint-reader?style=for-the-badge\")
\n