{"_id":"@brave/utl-aggregator","_rev":"6-1c894a3f0dcd141a3e93491ba7e8f591","name":"@brave/utl-aggregator","dist-tags":{"latest":"0.0.15"},"versions":{"0.0.15":{"name":"@brave/utl-aggregator","version":"0.0.15","author":{"name":"Solflare Developers","email":"developers@solflare.com"},"license":"ISC","_id":"@brave/utl-aggregator@0.0.15","maintainers":[{"name":"darkdh","email":"darkdh@gmail.com"},{"name":"bcrypt","email":"yan@mit.edu"},{"name":"brianbondy","email":"bbondy@gmail.com"},{"name":"brave.com","email":"devops@brave.com"},{"name":"evq","email":"ev@7pr.xyz"},{"name":"mrose17","email":"mrose17@gmail.com"},{"name":"petemill","email":"miller.pete+npm@gmail.com"}],"dist":{"shasum":"6714ceadc7cbec52a16e319861684fe333936f14","tarball":"https://registry.npmjs.org/@brave/utl-aggregator/-/utl-aggregator-0.0.15.tgz","fileCount":62,"integrity":"sha512-cqKrb5pne9htOU1fER1+BmN7XDIDfWk/Hm+9KRAxbn1ReqU6Y7mlwxIcYwEEtsPIngpdf7sOcG5rinbc/NJNaw==","signatures":[{"sig":"MEUCIDNqi1j2/9MkUty23TWIM+fFRv5RWIkYmBeGeyOpNxJUAiEA5ScrQ352NIEffcheRJQb2jyU+7cm2P67ewXEMrrRy3I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":185347},"main":"lib/cjs/index.js","types":"./lib/cjs/index.d.ts","module":"lib/esm/index.js","engines":{"node":">=16"},"gitHead":"0e7a2bb7e42c27b3686c6d390cc221cb7a64e914","scripts":{"build":"npm run build:esm && npm run build:cjs","start":"npm-run-all -p start:esm start:cjs","deploy":"npm run build && npm publish --access public","build:cjs":"tsc --project tsconfig.cjs.json","build:esm":"tsc","start:cjs":"tsc --project tsconfig.cjs.json --watch","start:esm":"tsc --watch"},"_npmUser":{"name":"brave.com","email":"devops@brave.com"},"_npmVersion":"9.5.1","description":"The Unified Token List Aggregator (`UTL`) module generates Solana token list JSON based on user specified list of `provider` sources.","directories":{},"_nodeVersion":"18.16.0","dependencies":{"axios":"^0.27.2","lodash":"^4.17.21","temp-dir":"^2.0.0","axios-retry":"^3.2.5"},"_hasShrinkwrap":false,"devDependencies":{"eslint":"^8.15.0","prettier":"^2.6.2","typescript":"^4.6.4","@types/node":"^18.11.18","npm-run-all":"^4.1.5","@types/axios":"^0.14.0","@types/lodash":"^4.14.182","eslint-plugin-node":"^11.1.0","eslint-plugin-import":"^2.26.0","@typescript-eslint/parser":"^5.25.0","@typescript-eslint/eslint-plugin":"^5.25.0"},"peerDependencies":{"@solana/web3.js":"*"},"_npmOperationalInternal":{"tmp":"tmp/utl-aggregator_0.0.15_1702468258431_0.27522375651137265","host":"s3://npm-registry-packages"}}},"time":{"created":"2023-12-13T11:50:58.337Z","modified":"2026-06-18T20:44:40.126Z","0.0.15":"2023-12-13T11:50:58.611Z"},"author":{"name":"Solflare Developers","email":"developers@solflare.com"},"license":"ISC","description":"The Unified Token List Aggregator (`UTL`) module generates Solana token list JSON based on user specified list of `provider` sources.","maintainers":[{"email":"yan@mit.edu","name":"bcrypt"},{"email":"bbondy@gmail.com","name":"brianbondy"},{"email":"devops@brave.com","name":"brave.com"},{"email":"darkdh@gmail.com","name":"darkdh"},{"email":"mrose17@gmail.com","name":"mrose17"},{"email":"miller.pete+npm@gmail.com","name":"petemill"},{"email":"sceli@brave.com","name":"claucece9"},{"email":"pes@peteresnyder.com","name":"pes10k"},{"email":"jjdsampson@gmail.com","name":"jonathansampson"},{"email":"alazarev@brave.com","name":"antonok"}],"readme":"# Unified Token List Aggregator\n\nThe Unified Token List Aggregator (`UTL`) module generates Solana token list JSON based on user specified list of `provider` sources.\n\nBy changing the provider source list in the aggregator config one can fine tune the output (explained below), and choose which providers are trusted, and filter out tokens (for example exclude Liquidity Pool (`LP`)-tokens which could be consumed from other sources).\n\nRunning a script to call this module periodically will ensure that generated UTL is up-to-date.\n\nGenerated JSON can be hosted on CDN or imported in DB to be exposed through API.\n\nThe UTL generated through the aggregation process should be considered as a common source of truth for verified tokens across wallets and dApps.\n\n## Our Goal\n\nWe want to provide every community member a same base source of truth generated by Token List Aggregator - by soing do, we'll provide the community with a base verified token list. Anyone can use this module without any infrastructure or cost.\n\nEverything after that is only building on top of that, so Token List API is extension, and Token List SDK is extension on top of that. Every step is making things more efficient and optimised.\n\nEveryone can choose what they want to use, host and consume depending on their needs and requirements.\n\n## Related repos\n\n- [Token List API](https://github.com/solflare-wallet/utl-api)\n- [Token List SDK](https://github.com/solflare-wallet/utl-sdk)\n- [Solflare Token List](https://github.com/solflare-wallet/token-list)\n\n\n## Installation\n```shell\nnpm i @solflare-wallet/utl-aggregator\n```\n\n## Usage\n\nExample usage can be found in [Solfare's Token List repo](https://github.com/solflare-wallet/token-list).\n\n\nSimple usage: \n\n```ts\nimport {\n  Generator,\n  ProviderCoinGecko,\n  ProviderLegacyToken,\n  ProviderTrusted,\n  ProviderIgnore,\n  ChainId,\n  Tag,\n} from \"@solflare-wallet/utl-aggregator\";\nimport { clusterApiUrl } from '@solana/web3.js'\nimport { writeFile } from 'fs/promises'\n\nconst SECOND = 1000;\nconst SECONDS = SECOND;\nconst MINUTE = 60 * SECOND;\nconst MINUTES = MINUTE;\n\n// Your Solana RPC URL - may be an open provider like\n//   clusterApiUrl(\"mainnet-beta\")\n// Or your own RPC instance like QuickNode etc.\nconst SOLANA_RPC_URL = clusterApiUrl(\"mainnet-beta\")\n\nasync function main() {\n  // Optionally clear the cache for each provider:\n  //   ProviderLegacyToken.clearCache(ChainId.MAINNET)\n  //   ProviderLegacyToken.clearCache(ChainId.DEVNET)\n\n  const generator = new Generator([\n    // Providers are listen in order of preference\n    new ProviderCoinGecko(null, SOLANA_RPC_URL, {\n      // Add sleep after batch RPC request to avoid rate limits\n      throttle: 1 * SECOND,\n      // Add sleep after batch HTTP calls for CoinGecko\n      throttleCoinGecko: 65 * SECONDS,\n      // Batch RPC calls in single RPC request\n      batchAccountsInfo: 100,\n      // Batch CoinGecko token HTTP call\n      batchCoinGecko: 25,\n    }),\n    new ProviderLegacyToken(\n      'https://cdn.jsdelivr.net/gh/solana-labs/token-list@main/src/tokens/solana.tokenlist.json',\n      SOLANA_RPC_URL,\n      {\n        // Add sleep after batch RPC request to avoid rate limits\n        throttle: 1000,\n        // Batch RPC calls in single RPC request\n        batchSignatures: 100,\n        batchAccountsInfo: 100,\n        // Batch parallel RPC requests\n        batchTokenHolders: 1,\n      },\n      // Filter out by tags, eg. remove Liquidity Pool (LP) tokens\n      [Tag.LP_TOKEN],\n      // Make sure ChainId is for RPC endpoint above\n      ChainId.MAINNET,\n      // Signature date filter, keep tokens with latest signature in last 30 days\n      30,\n      // Keep tokens with more than 100 holders\n      100\n    ),\n    new ProviderTrusted(\n      'https://raw.githubusercontent.com/solflare-wallet/token-list/master/trusted-tokenlist.json',\n      // Filter out by tags, eg. remove Liquidity Pool (LP) tokens\n      [Tag.LP_TOKEN],\n        // Filter by chainId\n      ChainId.MAINNET,\n    ),\n  ],\n  [\n      new ProviderIgnore(\n          'https://raw.githubusercontent.com/solflare-wallet/token-list/master/ignore-tokenlist.json',\n          // Filter out by tags\n          [],\n          // Filter by chainId\n          ChainId.MAINNET,\n      ),\n  ]\n  )\n    \n\n  const tokenList = await generator.generateTokenList()\n\n  await writeFile('./solana-tokenlist.json', JSON.stringify(tokenList), 'utf8')\n\n  console.log('UTL Completed, the file was saved!')\n}\n\nmain()\n\n```\n\n\n## Token List Providers\nProviders are listed in an aggregator. If for example mint/token A is in both CoinGecko and Orca list, only one instance/data will be kept for the final token list, and this is determined based on whether CoinGecko or Orca is positioned higher in the list. If Orca is above CoinGecko, mint A from Orca will be kept, and CoinGecko's mint A will be ignored.\n\n_**Built-in provider sources**_ will be the Pruned Legacy Token List (`LTL`) and CoinGecko (`CG`).\nCoinGecko has high barrier of entry for tokens, and is generally excellent when it comes to maintaining token list (since it's their job and business to do so).\nLegacy token list will be pruned (remove invalid mints, filtering by holders, last activity, LP tokens, scam tokens; this processed was described in Telegram chat) and transformed into the new standardized format.\n\n[To-Do]  _**External Provider sources**_ (Orca, Raydium, Saber, etc..) can host and maintain their own list of verified tokens, that aggregator can use when generating unified token list. \nEach external provider will have to expose endpoint with a list of tokens they view as verified. This list will be in standardize format (which will include if token is LP-token, etc).\n\n[To-Do] Base external provider repo so any project (Orca, Raydium, Saber..) can host and expose their own verified token list with little developer effort. This allows them to serve as trusted providers for other.\n\n### CoinGecko Provider\nUses CoinGecko API to fetch all tokens with valid Solana mint address. \nToken' logoURI is fetched from CoinGecko also, while decimal is fetched from chain.\nThat is why this provider also requires Solana RPC mainnet endpoint.\n\n**Throttle notes:**\n\nCoinGecko Free API usually has 25-50 calls/min limit, to avoid `HTTP 429 Too Many Requests` use `batchCoinGecko: 25` \nand `throttleCoinGecko: 65 * 1000`\n\nWith CoinGecko Pro API Key, you can increase request sizes eg. `batchCoinGecko: 400`\n\n```ts\nnew ProviderCoinGecko(\n  COINGECKO_API_KEY,\n  RPC_URL,\n  { // ThrottleOptions\n    throttle: 1 * SECOND, // Add sleep after batch RPC request to avoid rate limits\n    throttleCoinGecko: 65 * SECONDS, // Add sleep after batch HTTP calls for CoinGecko\n    batchAccountsInfo: 100, // Batch RPC calls in single RPC request\n    batchCoinGecko: 25, // Batch CoinGecko token HTTP call\n  }\n)\n\n```\n\n\n### Legacy Token List Provider\nThis provider uses existing token list and pulls active and relevant tokens from it.\n\nThis is done in following steps:\n- Filter by chainId and tags \n- Remove by token content (remove already labeled scam and phishing)\n- Check if account is a mint (using getAccountInfo)\n- Remove by latest signature date\n- Remove by holders count\n\n**Caching:**\n\nSince RPC endpoints calls can fail or take long time on larger requests,\nthis provider caches few result sets to increase speed for subsequent runs.\n\nLatest signatures are cached and tokens with holder count larger than 1000 are cached.\nThis means that after first run, every other run will be faster.\n\nTo clear cache you can use:\n```javascript\nProviderLegacyToken.clearCache(ChainId.MAINNET)\nProviderLegacyToken.clearCache(ChainId.DEVNET)\n```\n\n\n**Throttle notes:**\n\nDifferent RPC endpoints have very different limits, to avoid `HTTP 429 Too Many Requests` try to thinker with `ThrottleOptions`.\n\n\n```ts\nnew ProviderLegacyToken(\n  CDN_URL,\n  RPC_URL, // Make sure RPC Endpoint is for ChainId specified below\n  { // ThrottleOptions\n    throttle: 1 * SECOND, // Add sleep after batch RPC request to avoid rate limits\n    batchSignatures: 100, // Batch RPC calls in single RPC request\n    batchAccountsInfo: 100, // Batch RPC calls in single RPC request\n    batchTokenHolders: 1, // Batch parallel RPC requests\n  },\n  [Tag.LP_TOKEN], // Filter out by tags, eg. remove LP tokens\n  ChainId.MAINNET, // Keep only chainId 101 tokens \n  30, // Signature date filter, keep tokens with latest signature in last 30 days\n  100, // Keep tokens with more than 100 holders \n)\n\n```\n","readmeFilename":"README.md"}