{"_id":"@aprameyakannan/llm-load-balancer","_rev":"2-28a76cd1c6dec768872fc185cbebd350","name":"@aprameyakannan/llm-load-balancer","dist-tags":{"latest":"1.1.0"},"versions":{"1.0.0":{"name":"@aprameyakannan/llm-load-balancer","version":"1.0.0","keywords":["llm","load-balancer","ai","openai","claude","gemini","typescript","api"],"author":{"name":"Aprameya Kannan","email":"aprameya.kannan@gmail.com"},"license":"MIT","_id":"@aprameyakannan/llm-load-balancer@1.0.0","maintainers":[{"name":"aprameyakannan","email":"aprameyakannan@gmail.com"}],"homepage":"https://github.com/aprameyakannan/llmLoadBalancer#readme","bugs":{"url":"https://github.com/aprameyakannan/llmLoadBalancer/issues"},"dist":{"shasum":"68be6bf640931e96595075a0cc8bca35a4cc3320","tarball":"https://registry.npmjs.org/@aprameyakannan/llm-load-balancer/-/llm-load-balancer-1.0.0.tgz","fileCount":23,"integrity":"sha512-c4pyjVgJ9VeUEDjOQjU4OnHaqOVJG9xtWUL2MlXTXuAncuUAKGy5KdCygQmqlK8e1FqBluIsMQAvYIkFELbcDQ==","signatures":[{"sig":"MEUCIFWMedWiOfKGviDKq8IF3jLtosKNfE6uvR6RGPJIYgGaAiEAvIilGpqKWqSdL+ZtcJxOGPke1yTosbiWRztAJgIOLp4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":57975},"main":"dist/index.js","types":"dist/index.d.ts","engines":{"node":">=16.0.0"},"gitHead":"b3a2f930919d1ba6d59370558f3454b5b9671a7c","scripts":{"dev":"tsc --watch","lint":"eslint src/**/*.ts","test":"jest","build":"tsc","prepare":"npm run build","lint:fix":"eslint src/**/*.ts --fix","test:watch":"jest --watch","prepublishOnly":"npm test && npm run lint"},"_npmUser":{"name":"aprameyakannan","email":"aprameyakannan@gmail.com"},"repository":{"url":"git+https://github.com/aprameyakannan/llmLoadBalancer.git","type":"git"},"_npmVersion":"11.4.2","description":"A TypeScript library for load balancing calls across multiple LLM APIs with support for various routing strategies","directories":{},"_nodeVersion":"23.11.0","dependencies":{"axios":"^1.5.0"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^29.7.0","eslint":"^8.49.0","ts-jest":"^29.1.1","typescript":"^5.2.2","@types/jest":"^29.5.5","@types/node":"^20.6.0","@typescript-eslint/parser":"^6.21.0","@typescript-eslint/eslint-plugin":"^6.21.0"},"_npmOperationalInternal":{"tmp":"tmp/llm-load-balancer_1.0.0_1753806570414_0.02294918325659001","host":"s3://npm-registry-packages-npm-production"}},"1.1.0":{"name":"@aprameyakannan/llm-load-balancer","version":"1.1.0","description":"A TypeScript library for load balancing calls across multiple LLM APIs with support for various routing strategies","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"build":"tsc","dev":"tsc --watch","test":"jest","test:watch":"jest --watch","lint":"eslint src/**/*.ts","lint:fix":"eslint src/**/*.ts --fix","prepare":"npm run build","prepublishOnly":"npm test && npm run lint"},"keywords":["llm","load-balancer","ai","openai","claude","gemini","typescript","api"],"author":{"name":"Aprameya Kannan","email":"aprameya.kannan@gmail.com"},"license":"MIT","repository":{"type":"git","url":"git+https://github.com/aprameyakannan/llmLoadBalancer.git"},"bugs":{"url":"https://github.com/aprameyakannan/llmLoadBalancer/issues"},"homepage":"https://github.com/aprameyakannan/llmLoadBalancer#readme","devDependencies":{"@types/jest":"^29.5.5","@types/node":"^20.6.0","@typescript-eslint/eslint-plugin":"^6.21.0","@typescript-eslint/parser":"^6.21.0","eslint":"^8.49.0","jest":"^29.7.0","ts-jest":"^29.1.1","typescript":"^5.2.2"},"dependencies":{"axios":"^1.5.0"},"engines":{"node":">=16.0.0"},"_id":"@aprameyakannan/llm-load-balancer@1.1.0","gitHead":"772d6999de08423f9f5363a2187f1340bf0173a1","_nodeVersion":"23.11.0","_npmVersion":"11.4.2","dist":{"integrity":"sha512-3YP0AHkiCfk62xFf2H90jK01oj2V21WdVpmie0tkHrLTWXcmfMVnGLjT+ZhrOvvE2nyB2ztPdY0yl+vLF98tqQ==","shasum":"725e1245da9c6a189c20ba90e59ce68792f3ac2a","tarball":"https://registry.npmjs.org/@aprameyakannan/llm-load-balancer/-/llm-load-balancer-1.1.0.tgz","fileCount":23,"unpackedSize":57302,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIHkBA9fV9aC5mF3KAebDo8D6LgSbt0+jFpoihwqNz/gzAiBDJ92CgGyL035loTrSByM/jNWRjQCjmsInzaYELWD0/g=="}]},"_npmUser":{"name":"aprameyakannan","email":"aprameyakannan@gmail.com"},"directories":{},"maintainers":[{"name":"aprameyakannan","email":"aprameyakannan@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/llm-load-balancer_1.1.0_1753852321599_0.06071599101112879"},"_hasShrinkwrap":false}},"time":{"created":"2025-07-29T16:29:30.321Z","modified":"2025-07-30T05:12:02.020Z","1.0.0":"2025-07-29T16:29:30.614Z","1.1.0":"2025-07-30T05:12:01.816Z"},"bugs":{"url":"https://github.com/aprameyakannan/llmLoadBalancer/issues"},"author":{"name":"Aprameya Kannan","email":"aprameya.kannan@gmail.com"},"license":"MIT","homepage":"https://github.com/aprameyakannan/llmLoadBalancer#readme","keywords":["llm","load-balancer","ai","openai","claude","gemini","typescript","api"],"repository":{"type":"git","url":"git+https://github.com/aprameyakannan/llmLoadBalancer.git"},"description":"A TypeScript library for load balancing calls across multiple LLM APIs with support for various routing strategies","maintainers":[{"name":"aprameyakannan","email":"aprameyakannan@gmail.com"}],"readme":"# LLM Load Balancer\n\nA TypeScript library for load balancing calls across multiple LLM APIs with support for various routing strategies.\n\n## Features\n\n- **Multiple Load Balancing Strategies**: Round-robin, failover, weighted, and custom strategies\n- **Multi-Provider Support**: OpenAI, Claude, Gemini, Cohere, Mistral, and extensible for custom providers\n- **Built-in Statistics**: Track performance, success rates, and health metrics\n- **Error Handling & Retries**: Automatic retries with exponential backoff\n- **Health Checks**: Monitor provider health and automatic failover\n- **Dynamic Provider Management**: Add/remove providers at runtime\n- **TypeScript Support**: Full type safety and IntelliSense support\n\n## Installation\n\n```bash\nnpm install @aprameyakannan/llm-load-balancer\n```\n\n## Quick Start\n\n```typescript\nimport { createLLMBalancer } from '@aprameyakannan/llm-load-balancer';\n\nconst llm = createLLMBalancer({\n  strategy: 'round-robin',\n  providers: [\n    { name: 'openai', apiKey: 'your-openai-key', model: 'gpt-3.5-turbo' },\n    { name: 'claude', apiKey: 'your-claude-key', model: 'claude-3-haiku' },\n    { name: 'gemini', apiKey: 'your-gemini-key', model: 'gemini-pro' },\n  ],\n});\n\nconst response = await llm.request({\n  prompt: 'Write a funny dad joke.',\n});\n\nconsole.log(response.content);\n```\n\n## Load Balancing Strategies\n\n### Round Robin\nDistributes requests evenly across all providers in sequence.\n\n```typescript\nconst llm = createLLMBalancer({\n  strategy: 'round-robin',\n  providers: [/* ... */],\n});\n```\n\n### Failover\nUses the first healthy provider, falling back to others if needed.\n\n```typescript\nconst llm = createLLMBalancer({\n  strategy: 'failover',\n  providers: [/* ... */],\n});\n```\n\n### Weighted\nDistributes requests based on provider weights.\n\n```typescript\nconst llm = createLLMBalancer({\n  strategy: 'weighted',\n  providers: [\n    { name: 'openai', apiKey: '...', model: 'gpt-4', weight: 3 },\n    { name: 'claude', apiKey: '...', model: 'claude-3-haiku', weight: 2 },\n    { name: 'gemini', apiKey: '...', model: 'gemini-pro', weight: 1 },\n  ],\n});\n```\n\n### Custom Strategy\nUse your own load balancing logic.\n\n```typescript\nconst customStrategy = (providers) => {\n  // Your custom logic here\n  return providers.find(p => p.name === 'preferred') || providers[0];\n};\n\nconst llm = createLLMBalancer({\n  strategy: 'custom',\n  providers: [/* ... */],\n  customStrategy,\n});\n```\n\n## Configuration Options\n\n```typescript\ninterface LoadBalancerConfig {\n  strategy: 'round-robin' | 'failover' | 'weighted' | 'custom';\n  providers: ProviderConfig[];\n  customStrategy?: (providers: ProviderConfig[]) => ProviderConfig;\n  globalTimeout?: number;        // Global timeout in ms (default: 30000)\n  maxRetries?: number;          // Max retry attempts (default: 3)\n  retryDelay?: number;          // Base retry delay in ms (default: 1000)\n}\n\ninterface ProviderConfig {\n  name: string;                 // Provider identifier\n  apiKey: string;              // API key\n  model: string;               // Model name\n  baseUrl?: string;            // Custom API base URL\n  weight?: number;             // Weight for weighted strategy (default: 1)\n  timeout?: number;            // Provider-specific timeout\n  maxRetries?: number;         // Provider-specific max retries\n}\n```\n\n## Request Options\n\n```typescript\ninterface LLMRequest {\n  prompt: string;              // The text prompt\n  maxTokens?: number;          // Maximum tokens to generate\n  temperature?: number;        // Creativity/randomness (0-1)\n  topP?: number;              // Nucleus sampling parameter\n  stream?: boolean;           // Whether to stream the response\n}\n```\n\n## Monitoring and Statistics\n\n### Get Statistics\n```typescript\nconst stats = llm.getStats();\nconsole.log(stats);\n// {\n//   openai: {\n//     totalRequests: 10,\n//     successfulRequests: 8,\n//     failedRequests: 2,\n//     averageResponseTime: 1500,\n//     isHealthy: true,\n//     lastRequestTime: Date\n//   }\n// }\n```\n\n### Health Checks\n```typescript\nconst healthStatus = await llm.healthCheck();\nconsole.log(healthStatus);\n// { openai: true, claude: false, gemini: true }\n```\n\n### Get Healthy Providers\n```typescript\nconst healthyProviders = llm.getHealthyProviders();\n```\n\n## Dynamic Provider Management\n\n### Add Provider\n```typescript\nllm.addProvider({\n  name: 'cohere',\n  apiKey: 'your-cohere-key',\n  model: 'command',\n  weight: 2\n});\n```\n\n### Remove Provider\n```typescript\nllm.removeProvider('claude');\n```\n\n## Supported Providers\n\n| Provider | Models | Notes |\n|----------|--------|-------|\n| OpenAI | gpt-3.5-turbo, gpt-4, etc. | Chat completions API |\n| Claude | claude-3-haiku, claude-3-sonnet, etc. | Anthropic API |\n| Gemini | gemini-pro, gemini-pro-vision | Google AI API |\n| Cohere | command, command-light | Generate API |\n| Mistral | mistral-tiny, mistral-small, etc. | Chat completions API |\n\n## Error Handling\n\nThe library provides comprehensive error handling with custom error types:\n\n```typescript\nimport { LLMError, LoadBalancerError } from 'llm-load-balancer';\n\ntry {\n  const response = await llm.request({ prompt: 'Hello' });\n} catch (error) {\n  if (error instanceof LoadBalancerError) {\n    console.log('All providers failed:', error.errors);\n  } else if (error instanceof LLMError) {\n    console.log(`Provider ${error.provider} failed:`, error.message);\n  }\n}\n```\n\n## Advanced Usage\n\n### Custom Provider Implementation\n```typescript\nimport { BaseProvider, LLMRequest, LLMResponse } from 'llm-load-balancer';\n\nclass CustomProvider extends BaseProvider {\n  async makeRequest(request: LLMRequest): Promise<LLMResponse> {\n    // Your custom implementation\n    return {\n      content: 'Custom response',\n      model: this.config.model,\n      provider: this.config.name,\n      usage: { promptTokens: 10, completionTokens: 5, totalTokens: 15 }\n    };\n  }\n}\n```\n\n### Environment Variables\nFor production use, store API keys in environment variables:\n\n```typescript\nconst llm = createLLMBalancer({\n  strategy: 'round-robin',\n  providers: [\n    { name: 'openai', apiKey: process.env.OPENAI_API_KEY!, model: 'gpt-3.5-turbo' },\n    { name: 'claude', apiKey: process.env.CLAUDE_API_KEY!, model: 'claude-3-haiku' },\n  ],\n});\n```\n\n## Examples\n\nSee the `examples/` directory for complete working examples:\n- Basic usage\n- All strategies demonstration\n- Error handling\n- Health monitoring\n- Custom providers\n","readmeFilename":"README.md"}