{"_id":"@delta-ltsc/ml","_rev":"2-8ec1127242aa742abcba84127cc395cb","name":"@delta-ltsc/ml","dist-tags":{"latest":"0.3.1"},"versions":{"0.3.1":{"name":"@delta-ltsc/ml","version":"0.3.1","keywords":["compression","llm","tokens","transformer","ml","embeddings"],"author":{"name":"Triage Sec","email":"nicks@triage-sec.com"},"license":"MIT","_id":"@delta-ltsc/ml@0.3.1","maintainers":[{"name":"nicksriv","email":"nicks@triage-sec.com"}],"contributors":[{"name":"Nikhil Srivastava"},{"name":"Omansh Bainsla"},{"name":"Sahil Chatiwala"}],"homepage":"https://github.com/delta-ltsc/delta#readme","bugs":{"url":"https://github.com/delta-ltsc/delta/issues"},"dist":{"shasum":"ddd18645a320d5ab9e2b79468393d0857355b1e6","tarball":"https://registry.npmjs.org/@delta-ltsc/ml/-/ml-0.3.1.tgz","fileCount":18,"integrity":"sha512-UPb+9zL/6PvRsKtDrB6tOWn1Qy+yZ3wPy8/Hf+xQv3IrykMixEKrlTRKYo7itISVq0vQYMPmRyeF8cr5G3cjjA==","signatures":[{"sig":"MEQCIEiJZsggavDYsNQNP/yESkvisX8HAFk7tgEzKjick4ixAiAwpQ6LY1L9m3C4nqmIMy48YXskVX/D0aOuoZdYIiIGXg==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":67816},"main":"./dist/esm/index.js","type":"module","types":"./dist/types/index.d.ts","module":"./dist/esm/index.js","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js"}},"gitHead":"57b3fc5157a3ecd50d50ad33b7fae10de15a7079","scripts":{"lint":"eslint src --ext .ts","test":"vitest","build":"npm run build:esm && npm run build:types","clean":"rm -rf dist","build:esm":"tsc -p tsconfig.esm.json","build:types":"tsc -p tsconfig.types.json","prepublishOnly":"npm run clean && npm run build"},"_npmUser":{"name":"nicksriv","email":"nicks@triage-sec.com"},"repository":{"url":"git+https://github.com/delta-ltsc/delta.git","type":"git"},"_npmVersion":"10.8.2","description":"ML features for Delta LTSC - Pattern importance scoring and quality prediction","directories":{},"sideEffects":false,"_nodeVersion":"20.20.0","_hasShrinkwrap":false,"devDependencies":{"eslint":"^8.55.0","vitest":"^1.0.0","typescript":"^5.3.0","@types/node":"^20.10.0"},"peerDependencies":{"@delta-ltsc/sdk":"^0.3.1"},"_npmOperationalInternal":{"tmp":"tmp/ml_0.3.1_1769850595187_0.7579870045090149","host":"s3://npm-registry-packages-npm-production"}}},"time":{"created":"2026-01-31T09:09:55.111Z","modified":"2026-02-01T01:51:54.922Z","0.3.1":"2026-01-31T09:09:55.334Z"},"bugs":{"url":"https://github.com/delta-ltsc/delta/issues"},"author":{"name":"Triage Sec","email":"nicks@triage-sec.com"},"license":"MIT","homepage":"https://github.com/delta-ltsc/delta#readme","keywords":["compression","llm","tokens","transformer","ml","embeddings"],"repository":{"url":"git+https://github.com/delta-ltsc/delta.git","type":"git"},"description":"ML features for Delta LTSC - Pattern importance scoring and quality prediction","contributors":[{"name":"Nikhil Srivastava"},{"name":"Omansh Bainsla"},{"name":"Sahil Chatiwala"}],"maintainers":[{"email":"nicks@triage-sec.com","name":"nicksriv"},{"email":"sahil@triage-sec.com","name":"sahil_triage"}],"readme":"# @delta-ltsc/ml\n\n[![npm](https://img.shields.io/npm/v/@delta-ltsc/ml)](https://www.npmjs.com/package/@delta-ltsc/ml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](../../LICENSE)\n\nOptional ML features for **Delta LTSC** - Pattern importance scoring, quality prediction, and adaptive region detection.\n\n## Installation\n\n```bash\nnpm install @delta-ltsc/ml @delta-ltsc/sdk\n```\n\nNote: `@delta-ltsc/sdk` is a required peer dependency.\n\n## Features\n\n- **Pattern Importance Scoring** - Determine which patterns are semantically important and should be preserved\n- **Quality Prediction** - Predict if compression will degrade model performance before applying it\n- **Region Detection** - Identify system prompts, user input, and context for adaptive compression strategies\n\n## Pattern Importance\n\nScore patterns to preserve semantically important content:\n\n```typescript\nimport { PositionalImportanceScorer, filterByImportance } from '@delta-ltsc/ml';\nimport { discoverPatterns } from '@delta-ltsc/sdk';\n\nconst scorer = new PositionalImportanceScorer({ decayRate: 2.0 });\nconst patterns = await discoverPatterns(tokens);\nconst scores = await scorer.scorePatterns(tokens, patterns);\n\n// Filter out high-importance patterns (preserve them from compression)\nconst safeToCompress = filterByImportance(patterns, scores, 0.8);\n```\n\n### Embedding-Based Scoring\n\nFor more accurate importance scoring using an embedding model:\n\n```typescript\nimport { EmbeddingImportanceScorer } from '@delta-ltsc/ml';\n\nconst scorer = new EmbeddingImportanceScorer(embeddingProvider, {\n  contextWindow: 5,\n});\n\nconst scores = await scorer.scorePatterns(tokens, patterns);\n```\n\n## Quality Prediction\n\nPredict if compressed output will maintain quality before committing:\n\n```typescript\nimport { createQualityPredictor } from '@delta-ltsc/ml';\nimport { compress } from '@delta-ltsc/sdk';\n\nconst predictor = createQualityPredictor();\nconst result = await compress(tokens);\nconst prediction = await predictor.predict(result);\n\nif (!prediction.acceptable) {\n  console.log(`Recommendation: ${prediction.recommendation}`);\n  // 'accept' | 'retry_conservative' | 'skip_compression'\n}\n```\n\n### Quality Features\n\n```typescript\nconsole.log(prediction.features);\n// {\n//   compressionRatio: 0.65,\n//   dictionaryOverhead: 0.15,\n//   diversityReduction: 0.2,\n//   averagePatternLength: 4.5,\n//   patternCount: 12,\n// }\n```\n\n## Region Detection\n\nDetect semantic regions for adaptive compression strategies:\n\n```typescript\nimport { detectRegions, RegionType, filterPatternsByRegion } from '@delta-ltsc/ml';\n\nconst regions = detectRegions(tokens, {\n  systemMarkers: [[58, 71905, 60]], // [SYSTEM] token sequence\n  retentionTargets: {\n    [RegionType.SYSTEM]: 0.98,  // Minimal compression (preserve instructions)\n    [RegionType.USER]: 0.85,    // Moderate compression\n    [RegionType.CONTEXT]: 0.6,  // Aggressive compression (RAG content)\n  },\n});\n\n// Filter patterns based on region constraints\nconst filtered = filterPatternsByRegion(patterns, regions, tokens);\n```\n\n### Region Types\n\n| Region | Description | Default Retention |\n|--------|-------------|-------------------|\n| `SYSTEM` | System instructions | 98% (minimal compression) |\n| `USER` | User input | 85% (moderate) |\n| `CONTEXT` | Injected context/documents | 60% (aggressive) |\n| `CODE` | Code blocks | 80% (moderate) |\n| `UNKNOWN` | Default region | 75% |\n\n## Custom Embedding Provider\n\nImplement the `EmbeddingProvider` interface for your embedding model:\n\n```typescript\nimport type { EmbeddingProvider } from '@delta-ltsc/ml';\n\nclass OpenAIEmbeddings implements EmbeddingProvider {\n  async embed(tokens: readonly number[]): Promise<Float32Array> {\n    const text = tokenizer.decode(tokens);\n    const response = await openai.embeddings.create({\n      model: 'text-embedding-3-small',\n      input: text,\n    });\n    return new Float32Array(response.data[0].embedding);\n  }\n\n  dimension(): number {\n    return 1536;\n  }\n}\n\nconst scorer = new EmbeddingImportanceScorer(new OpenAIEmbeddings());\n```\n\n## API Reference\n\n### Importance Scoring\n\n| Export | Description |\n|--------|-------------|\n| `PositionalImportanceScorer` | Score patterns by position (earlier = more important) |\n| `EmbeddingImportanceScorer` | Score patterns by contextual diversity |\n| `CombinedImportanceScorer` | Combine positional and embedding scoring |\n| `adjustPrioritiesByImportance()` | Adjust pattern priorities based on scores |\n| `filterByImportance()` | Filter out high-importance patterns |\n\n### Quality Prediction\n\n| Export | Description |\n|--------|-------------|\n| `HeuristicQualityPredictor` | Rule-based quality prediction |\n| `EmbeddingQualityPredictor` | Enhanced prediction with embedding similarity |\n| `createQualityPredictor()` | Factory function for creating predictors |\n\n### Region Detection\n\n| Export | Description |\n|--------|-------------|\n| `detectRegions()` | Detect semantic regions in token sequence |\n| `detectRegionsHeuristic()` | Simple heuristic-based detection |\n| `filterPatternsByRegion()` | Filter patterns based on region constraints |\n| `getRegionCompressionSettings()` | Get default settings for a region type |\n| `RegionType` | Enum of available region types |\n\n## License\n\nMIT License - see [LICENSE](../../LICENSE) for details.\n\n## Contributors\n\nBuilt by [Triage Sec](https://triage-sec.com) - an applied team of researchers and engineers working towards building resiliency for AI systems.\n\n- Nikhil Srivastava (University of California, Berkeley)\n- Omansh Bainsla (Georgia Tech)\n- Sahil Chatiwala (Georgia Tech)\n","readmeFilename":"README.md"}