{"_id":"@brochington/arrowstore","name":"@brochington/arrowstore","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@brochington/arrowstore","version":"1.0.0","type":"module","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"types":"./dist/index.d.ts","scripts":{"build":"rslib build","check":"biome check --write","dev":"rslib build --watch","format":"biome format --write","test":"jest"},"devDependencies":{"@biomejs/biome":"^1.9.4","@rslib/core":"^0.9.2","@types/node":"^22.15.30","ts-jest":"^29.3.4","typescript":"^5.8.3"},"dependencies":{"@types/jest":"^29.5.14","apache-arrow":"^20.0.0"},"_id":"@brochington/arrowstore@1.0.0","gitHead":"f2811a38dc312b6908f63824d346c4627eb4fc5f","description":"ArrowStore is a high-performance data store implementation using Apache Arrow Tables with vectorized operations for improved performance and memory efficiency. It provides a comprehensive API for data manipulation, filtering, sorting, and aggregation oper","_nodeVersion":"22.15.0","_npmVersion":"10.9.2","dist":{"integrity":"sha512-zgwT1f3DakorPT0wGqzf+IJTgpBsyxq2UrK8vvOIwfY8qSVb4ojRbmCxb9mH2HNOoJZU5kxgKvtA+T9sOm/x/A==","shasum":"ce8d3cbee8b56dee32250c8742ff8ea3220074bc","tarball":"https://registry.npmjs.org/@brochington/arrowstore/-/arrowstore-1.0.0.tgz","fileCount":13,"unpackedSize":123596,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQDWBbwRtZLVj0GoTofHSo+467tmLdoR4GSsqV4j7aRJXQIhAKZO8tLaDfgn87GaM8+c3x8iIvAtmpRiDfRWuGQ3KTa9"}]},"_npmUser":{"name":"brochington","email":"brochington@gmail.com"},"directories":{},"maintainers":[{"name":"brochington","email":"brochington@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/arrowstore_1.0.0_1749436476497_0.4751665354677097"},"_hasShrinkwrap":false}},"time":{"created":"2025-06-09T02:34:36.495Z","1.0.0":"2025-06-09T02:34:36.672Z","modified":"2025-06-09T02:34:37.299Z"},"maintainers":[{"name":"brochington","email":"brochington@gmail.com"}],"description":"ArrowStore is a high-performance data store implementation using Apache Arrow Tables with vectorized operations for improved performance and memory efficiency. It provides a comprehensive API for data manipulation, filtering, sorting, and aggregation oper","readme":"# ArrowStore\n\nArrowStore is a high-performance data store implementation using Apache Arrow Tables with vectorized operations for improved performance and memory efficiency. It provides a comprehensive API for data manipulation, filtering, sorting, and aggregation operations with lazy evaluation.\n\n## Core Features\n\n- **Lazy Evaluation**: Operations are queued and only executed when data is actually needed\n- **Vectorized Operations**: Optimized for performance using Apache Arrow's columnar memory format\n- **Memory Efficiency**: Batch processing and smart memory management for large datasets\n- **Comprehensive Query API**: Rich set of operations for filtering, transforming, and analyzing data\n- **SQL-like Capabilities**: Support for SQL-like filtering and queries\n\n## Installation\n\n```bash\nnpm install @brochington/arrowstore\n```\n\n## Basic Usage\n\n```typescript\nimport { ArrowStore, Aggregations } from 'arrow-store';\nimport { tableFromArrays } from 'apache-arrow';\n\n// Create Arrow table from data\nconst data = {\n  id: [1, 2, 3, 4, 5],\n  name: ['Alice', 'Bob', 'Charlie', 'Dave', 'Eve'],\n  age: [25, 30, 35, 40, 45],\n  department: ['Engineering', 'Product', 'Engineering', 'HR', 'Product']\n};\n\nconst table = tableFromArrays(data);\n\n// Create ArrowStore instance\nconst store = new ArrowStore(table);\n\n// Chain operations (these are lazily evaluated)\nconst result = await store\n  .filter([\n    { field: 'age', filter: { op: 'gte', value: 30 } }\n  ])\n  .sort([{ field: 'name', direction: 'asc' }])\n  .getAll();\n\nconsole.log(result);\n```\n\n## Constructor\n\n### `new ArrowStore<T>(table, schema?, options?)`\n\nCreates a new ArrowStore instance.\n\n**Parameters:**\n- `table`: Arrow Table - The table containing the data\n- `schema?`: TableSchema - Optional schema definition\n- `options?`: ArrowStoreOptions - Optional configuration options\n\n**Type Parameters:**\n- `T`: Record<string, any> - Type of the row objects\n\n**Example:**\n```typescript\nconst store = new ArrowStore(table);\n```\n\n## Core Methods\n\n### Data Retrieval\n\n#### `getAll(): Promise<T[]>`\n\nReturns all data from the store.\n\n**Returns:** Promise resolving to an array of row objects\n\n**Example:**\n```typescript\nconst allData = await store.getAll();\n```\n\n#### `count(): Promise<number>`\n\nCounts the number of rows in the store.\n\n**Returns:** Promise resolving to the number of rows\n\n**Example:**\n```typescript\nconst rowCount = await store.count();\n```\n\n#### `getSource(): Table`\n\nGets the underlying Apache Arrow Table.\n\n**Returns:** The Apache Arrow Table\n\n**Example:**\n```typescript\nconst arrowTable = store.getSource();\n```\n\n### Filtering\n\n#### `filter<R extends T = T>(filters: FilterCondition<T>[]): ArrowStore<R>`\n\nFilters the data based on the provided filter conditions.\n\n**Parameters:**\n- `filters`: Array of filter conditions to apply\n\n**Returns:** A new ArrowStore instance with filtered data\n\n**Example:**\n```typescript\nconst filteredStore = store.filter([\n  { field: 'age', filter: { op: 'gte', value: 30 } },\n  { \n    OR: [\n      { field: 'department', filter: { op: 'eq', value: 'Engineering' } },\n      { field: 'department', filter: { op: 'eq', value: 'Product' } }\n    ]\n  }\n]);\n```\n\n#### `filterSql<R extends T = T>(sqlFilter: string): ArrowStore<R>`\n\nFilters data using a SQL-like WHERE clause.\n\n**Parameters:**\n- `sqlFilter`: SQL-like WHERE clause string (without the \"WHERE\" keyword)\n\n**Returns:** A new ArrowStore instance with filtered data\n\n**Example:**\n```typescript\nconst filteredStore = store.filterSql(\n  \"age >= 30 AND department IN ('Engineering', 'Product')\"\n);\n```\n\n#### `filterEquals<R extends T = T>(simpleFilters: Partial<T>): ArrowStore<R>`\n\nFilters data with simple field-value equality pairs.\n\n**Parameters:**\n- `simpleFilters`: Object where keys are field names and values are what to match\n\n**Returns:** A new ArrowStore instance with filtered data\n\n**Example:**\n```typescript\nconst filteredStore = store.filterEquals({\n  department: 'Engineering',\n  active: true\n});\n```\n\n### Transformation\n\n#### `select<K extends keyof T>(fields: K[]): ArrowStore<Pick<T, K>>`\n\nSelects specific fields/columns.\n\n**Parameters:**\n- `fields`: Array of field names to select\n\n**Returns:** A new ArrowStore instance with only the selected fields\n\n**Example:**\n```typescript\nconst nameAndAgeStore = store.select(['name', 'age']);\n```\n\n#### `map<R extends Record<string, any>>(mapFn: (item: T) => R, resultSchema?: TableSchema): ArrowStore<R>`\n\nMaps each row to a new object structure.\n\n**Parameters:**\n- `mapFn`: Function to transform each row\n- `resultSchema?`: Optional schema for the transformed data\n\n**Returns:** A new ArrowStore instance with mapped data\n\n**Example:**\n```typescript\nconst mappedStore = store.map(person => ({\n  fullName: `${person.firstName} ${person.lastName}`,\n  birthYear: new Date().getFullYear() - person.age\n}));\n```\n\n### Sorting and Pagination\n\n#### `sort(options: SortOptions[]): ArrowStore<T>`\n\nSorts the data based on one or more fields.\n\n**Parameters:**\n- `options`: Array of sort configurations with field and direction\n\n**Returns:** A new ArrowStore instance with sorted data\n\n**Example:**\n```typescript\nconst sortedStore = store.sort([\n  { field: 'age', direction: 'desc' },\n  { field: 'name', direction: 'asc' }\n]);\n```\n\n#### `paginate(page: number, pageSize: number): ArrowStore<T>`\n\nPaginates data with the specified page and page size.\n\n**Parameters:**\n- `page`: Page number (1-based)\n- `pageSize`: Number of items per page\n\n**Returns:** A new ArrowStore instance with paginated data\n\n**Example:**\n```typescript\nconst pageTwo = store.paginate(2, 10); // Second page with 10 items per page\n```\n\n#### `slice(start: number, end: number): ArrowStore<T>`\n\nReturns a slice of the data from start to end.\n\n**Parameters:**\n- `start`: Start index (inclusive)\n- `end`: End index (exclusive)\n\n**Returns:** A new ArrowStore instance with the sliced data\n\n**Example:**\n```typescript\nconst slicedStore = store.slice(10, 20); // Items 10-19\n```\n\n### Aggregation\n\n#### `groupBy<K extends keyof T, R extends Record<string, any>>(field: K, aggregations: Record<string, (values: any[]) => any>): ArrowStore<R>`\n\nGroups data by a field and computes aggregations.\n\n**Parameters:**\n- `field`: Field to group by\n- `aggregations`: Object mapping output field names to aggregation functions\n\n**Returns:** A new ArrowStore instance with grouped and aggregated data\n\n**Example:**\n```typescript\nconst departmentStats = store.groupBy('department', {\n  count: Aggregations.count(),\n  avgAge: Aggregations.avg('age'),\n  totalSalary: Aggregations.sum('salary')\n});\n\n// Result structure example:\n// [\n//   { department: 'Engineering', count: 2, avgAge: 30, totalSalary: 200000 },\n//   { department: 'Product', count: 2, avgAge: 37.5, totalSalary: 220000 },\n//   { department: 'HR', count: 1, avgAge: 40, totalSalary: 90000 }\n// ]\n```\n\n### Reduction Operations\n\n#### `reduce<R>(reducer: (accumulator: R, current: T, index: number) => R, initialValue: R): Promise<R>`\n\nReduces the data to a single value.\n\n**Parameters:**\n- `reducer`: Function to apply to each row with an accumulator\n- `initialValue`: Initial value for the accumulator\n\n**Returns:** Promise resolving to the accumulated result\n\n**Example:**\n```typescript\nconst totalAge = await store.reduce(\n  (sum, person, index) => sum + person.age, \n  0\n);\n```\n\n#### `fold<R>(folder: (accumulator: R, current: T, index: number) => R, initialValue: R): Promise<R>`\n\nAlias for `reduce`.\n\n#### `toSet<K>(keyFn?: (item: T) => K): Promise<Set<K | T>>`\n\nConverts the data to a Set with optional key extraction.\n\n**Parameters:**\n- `keyFn?`: Optional function to extract a key from each row\n\n**Returns:** Promise resolving to a Set of unique values\n\n**Example:**\n```typescript\n// Get unique departments\nconst departments = await store.toSet(person => person.department);\n```\n\n#### `toMap<K, V = T>(keyFn: (item: T) => K, valueFn?: (item: T) => V): Promise<Map<K, V>>`\n\nConverts the data to a Map with keys and values extracted from rows.\n\n**Parameters:**\n- `keyFn`: Function to extract a key from each row\n- `valueFn?`: Optional function to transform each row into a value\n\n**Returns:** Promise resolving to a Map\n\n**Example:**\n```typescript\n// Create a map of id -> name\nconst idToName = await store.toMap(\n  person => person.id,\n  person => person.name\n);\n```\n\n## Memory Management\n\n#### `flush(): Promise<ArrowStore<T>>`\n\nExecutes all pending operations and returns a new store.\n\n**Returns:** Promise resolving to a new ArrowStore with computed results\n\n**Example:**\n```typescript\nconst computedStore = await store.flush();\n```\n\n#### `estimateMemoryUsage(): Promise<MemoryUsageInfo>`\n\nEstimates memory usage of the store.\n\n**Returns:** Promise resolving to detailed memory usage information\n\n**Example:**\n```typescript\nconst memoryInfo = await store.estimateMemoryUsage();\nconsole.log(`Total estimated memory: ${memoryInfo.humanReadable.total}`);\n```\n\n## Column Operations\n\n#### `sumColumn(columnName: string): Promise<number>`\n\nComputes the sum of values in a numeric column.\n\n**Parameters:**\n- `columnName`: Name of the column to sum\n\n**Returns:** Promise resolving to the sum\n\n**Example:**\n```typescript\nconst totalSalary = await store.sumColumn('salary');\n```\n\n#### `averageColumn(columnName: string): Promise<number | null>`\n\nComputes the average of values in a numeric column.\n\n**Parameters:**\n- `columnName`: Name of the column to average\n\n**Returns:** Promise resolving to the average or null if no data\n\n**Example:**\n```typescript\nconst avgAge = await store.averageColumn('age');\n```\n\n#### `minMaxColumn(columnName: string): Promise<{ min: any; max: any }>`\n\nFinds the minimum and maximum values in a column.\n\n**Parameters:**\n- `columnName`: Name of the column to analyze\n\n**Returns:** Promise resolving to an object with min and max values\n\n**Example:**\n```typescript\nconst { min, max } = await store.minMaxColumn('age');\nconsole.log(`Age range: ${min} - ${max}`);\n```\n\n#### `countDistinct(columnName: string): Promise<number>`\n\nCounts distinct values in a column.\n\n**Parameters:**\n- `columnName`: Name of the column to analyze\n\n**Returns:** Promise resolving to the count of distinct values\n\n**Example:**\n```typescript\nconst departmentCount = await store.countDistinct('department');\n```\n\n## Advanced Usage Examples\n\n### Chaining Operations\n\n```typescript\nconst results = await store\n  .filter([{ field: 'active', filter: { op: 'eq', value: true } }])\n  .select(['id', 'name', 'department', 'salary'])\n  .sort([{ field: 'salary', direction: 'desc' }])\n  .slice(0, 10)\n  .getAll();\n```\n\n### Complex Filtering\n\n```typescript\nimport { and, or, not, field, gt, eq, inArray } from 'arrow-store';\n\nconst filtered = store.filter([\n  and([\n    field('age', gt(30)),\n    or([\n      field('department', eq('Engineering')),\n      field('department', eq('Product'))\n    ]),\n    not(field('isContractor', eq(true)))\n  ])\n]);\n```\n\n### Using SQL-like Filters\n\n```typescript\nconst filtered = store.filterSql(\n  \"age > 30 AND department IN ('Engineering', 'Product') AND NOT isContractor = true\"\n);\n```\n\n### Grouping and Aggregation\n\n```typescript\nconst stats = await store\n  .groupBy('department', {\n    count: Aggregations.count(),\n    avgAge: Aggregations.avg('age'),\n    minSalary: Aggregations.min('salary'),\n    maxSalary: Aggregations.max('salary')\n  })\n  .sort([{ field: 'count', direction: 'desc' }])\n  .getAll();\n```\n\n### Processing Large Datasets Efficiently\n\n```typescript\n// For very large datasets, use batch processing patterns\nconst store = new ArrowStore(largeTable);\n\n// Use aggregations rather than loading all data\nconst summary = {\n  total: await store.count(),\n  averageAge: await store.averageColumn('age'),\n  departmentCounts: await store\n    .groupBy('department', { count: Aggregations.count() })\n    .getAll()\n};\n\n// Only retrieve necessary data\nconst topEmployees = await store\n  .sort([{ field: 'performance', direction: 'desc' }])\n  .slice(0, 100)  // Only get top 100\n  .select(['id', 'name', 'performance'])  // Only select needed fields\n  .getAll();\n```\n\n## Type Definitions\n\nFor complete type definitions, refer to the source code or TypeScript declaration files.","readmeFilename":"README.md","_rev":"1-73dfef61014ae1f88df26e52d4bc3522"}