\n \n \n
Dead simple data pipeline utility belt.
\n\n\n# vandelay [![NPM version][npm-image]][npm-url] [![Downloads][downloads-image]][npm-url] [![Build Status][travis-image]][travis-url]\n\n\n## Install\n\n```\nnpm install vandelay --save\n```\n\n## Example - Flat File\n\n```js\nimport { tap, fetch, transform, parse } from 'vandelay'\n\nfetch({\n url: 'http://google.com/example.geojson',\n parser: parse('geojson')\n})\n .pipe(transform(async (row) => {\n const external = await otherApi(row.field)\n return {\n ...row,\n external\n }\n }))\n .pipe(tap(async (row, meta) => {\n // send row to an external api, db, or whatever!\n }))\n```\n\n## Example - API\n\n```js\nimport { tap, fetch, transform, parse } from 'vandelay'\n\nfetch({\n url: 'http://google.com/api/example',\n parser: parse('json', { selector: 'results.*' }),\n pagination: {\n offsetParam: 'offset',\n limitParam: 'limit'\n }\n})\n .pipe(transform(async (row, meta) => {\n const external = await otherApi(row.field)\n return {\n ...row,\n external\n }\n }))\n .pipe(tap(async (row, meta) => {\n // send row to an external api, db, or whatever!\n }))\n```\n\n## API\n\n### fetch(source[, options])\n\nReturns a stream that fetches the given source and emits the parsed and selected objects.\n\n#### source\n\n- url - Required `String`\n- parser - Required `Function`\n- pagination - Optional `Object`\n - offsetParam - Required `String` (if not using pageParam)\n - pageParam - Required `String` (if not using offsetParam)\n - limitParam - Required `String`\n - startPage - Optional `Number`, defaults to 0\n - limit - Required `Number`\n - nextPageSelector - Optional `String`\n - If needed, you may provide multiple selectors as an array (`nextPageSelector: [ 'a.*', 'b.*' ]`)\n - If provided, all other pagination options will only be applied on page 1 and the selector will be used to pluck the next page URL going forward\n- setup - Optional `Function` or `String`\n - Asynchronous function, runs once before the request starts for each source. Receives `source` and `meta` as arguments.\n - First argument is the source object being set up.\n - Second arguments is a meta information argument, that contains a context key if provided.\n - If it a string, it will compile it and sandbox it using [vm2](https://github.com/patriksimek/vm2).\n - Returns an object that controls request parameters.\n- oauth - Optional `Object`\n - grant - Required `Object`\n - url - Required `String`\n - OAuth2 API URL\n - type - Required `String`\n - Grant type, can be any [OAuth2 grant type](https://oauth.net/2/grant-types/)\n- headers - Optional `Object`\n- query - Optional `Object`\n\n#### options\n\n- concurrency - Optional `Number`, defaults to 8\n- timeout - Optional `Number`\n - Timeout for the entire request, defaults to one day\n- connectTimeout - Optional `Number`\n - Timeout to establish the initial connection, defaults to five minutes\n- context - Optional `Object`\n - If specified, will be templated into the URL via [RFC6570](https://tools.ietf.org/html/rfc6570)\n- setup - Optional `Object`\n - sandbox - Optional `Object`\n - Creates a frozen global context, used for sandboxed setup functions\n - Only applies when using a string setup function\n - timeout - Optional `Number`\n - Only applies when using a string setup function\n - compiler - Optional `Function`\n - Only applies when using a string setup function\n- onError - Optional `Function`\n - Receives a context object when an error occurs, so you can decide how to handle the error and opt out of the default behavior.\n - The default handler will emit an error on the stream.\n- onFetch - Optional `Function`\n - Receives the URL as the only argument, for debugging or logging purposes.\n - Called every time an HTTP request is created.\n\n### parse(format[, options])\n\nReturns a function that creates a parser stream. Parser streams receive text as input, and output objects.\n\n#### format\n\nBuilt in parsers are:\n\n- csv\n - Optional `autoFormat` option, to automatically infer types of values and convert them.\n - Optional `camelcase` option, to camelcase and normalize header keys.\n - Optional `zip` option, if the content is a zip file it will parse each CSV file in the zip.\n- tsv\n - Optional `autoFormat` option, to automatically infer types of values and convert them.\n - Optional `camelcase` option, to camelcase and normalize header keys.\n - Optional `zip` option, if the content is a zip file it will parse each TSV file in the zip.\n- excel\n - Optional `autoFormat` option, to automatically infer types of values and convert them.\n - Optional `camelcase` option, to camelcase and normalize header keys.\n - Optional `zip` option, if the content is a zip file it will parse each XLSX file in the zip.\n- json\n - Requires a `selector` option that specifies where to grab rows in the data.\n - If needed, you may provide multiple selectors as an array (`selector: [ 'a.*', 'b.*' ]`)\n - Optional `zip` option, if the content is a zip file it will parse each JSON file in the zip.\n- xml\n - Requires a `selector` option that specifies where to grab rows in the data.\n - Note that the selector applies to the [xml2js](https://github.com/Leonidas-from-XIV/node-xml2js) output.\n - Optional `autoFormat` option, to automatically infer types of values and convert them.\n - Optional `camelcase` option, to camelcase and normalize header keys.\n - Optional `zip` option, if the content is a zip file it will parse each XML file in the zip.\n- ndjson\n- shp\n- kml\n- kmz\n- gdb\n- gpx\n- gtfs\n- gtfsrt\n\n### options\n\n- Optional `autoFormat` option, to automatically infer types of values and convert them.\n - If `simple` it will only infer types from values and trim keys\n - If `aggressive` it will add camelcasing of keys on top of simple mode\n - If `extreme` it will add more complex mapping on top of aggressive mode\n - For example, converting `lat` and `lon` fields to a GeoJSON Point\n\n### transform(transformer[, options])\n\n#### transformer(row, meta)\n\n- Asynchronous function, receives the current row and the meta information object.\n - Meta information object contains: `row`, `url`, `accessToken`, `context`, `source`, and `header` (if using a JSON parser)\n- If transformer is a string, it will compile it and sandbox it using [vm2](https://github.com/patriksimek/vm2).\n- If transformer is an object, it will use [object-transform-stack](https://github.com/staeco/object-transform-stack) to map objects.\n- Returning an object will pass it on, and null or undefined will remove the item from the stream (skip).\n\n#### options\n\n- concurrency - Optional `Number`, defaults to 8\n- onBegin(row, meta) - Optional `Function`\n- onError(err, row, meta) - Optional `Function`\n- onSkip(row, meta) - Optional `Function`\n- onSuccess(row, meta) - Optional `Function`\n\nThe following are also available if the transformer is compiled code:\n\n- timeout - Optional `Number`\n- compiler - Optional `Function`\n - If you are using babel, make sure you add `[ 'core-js', 'core-js/*' ]` to the `externalModules` option.\n- pooling - Optional `Boolean`\n - When true, runs a pool of worker threads for your transform functions. This is incompatible with the `compiler` and `sandbox` options, due to issues transferring complex functions between threads.\n- externalModules - Optional `Array