{"_id":"@archivator/archivable","_rev":"1-d580ef33028eff3e69979c841be49252","name":"@archivator/archivable","dist-tags":{"latest":"0.2.0"},"versions":{"0.2.0":{"name":"@archivator/archivable","version":"0.2.0","description":"Archivable Data Transfer Object and NormalizedAsset Entities for archiving web pages and their assets","keywords":["archiving","normalization"],"homepage":"https://github.com/renoirb/archivator/blob/v3.x-dev/packages/archivable","repository":{"type":"git","url":"git+https://github.com/renoirb/archivator.git","directory":"packages/archivable"},"license":"MIT","author":{"name":"Renoir Boulanger","email":"contribs@renoirboulanger.com"},"type":"module","exports":{"import":"./dist/index.mjs","require":"./main.cjs"},"main":"main.cjs","module":"dist/index.mjs","types":"dist/index.d.ts","scripts":{"api":"use-run-all api-extractor api-documenter fix","api-documenter":"api-documenter markdown --input temp --output docs","api-extractor":"api-extractor run -c config/api-extractor.json --verbose --local","build":"use-cross-env use-run-all clean build:* api","build:node-esm":"use-cross-env BROWSERSLIST='maintained node versions' use-bili -c bili.config.ts --format esm --file-name index.mjs","clean":"use-cross-env use-rimraf .rpt2_cache dist *.*.log","fix":"use-cross-env conventions-code-formatter prettier '**/*.{ts,json,md}' --write","lint":"use-cross-env use-eslint --fix 'src/**/*.ts'","prepublishOnly":"rushx build","sort-package-json":"use-cross-env conventions-code-formatter sort-package-json","test":"use-cross-env use-jest --detectOpenHandles","test:snapshots":"use-cross-env use-jest --detectOpenHandles -u"},"dependencies":{"esm":"^3.2.0"},"devDependencies":{"@babel/core":"^7.10.0","@babel/plugin-transform-runtime":"^7.10.0","@babel/preset-env":"^7.10.0","@babel/preset-typescript":"^7.10.0","@babel/runtime-corejs3":"^7.10.0","@microsoft/api-documenter":"^7.8.17","@microsoft/api-extractor":"^7.8.15","@microsoft/node-library-build":"^6.4.0","@renoirb/conventions-code-formatter":"^1.3.0","@renoirb/conventions-use-bili":"^1.3.0","@renoirb/conventions-use-eslint":"^1.2.1","@renoirb/conventions-use-jest":"1.1.1","@renoirb/conventions-use-prettier":"^1.3.0","@renoirb/conventions-use-typescript-3":"^1.2.1","@renoirb/tools-bundling-helpers":"^1.2.1","@rushstack/node-core-library":"^3.25.0","@types/jest":"^26.0.0","@types/node":"^14.14.11","core-js":"^3.8.0","jest":"^25.5.0","ts-jest":"^25.5.0","tslib":"^2.0.0","typescript":"^3.9.0","url-dirname-normalizer":"^1.4.0"},"peerDependencies":{"url-dirname-normalizer":"workspace:^1.4.0"},"engines":{"node":">=14"},"publishConfig":{"access":"public"},"bugs":{"url":"https://github.com/renoirb/archivator/issues"},"_id":"@archivator/archivable@0.2.0","_nodeVersion":"14.15.1","_npmVersion":"6.14.9","dist":{"integrity":"sha512-ITDGdABcYoqccS5iCaQfuKM7SF8TkE1J5h6mAHoZybY8JoVxvJ0RyS7wm8mPZuLjmrITa2B3iXr8a4wpLl1duw==","shasum":"864dbdd39a0de80723dfe555e44de37c874a473d","tarball":"https://registry.npmjs.org/@archivator/archivable/-/archivable-0.2.0.tgz","fileCount":74,"unpackedSize":372784,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf2pmwCRA9TVsSAnZWagAAmfAQAIHbl4gGQFmnsIJVsPJA\niXUQ7jw5hcmzqxZBgd+0N7R9j3pGdqmuqO4Rm4z1cu04ICW2SOW8lCSF7CNE\nxOh3Bib1Ivfm7m0BHlTucMxnmctKJe36tGzr8w22QSxwD4+Bt9EpVZ9jzVv0\nzrLntNoWrxPz2szOG0BTb8k5L23QEDP3KJqlKpJmGHNUx4ILNfKc9rT75Spr\nOLYsUvp1T8FHgWwZxw25bhXWShPwLXLR+yx6adnGa9g9kBLjxzAhJRgXLFMJ\nu0oYDJU72jLNaPiiwlIx4kWs8wHefXKUtXVzU146z/fA4Zxp+ANejTkyaAr2\nVbvCWPPvDUynCvXl5xxVZSKgVODUoEb9+fL+fXi2EjZSDihJDm1YmmM+AESb\nB4JfrpCuBTjIFXg81Zy91X8dy7QR7uyWVsb/PzREszVK7Upde3IyCXLq5/Cu\nROqhD0O6fgz+zZuLKHnIRHJYsEVo+XyYV7JC8FIMwWj+uliP2Mo8LjAaOotA\n4bVTqRO7Kkv2aG1p7FJD4LWLCMnxN+XQ2mNMy7jArBmH/txoDa0GiVG/dMD8\nhW8br/q2y67BaPSBaSyWzDncwv+aU+Pj+thLOLDGu7MLSJuxuTK+9eshYH5j\n+zx1kqVFgt29vTEHY2MWHAjGExX1SLL2eD2uJm9yBEubCCrVa7nkQEfnVgk/\neAnb\r\n=cXLr\r\n-----END PGP SIGNATURE-----\r\n","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIA3h7ZT06eLvJFEcN63YpLGj+SfoyrzO7fT/cIH9/QCpAiB6uPXg1ACUgTXSI1/wLMDwQO4EkMvN7hSC2F+XhogCrw=="}]},"_npmUser":{"name":"renoirb","email":"hello@renoirboulanger.com"},"directories":{},"maintainers":[{"name":"renoirb","email":"hello@renoirboulanger.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/archivable_0.2.0_1608161712050_0.042753468515327686"},"_hasShrinkwrap":false}},"time":{"created":"2020-12-16T23:35:11.824Z","0.2.0":"2020-12-16T23:35:12.198Z","modified":"2022-04-04T15:18:34.617Z"},"maintainers":[{"name":"renoirb","email":"hello@renoirboulanger.com"}],"description":"Archivable Data Transfer Object and NormalizedAsset Entities for archiving web pages and their assets","homepage":"https://github.com/renoirb/archivator/blob/v3.x-dev/packages/archivable","keywords":["archiving","normalization"],"repository":{"type":"git","url":"git+https://github.com/renoirb/archivator.git","directory":"packages/archivable"},"author":{"name":"Renoir Boulanger","email":"contribs@renoirboulanger.com"},"bugs":{"url":"https://github.com/renoirb/archivator/issues"},"license":"MIT","readme":"# [@archivator/archivable][repo-url]\n\n> Archivable Data Transfer Object and NormalizedAsset Entities for archiving web\n> pages and their assets\n\nA _Data Transfer Object_ (or _Entity_ object) and related utilities to\nmanipulate Web Page Metadata while archiving.\n\n[repo-url]:\n  https://github.com/renoirb/archivator/blob/v3.x-dev/packages/archivable\n  'Archivable Data Transfer Object'\n[npmjs-package-badge]:\n  https://img.shields.io/npm/v/%40archivator%2Farchivable?style=flat-square&logo=appveyor&label=npm&logo=npm\n[npmjs-package]: https://www.npmjs.com/package/%40archivator%2Farchivable\n[bundlesize-badge]:\n  https://img.shields.io/bundlephobia/min/%40archivator%2Farchivable?style=flat-square\n[dependabot-badge]:\n  https://img.shields.io/librariesio/release/npm/%40archivator%2Farchivable?style=flat-square&logo=appveyor&logo=dependabot\n\n| Version                                      | Size                                 | Dependencies                                                           |\n| -------------------------------------------- | ------------------------------------ | ---------------------------------------------------------------------- |\n| [![npm][npmjs-package-badge]][npmjs-package] | ![npm bundle size][bundlesize-badge] | ![Libraries.io dependency status for latest release][dependabot-badge] |\n\n## Usage\n\n_See also_:\n\n- [API Extractor code-review signature](../../common/reviews/api/archivable.api.md)\n- [API Documentor generated docs](./docs/index.md)\n\n### Archivable\n\n```js\nimport Archivable from '@archivator/archivable'\n\n// HTML Source document URL from where the asset is embedded\n// Ignore document origin if resource has full URL, protocol relative, non TLS\nconst sourceDocument =\n  'http://example.org/@ausername/some-lengthy-string-ending-with-a-hash-1a2d8a61510'\n\nconst selector = '#main'\nconst truncate = '.ad,.sponsor'\n\n/** @type {import('@archivator/archivable').ArchivableType} */\nconst dto = new Archivable(sourceDocument, selector, truncate)\n// ... Do things with `dto`\n```\n\nWhen using CSV format\n\n```js\nimport Archivable from '@archivator/archivable'\n\n// The following lines would be from a text file where we have one item per line\n// Each item MUST have two semi-columns\nconst lines = [\n  'http://example.org/@ausername/some-lengthy-string-ending-with-a-hash-1a2d8a61510;#main;.ad,.sponsor',\n]\n\nfor (const line of lines) {\n  /** @type {import('@archivator/archivable').ArchivableType} */\n  const dto = Archivable.fromLine(line)\n  // ... Do things with `dto`\n}\n```\n\n### DocumentAssets and NormalizedAsset\n\nWhile archiving a web page, we might have a list of all assets the document\nmakes references to. They can be embedded inside `<img src=\"...\">` tags and\nother similar schemes.\n\nEach \"NormalizedAsset\" is an entity from which we can figure out where an asset\ncan be downloaded in relation to the current source document URL, like web\nbrowsers do.\n\nNormalizedAsset contains:\n\n- `match`: is the initial value passed in, that can be useful if we want to\n  rewrite the source document\n- `reference`: is the normalized hash for the asset, we could use that value to\n  replace the source document's HTML with a local name\n- `dest`: would be where we would archive the asset, it is basically\n  `directoryNameNormalizer(sourceDocument) + reference`\n- `src`: is where we should attempt downloading the asset from\n\n````ts\nimport { DocumentAssets, NormalizedAsset } from '@archivator/archivable'\n\n// HTML Source document URL from where the asset is embedded\n// Notice the source might not be the same as where images are stored\nconst sourceDocument = 'http://www1.example.net/articles/1'\n\n// Image tag src attribute value, e.g. `<img src=\"//example.org/a/b.png\" />`\n// Notice we used protocol relative URL\n// (i.e. not specify https, meaning we'll use from source document)\nconst assetUrl = '//www.example.org/a/b/c.png'\n\n/**\n * `normalized` is an instance of `NormalizedAsset`, and should look like this\n *\n * ```json\n * {\n *   \"dest\": null,\n *   \"match\": \"//www.example.org/a/b/c.png\",\n *   \"reference\": null,\n *   \"src\": \"http://www.example.org/a/b/c.png\",\n * }\n * ```\n *\n * @type {import('@archivator/archivable').NormalizedAssetType}\n */\nconst normalized = new NormalizedAsset(sourceDocument, assetUrl)\n````\n\n### DocumentAssets\n\nWhen we have more than one asset to download, we might have a list of assets, we\ncan use `DocumentAssets` _class_.\n\nUsing it, we can iterate from it [because it implements `Iterable` the\nprotocol][exploringjs--ch_sync-generators] and treat it as if it's an array of\n`NormalizedAsset` items.\n\n[exploringjs--ch_sync-generators]:\n  https://exploringjs.com/impatient-js/ch_sync-generators.html\n  '35 Synchronous generators (advanced)'\n\n```js\nimport { DocumentAssets } from '@archivator/archivable'\n\n// HTML Source document URL from where the asset is embedded\nconst sourceDocument = 'http://renoirboulanger.com/about/projects/'\n\n// List of URLs you might find on that URL\n// e.g. `<img src=\"//example.org/a/b.png\" />`\n// Notice some URLs are relative, protocol-relative, others are going on another domain\nconst matches = [\n  // Case 1: On an almost (no protocol) fully-qualified URL, on another domain\n  '//www.example.org/a/b/c.png',\n  // Case 2: Relative URL to the current source document\n  '../../avatar.jpg',\n  // Case 3: Fully qualified URL that is local to the site\n  'http://renoirboulanger.com/wp-content/themes/twentyseventeen/assets/images/header.jpg',\n  // Case 4: Fully qualified URL that is outside\n  'https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png',\n  // Case 5: Fully qualified  URL that is outside and protocol relative\n  '//www.gravatar.com/avatar/cbf8c9036c204fe85e15155f9d70faec?s=500',\n  // Case 6: Relative URL to the domain name, starting at root\n  '/wp-content/themes/renoirb/assets/img/zce_logo.jpg',\n]\n\n/**\n * Leverage ECMAScript 2015+ Iteration prototocol.\n *\n * Pass a collection of strings, get a normalized list with iteration.\n *\n * @type {Iterable<import('@archivator/archivable').NormalizedAssetType>}\n */\nconst assets = new DocumentAssets(sourceDocument, matches)\nfor (const normalized of assets) {\n  // It is a generator function, we can iterate normalized like an array.\n  // If we were in an asychronous function, we'd be able to await each step.\n  // In this example, we're simply using the return of assetCollectionNormalizer like we would with an array.\n  console.log(normalized)\n}\n```\n\n#### Change `reference` hashing format\n\nIn the above example, the first item looks like this;\n\n```json\n{\n  \"match\": \"//www.example.org/a/b/c.png\",\n  \"src\": \"http://www.example.org/a/b/c.png\",\n  \"dest\": \"renoirboulanger.com/about/projects/4c49ccbf4cdbdbcfc7f91cf87f6e9636008e4a97.png\",\n  \"reference\": \"4c49ccbf4cdbdbcfc7f91cf87f6e9636008e4a97.png\"\n}\n```\n\nThe asset file \"`4c49ccbf4cdbdbcfc7f91cf87f6e9636008e4a97.png`\" contains the\nSHA1 hash for \"`http://www.example.org/a/b/c.png`\".\n\nNotice that the initial match was \"`//www.example.org/a/b/c.png`\" (the \"`match`\"\nattribute), but the \"`src`\" (where we will download image from) saw that the\n\"`sourceDocument`\" had `http` as protocol. If the protocol was `https`, the\n\"`src`\" (and the hash) would be different.\n\nAbout the hashing, if you'd prefer a shorter file name, or use a different\nhashing function.\n\nYou can change it by using\n`DocumentAssets.setReferenceHandler(hasherFn, normalizerFn)` method.\n\nThe arguments are:\n\n`hasherFn` : Where you can provide your own hashing function. See\n[crypto.ts](https://github.com/renoirb/archivator/blob/v3.x-dev/packages/archivable/src/crypto.ts)\nif you're OK with\n[Node.js’ **Crypto** module](https://nodejs.org/api/crypto.html#crypto_crypto_createhash_algorithm_options)\n\n`normalizerFn` : A function with signature `(file: string) => string` where you\ncan append the file extension, refer to\n[normalizer/asset.ts](https://github.com/renoirb/archivator/blob/v3.x-dev/packages/archivable/src/normalizer/asset.ts)\nat `assetFileExtensionNormalizer`.\n\n```ts\n// ... Continuing from example above\nimport {\n  HashingFunctionType,\n  createHashFunction,\n  NormalizedAssetFileExtensionExtractorType,\n} from '@archivator/archivable'\n\n// One can set its own hash function\n// As long as the returned createHashFunction is of type `(msg: string) => string`\nconst hashingHandler = createHashFunction('md5', 'hex')\n\n/**\n * In the example below, in every case, the file extension would ALWAYS be \".foo\".\n * We could eventually use the file's mime-type, or the source's response headers. #TODO\n */\nconst extensionHandler: NormalizedAssetFileExtensionExtractorType = (\n  foo: string,\n): string => `.foo`\n\ncollection.setReferenceHandler(\n  assetReferenceHandlerFactory(hashingHandler, extensionHandler),\n)\n```\n\nWith the above configuration in place, for the item\n\"`//www.example.org/a/b/c.png`\", we'd have the md5 hash as\n`6a324cd1a0e4e480c4db3e0558360527` with `.foo`\n\nWhich would then look like this;\n\n```json\n[\n  {\n    \"dest\": \"renoirboulanger.com/page/3/6a324cd1a0e4e480c4db3e0558360527.foo\",\n    \"match\": \"//www.example.org/a/b/c.png\",\n    \"reference\": \"6a324cd1a0e4e480c4db3e0558360527.foo\",\n    \"src\": \"http://www.example.org/a/b/c.png\"\n  }\n]\n```\n","readmeFilename":"README.md"}