{"_id":"@demerstech/metadata-finder","_rev":"1-a56338c658df18b67285ecd0f2188a33","name":"@demerstech/metadata-finder","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@demerstech/metadata-finder","version":"1.0.0","description":"Request an http(s) url and scrape its metadata.","main":"index.js","scripts":{"lint":"standard","test":"npm run lint"},"repository":{"type":"git","url":"git+https://github.com/patrickdemers6/metadata-finder.git"},"author":{"name":"Patrick Demers","email":"patrickdemers6@gmail.com","url":"https://github.com/patrickdemers6"},"license":"MIT","bugs":{"url":"https://github.com/LevelNewsOrg/url-metadata/issues"},"homepage":"https://github.com/patrickdemers6/metadata-finder#readme","keywords":["http","https","url","metadata","metatags","meta","tags","article","scrape","scraper","node","node.js","javascript","js","og","open graph","og tags"],"dependencies":{"cheerio":"^0.22.0","q":"^1.4.1","request":"^2.88.0","standard":"^12.0.1","underscore":"^1.8.3"},"directories":{"example":"example","lib":"lib"},"gitHead":"d7d7b7042d859f64a67df6d99e0f4f96b736fe33","_id":"@demerstech/metadata-finder@1.0.0","_nodeVersion":"12.3.1","_npmVersion":"6.9.0","dist":{"integrity":"sha512-mPVW/sn48QKiByqC4/58XDQskXK7rJUBwIKZJLHjVe5ieMdqYXM2T4qF+u6ofgTLgEm91GT2v34bu1X352aXBQ==","shasum":"79e3a33ebb8202651f91e62cec77a5a4f7f87445","tarball":"https://registry.npmjs.org/@demerstech/metadata-finder/-/metadata-finder-1.0.0.tgz","fileCount":16,"unpackedSize":23442,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJdn4tJCRA9TVsSAnZWagAAXCAQAKQIhExKuHcK9BZvPFo2\nwtzznCxkq8brw60ONK54ThmhuAGvvok4xf6jnKGXeNneMzlmUQhaMz9x7L02\n5YDw8ZYJBEBvZRapVGHoYFx3kNUs3XPV0K/yywFcxR9YqYzDIRRNY/wQ3cJp\n6OcsQs9HDdSILqDNkJyeHyzceBtYQf7sFm8w6uMl5qniHdU1P+2xc1SZjFKp\nRmpu5z4MDZ1e7Ig1scU3ZWb6si3yZFJewwf2mGHvkioN4XDtoBxcyclVlRAE\n+kOknljrCC7WJYiK6lnhtzRup/k4TuedMENpDiS2Uqm+9xapTSPtX6mbz0zF\nJ3H0YfpwlS4Lm6G7dSFDU9p0sb1n7JI3ak0rNZOK8j6vlte/D+tzAYJdyp+H\nFPW3AuzwnOTAsiKqkKKA1t8mkfbnz2h2iqY2gSBngZquWhRrkEfcx+HdLby6\nWOHy142+XJxlUtz+A7zSnk3r4jMyHEs+5MRoJzVjbr4mu6yXE3vXI7Of3r2B\n8aMNYLx8z+sVx60/N35706Br89lC6GquaHysokhxlrzgItjE/AT8tAvgLBom\n7Zn6ruOlyFn3gW0PU8OhTMgPQvSsm0Qt3pVtiKPu2wMpBwEnOsk8OT/aMMhb\n5yUvVQyIwy5B7bwqN9GOfYfxO1XEZr47VYlkKajGVW/8GvIcTZNwrw0bLCj5\nvxeW\r\n=sLOM\r\n-----END PGP SIGNATURE-----\r\n","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIFXt8l5/JV8/7hUSgnD4FQJlGq6h3wbgeZJQwU6wIfhjAiALldxtLOpZtqcQEwJ7P/bMPVqpowOZ0vP68X9mRP+BLw=="}]},"maintainers":[{"name":"patrickdemers6","email":"patrickdemers6@gmail.com"}],"_npmUser":{"name":"patrickdemers6","email":"patrickdemers6@gmail.com"},"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/metadata-finder_1.0.0_1570736968865_0.8701350012241973"},"_hasShrinkwrap":false}},"time":{"created":"2019-10-10T19:49:28.635Z","1.0.0":"2019-10-10T19:49:28.982Z","modified":"2022-04-05T03:43:34.551Z"},"maintainers":[{"name":"patrickdemers6","email":"patrickdemers6@gmail.com"}],"description":"Request an http(s) url and scrape its metadata.","homepage":"https://github.com/patrickdemers6/metadata-finder#readme","keywords":["http","https","url","metadata","metatags","meta","tags","article","scrape","scraper","node","node.js","javascript","js","og","open graph","og tags"],"repository":{"type":"git","url":"git+https://github.com/patrickdemers6/metadata-finder.git"},"author":{"name":"Patrick Demers","email":"patrickdemers6@gmail.com","url":"https://github.com/patrickdemers6"},"bugs":{"url":"https://github.com/LevelNewsOrg/url-metadata/issues"},"license":"MIT","readme":"# url-metadata\r\nRequest an http(s) url and scrape its metadata. Many of the metadata fields returned are [Open Graph Protocol (og:)](http://ogp.me/) so far.\r\n\r\nUnder the hood, this package does some post-request processing on top of the [request](https://www.npmjs.com/package/request) module.\r\n\r\nIf you want a new feature, please open an issue or pull request in [GitHub](https://github.com/LevelNewsOrg/url-metadata).\r\n\r\n\r\n## Usage\r\n\r\nTo use in an npm/ Node.js project, install from your CLI:\r\n```\r\n$ npm install url-metadata --save`\r\n```\r\n\r\nThen in your project file (from example/basic.js):\r\n```javascript\r\nconst urlMetadata = require('url-metadata')\r\nurlMetadata('http://bit.ly/2ePIrDy').then(\r\n  function (metadata) { // success handler\r\n    console.log(metadata)\r\n  },\r\n  function (error) { // failure handler\r\n    console.log(error)\r\n  })\r\n```\r\n\r\nIf you'd like to override the default options (see below), pass in a second argument:\r\n```javascript\r\nconst urlMetadata = require('urlMetadata')\r\nurlMetadata('http://bit.ly/2ePIrDy', {fromEmail: 'me@myexample.com'}).then(...)\r\n```\r\n\r\n### Options\r\nThis package's default options are the values below that you may want to override:\r\n```javascript\r\n{\r\n  // custom name for the user agent and email that will make url request:\r\n  userAgent: 'MetadataScraper',\r\n  fromEmail: 'example@example.com',\r\n\r\n  // module will follow a maximum of 10 redirects\r\n  maxRedirects: 10,\r\n\r\n  // timeout in milliseconds, default below is 10 seconds:\r\n  timeout: 10000,\r\n\r\n  // number of characters to truncate description to:\r\n  descriptionLength: 750,\r\n\r\n  // force image urls in selected tags to use https,\r\n  // valid for 'image', 'og:image' and 'og:image:secure_url' tags:\r\n  ensureSecureImageRequest: true,\r\n\r\n  // object containing key/value pairs used for `source` attribution;\r\n  // defaults to empty object, see usage details below:\r\n  sourceMap: {},\r\n\r\n  // custom function to decode special-case encodings;\r\n  // defaults to undefined:\r\n  decode: undefined,\r\n\r\n  // custom function to encode the metadata fields before they are returned;\r\n  // defaults to undefined:\r\n  encode: undefined,\r\n  \r\n  // object containing custom headers\r\n  // defaults to empty object\r\n  headers: {}\r\n}\r\n```\r\n\r\n#### Option: Source Map\r\nThis module introduces and supports a metadata field called `source`. More details about `source` in the `Returns` section below. Example usage can be found in `example/source-map.js`\r\n\r\n`sourceMap` is used to override the default `source` attribution behavior, which derives `source` from the web host that the url resolves to.\r\n\r\n`sourceMap` is a simple object containing key/value pairs where the key is a YouTube username and the value is the source you'd like to attribute the content to (such as a domain, as in example below).\r\n```javascript\r\nconst options = {\r\n  sourceMap: { 'the guardian': 'theguardian.com' }\r\n}\r\n```\r\n\r\nIf you'd like to extend this functionality beyond YouTube attribution, create an issue or pull request in [GitHub](https://github.com/LevelNewsOrg/url-metadata).\r\n\r\n#### Option: Decode\r\nYou can supply a custom function to decode the metadata scraped from the url. Example decoding of EUC-JP (Japanese) metadata can be found in `example/decode.js`.\r\n\r\nIf you pass in an options.decode() function, this module will force the [request](https://www.npmjs.com/package/request) module to return the scraped metadata as a buffer to decode(). This module is not opinionated about what you do in the decode() function, only that it accepts a buffer as its argument and returns a string.\r\n\r\n#### Option: Encode\r\nYou can supply a custom function to encode the metadata fields before they are returned from this module, see `example/encode.js`:\r\n```javascript\r\nconst options = {\r\n  encode: function (value) {\r\n    return encodeURIComponent(value).replace(/['*]/g, escape)\r\n  }\r\n}\r\n```\r\n\r\n### Returns\r\nReturns a promise that gets resolved with the following url metadata if the url request response returns successfully. Note that the `url` field returned below will be the last hop in the request chain. So if you passed in a url that was generated by a link shortener, for example, you'll get back the final destination of the link as the `url`.\r\n```javascript\r\n{\r\n  'url'                  : '',\r\n  'canonical'            : '',\r\n  'title'                : '',\r\n  'image'                : '',\r\n  'author'               : '',\r\n  'description'          : '',\r\n  'keywords'             : '',\r\n  'source'               : '', // * see note below\r\n  'og:url'               : '',\r\n  'og:locale'            : '',\r\n  'og:locale:alternate'  : '',\r\n  'og:title'             : '',\r\n  'og:type'              : '',\r\n  'og:description'       : '',\r\n  'description'          : '',\r\n  'og:determiner'        : '',\r\n  'og:site_name'         : '',\r\n  'og:image'             : '',\r\n  'og:image:secure_url'  : '',\r\n  'og:image:type'        : '',\r\n  'og:image:width'       : '',\r\n  'og:image:height'      : '',\r\n  'price':               : '',\r\n  'priceCurrency':       : '',\r\n  'availability'         : ''\r\n}\r\n```\r\n\r\nAdditional fields are also returned if the url has an `og:type` set to `article`. These fields are:\r\n```javascript\r\n{\r\n  'article:published_time'     : '',\r\n  'article:modified_time'      : '',\r\n  'article:expiration_time'    : '',\r\n  'article:author'             : '',\r\n  'article:section'            : '',\r\n  'article:tag'                : '',\r\n  'og:article:published_time'  : '',\r\n  'og:article:modified_time'   : '',\r\n  'og:article:expiration_time' : '',\r\n  'og:article:author'          : '',\r\n  'og:article:section'         : '',\r\n  'og:article:tag'             : ''\r\n}\r\n```\r\n\r\n#### The `source` field\r\nThis module introduces and supports a metadata field called `source` which may be useful for attributing content hosted elsewhere to an original source.\r\n\r\nThe default behavior of this module is to derive the `source` field from the url's host (in cases where redirects take place before the last hop, the host is the last hop in the request chain):\r\n```javascript\r\nmetadata.set({ source: url.split('://')[1].split('/')[0] })\r\n```\r\n\r\nYou may be able to override this default behavior with the `sourceMap` option.\r\n","readmeFilename":"README.md"}