wtf_wikipedia
\n ![](\"https://user-images.githubusercontent.com/399657/68222691-6597f180-ffb9-11e9-8a32-a7f38aa8bded.png\"/)
\n parse data from wikipedia
\n npm install wtf_wikipedia\n \n by\n Spencer Kelly and\n \n many contributors\n \n \n
\n ![](\"https://user-images.githubusercontent.com/399657/68221824-09809d80-ffb8-11e9-9ef0-6ed3574b0ce8.png\"/)
\n\n npm install wtf_wikipedia\n![](\"https://user-images.githubusercontent.com/399657/68221862-17ceb980-ffb8-11e9-87d4-7b30b6488f16.png\"/)
\n\n\n\n\n![](\"https://user-images.githubusercontent.com/399657/43598341-75ca8f94-9652-11e8-9b91-cabae4fb1dce.png\"/)
\n\n\n we put our information where we can't take it out.\n\n\n\n```js\nimport wtf from 'wtf_wikipedia'\n\nlet doc = await wtf.fetch('Toronto Raptors')\nlet coach = doc.infobox().get('coach')\ncoach.text() //'Darko Rajaković'\n```\n\n\n\n\n## .text()\n\nget clean plaintext:\n\n```js\nlet str = `[[Greater_Boston|Boston]]'s [[Fenway_Park|baseball field]] has a {{convert|37|ft}} wall. Field of our Fathers: By Richard Johnson`\nwtf(str).text()\n// \"Boston's baseball field has a 37ft wall.\"\n```\n\n```js\nlet doc = await wtf.fetch('Glastonbury', 'en')\ndoc.sentences()[0].text()\n// 'Glastonbury is a town and civil parish in Somerset, England, situated at a dry point ...'\n```\n\n![](\"https://user-images.githubusercontent.com/399657/68221837-0d142480-ffb8-11e9-9d30-90669f1b897c.png\"/)
\n\n\n\nrun it on the client-side:\n\n```html\n\n\n```\n\nor the server-side:\n\n```js\nimport wtf from 'wtf_wikipedia'\n// or,\nconst wtf = require('wtf_wikipedia')\n```\n\n\n\n\n\n![](\"https://user-images.githubusercontent.com/399657/40262198-a268b95a-5ad3-11e8-86ef-29c2347eec81.gif\"/)
\n\n\n\n\n\n#### How about html scraping..?\n\nWikimedia's [official parser](https://www.mediawiki.org/wiki/Parsoid) turns wikitext ➔ HTML.\n\n\n\nif you prefer this **_screen-scraping_** workflow, you can pluck at parts of a page [like that](https://observablehq.com/@mbostock/working-with-wikipedia-data).\n\nthat's cool!\n\ngetting structured data this way is still a complex, weird process.\nManually _spelunking_ the html is sometimes just as tricky and error-prone as scanning the wikitext itself.\n\nThe contributors to this library have come to that conclusion, [as many others have](https://www.mediawiki.org/wiki/Alternative_parsers).\n\nThis library is gracious to the Parsoid contributors.\n\n\n\n\n\n\n## Fetch\n\nYou can grab and parse articles from _[any wiki api](https://www.mediawiki.org/wiki/API:Main_page)_.\nThis includes any language, any wiki-project, and most **3rd-party wikis**.\n\n```js\n// 3rd-party wiki\nlet doc = await wtf.fetch('https://muppet.fandom.com/wiki/Miss_Piggy')\n\n// wikipedia français\ndoc = await wtf.fetch('Tony Hawk', 'fr')\ndoc.sentence().text() // 'Tony Hplawk est un skateboarder professionnel et un acteur ...'\n\n// accept an array, or wikimedia pageIDs\nlet docs = wtf.fetch(['Whistling', 2983], { follow_redirects: false })\n\n// article from german wikivoyage\nwtf.fetch('Toronto', { lang: 'de', wiki: 'wikivoyage' }).then((doc) => {\n console.log(doc.sentences()[0].text()) // 'Toronto ist die Hauptstadt der Provinz Ontario'\n})\n```\n\nyou may also pass the wikipedia page id as parameter instead of the page title:\n\n```javascript\nlet doc = await wtf.fetch(64646, 'de')\n```\n\nthe fetch method follows redirects.\n\n### API plugin\n\n**wtf.getCategoryPages(title, [options])**\n\nretrieves all pages and sub-categories belonging to a given category:\n\n```js\nwtf.extend(require('wtf-plugin-api'))\nlet result = await wtf.getCategoryPages('Category:Politicians_from_Paris')\n/*\n{\n [\n {\"pageid\":52502362,\"ns\":0,\"title\":\"William Abitbol\"},\n {\"pageid\":50101413,\"ns\":0,\"title\":\"Marie-Joseph Charles des Acres de L'Aigle\"}\n ...\n {\"pageid\":62721979,\"ns\":14,\"title\":\"Category:Councillors of Paris\"},\n {\"pageid\":856891,\"ns\":14,\"title\":\"Category:Mayors of Paris\"}\n ]\n}\n*/\n```\n\n**wtf.random([options])**\n\nfetches a random wikipedia article, from a given language or domain\n\n```js\nwtf.extend(require('wtf-plugin-api'))\nwtf.random().then((doc) => {\n console.log(doc.title(), doc.categories())\n //'Whistling' ['Oral communication', 'Vocal skills']\n})\n```\n\nsee [wtf-plugin-api](./plugins/api)\n\n\n\n\n## Tutorials\n\n- [Gentle Introduction](https://observablehq.com/@spencermountain/wtf_wikipedia-tutorial?collection=@spencermountain/wtf_wikipedia) - Getting NBA Team data\n- [Parsing tables](https://observablehq.com/@spencermountain/parsing-wikipedia-tables) - getting all Apollo Astronauts as JSON\n- [Parsing Timezones](https://observablehq.com/@spencermountain/parsing-timezones-from-wikipedia)\n- [MBL season schedules](https://observablehq.com/@spencermountain/wikipedia-baseball-table-parser?collection=@spencermountain/wtf_wikipedia)\n- [Fetching a list of pages](https://observablehq.com/@spencermountain/parsing-a-list-of-wikipedia-articles)\n- [Parsing COVID outbreak table](https://observablehq.com/@spencermountain/parsing-wikipedias-coronavirus-outbreak-data?collection=@spencermountain/wtf_wikipedia)\n\n\n\n\n\n\n\n## Full API\n\n- **.title()** - get/set the title of the page from the first-sentence\n- **.pageID()** - get/set the wikimedia id of the page, if we have it.\n- **.wikidata()** - get/set the wikidata id of the page, if we have it.\n- **.domain()** - get/set the domain of the wiki we're on, if we have it.\n- **.url()** - (try to) generate the url for the current article\n- **.lang()** - get/set the current language (used for url method)\n- **.namespace()** - get/set the wikimedia namespace of the page, if we have it\n- **.isRedirect()** - if the page is just a redirect to another page\n- **.redirectTo()** - the page this redirects to\n- **.isDisambiguation()** - is this a placeholder page to direct you to one-of-many possible pages\n- **.isStub()** - if the page is flagged as incomplete\n- **.categories()** - return all categories of the document\n- **.sections()** - return a list of the Document's sections\n- **.paragraphs()** - return a list of Paragraphs, in all sections\n- **.sentences()** - return a list of all sentences in the document\n- **.images()** - return all images found in the document\n- **.links()** - return a list of all links, in all parts of the document\n- **.lists()** - sections in a page where each line begins with a bullet point\n- **.tables()** - return a list of all structured tables in the document\n- **.templates()** - any type of structured-data elements, typically wrapped in like {{this}}\n- **.infoboxes()** - specific type of template, that appear on the top-right of the page\n- **.references()** - return a list of 'citations' in the document\n- **.coordinates()** - geo-locations that appear on the page\n- **.text()** - plaintext, human-readable output for the page\n- **.json()** - a 'stringifyable' output of the page's main data\n- **.wikitext()** - original wiki markup\n- **.description()** - get/set the page's short description, if we have one.\n- **.pageImage()** - get/set the page's representative image, if we have one.\n- **.revisionID()** - get/set the latest edit id of the page, if we have it.\n- **.timestamp()** - get/set the time of the most recent edit of the page, if we have it.\n\n### Section\n\n- **.title()** - the name of the section, between ==these tags==\n- **.index()** - which number section is this, in the whole document.\n- **.indentation()** - how many steps deep into the table of contents it is\n- **.sentences()** - return a list of sentences in this section\n- **.paragraphs()** - return a list of paragraphs in this section\n- **.links()** - list of all links, in all paragraphs and templates\n- **.tables()** - list of all html tables\n- **.templates()** - list of all templates in this section\n- **.infoboxes()** - list of all infoboxes found in this section\n- **.coordinates()** - list of all coordinate templates found in this section\n- **.lists()** - list of all lists in this section\n- **.interwiki()** - any links to other language wikis\n- **.images()** - return a list of any images in this section\n- **.references()** - return a list of 'citations' in this section\n- **.remove()** - remove the current section from the document\n- **.nextSibling()** - a section following this one, under the current parent: eg. 1920s → 1930s\n- **.lastSibling()** - a section before this one, under the current parent: eg. 1930s → 1920s\n- **.children()** - any sections more specific than this one: eg. History → [PreHistory, 1920s, 1930s]\n- **.parent()** - the section, broader than this one: eg. 1920s → History\n- **.text()** - readable plaintext for this section\n- **.json()** - return all section data\n- **.wikitext()** - original wiki markup\n\n### Paragraph\n\n- **.sentences()** - return a list of sentence objects in this paragraph\n- **.references()** - any citations, or references in all sentences\n- **.lists()** - any lists found in this paragraph\n- **.images()** - any images found in this paragraph\n- **.links()** - list of all links in all sentences\n- **.interwiki()** - any links to other language wikis\n- **.text()** - generate readable plaintext for this paragraph\n- **.json()** - generate some generic data for this paragraph in JSON format\n- **.wikitext()** - original wiki markup\n\n### Sentence\n\n- **.links()** - list of all links\n- **.bolds()** - list of all bold texts\n- **.italics()** - list of all italic formatted text\n- **.text()** - generate readable plaintext\n- **.json()** - return all sentence data\n- **.wikitext()** - original wiki markup\n\n### Image\n\n- **.url()** - return url to full size image\n- **.thumbnail()** - return url to thumbnail (pass `size` to customize)\n- **.links()** - any links from the caption (if present)\n- **.format()** - get file format (e.g. `jpg`)\n- **.text()** - does nothing\n- **.json()** - return some generic metadata for this image\n- **.wikitext()** - original wiki markup\n\n### Template\n\n- **.text()** - does this template generate any readable plaintext?\n- **.json()** - get all the data for this template\n- **.wikitext()** - original wiki markup\n\n### Infobox\n\n- **.links()** - any internal or external links in this infobox\n- **.keyValue()** - generate simple key:value strings from this infobox\n- **.image()** - grab the main image from this infobox\n- **.get()** - lookup properties from their key\n- **.template()** - which infobox, eg 'Infobox Person'\n- **.text()** - generate readable plaintext for this infobox\n- **.json()** - generate some generic 'stringifyable' data for this infobox\n- **.wikitext()** - original wiki markup\n\n### List\n\n- **.lines()** - get an array of each member of the list\n- **.links()** - get all links mentioned in this list\n- **.text()** - generate readable plaintext for this list\n- **.json()** - generate some generic easily-parsable data for this list\n- **.wikitext()** - original wiki markup\n\n### Reference\n\n- **.title()** - generate human-facing text for this reference\n- **.links()** - get any links mentioned in this reference\n- **.text()** - returns nothing\n- **.json()** - generate some generic metadata data for this reference\n- **.wikitext()** - original wiki markup\n\n### Table\n\n- **.links()** - get any links mentioned in this table\n- **.keyValue()** - generate a simple list of key:value objects for this table\n- **.text()** - returns nothing\n- **.json()** - generate some useful metadata data for this table\n- **.wikitext()** - original wiki markup\n\n\n\n\n\n## Builds:\n\nthis library ships separate client-side and server-side builds, to preserve filesize.\n\n- _[./wtf_wikipedia-client.mjs](./builds/wtf_wikipedia-client.mjs)_ - as es-module (or Deno)\n- _[./wtf_wikipedia-client.min.js](./builds/wtf_wikipedia-client.min.js)_ - for production\n\n- _[./wtf_wikipedia.cjs](./builds/wtf_wikipedia.cjs)_ - node commonjs build\n- _[./wtf_wikipedia.mjs](./builds/wtf_wikipedia.mjs)_ - node/deno/typescript esm build\n\nthe browser version uses `fetch()` and the server version uses `require('https')`.\n\n\n\n\n## Performance:\n\nIt is not the fastest parser, and is very unlikely to beat a [single-pass parser](https://www.mediawiki.org/wiki/Alternative_parsers) in C or Java.\n\nUsing [dumpster-dive](https://github.com/spencermountain/dumpster-dive), this library can parse a full english wikipedia in around 4 hours on a macbook.\n\nThat's about 100 pages/second, per thread.\n\n\n\n\n![](\"https://img.shields.io/npm/v/wtf_wikipedia.svg?style=flat-square\")
\n ![](\"https://img.shields.io/bundlephobia/min/wtf_wikipedia\")
\n