\n\nWhat is Pelias?\n \nPelias is a search engine for places worldwide, powered by open data. It turns addresses and place names into geographic coordinates, and turns geographic coordinates into places and addresses. With Pelias, you’re able to turn your users’ place searches into actionable geodata and transform your geodata into real places.\n
\nWe think open data, open source, and open strategy win over proprietary solutions at any part of the stack and we want to ensure the services we offer are in line with that vision. We believe that an open geocoder improves over the long-term only if the community can incorporate truly representative local knowledge.\n\n\n# Pelias OpenStreetMap importer\n\n## Overview\n\nThe OpenStreetMap importer handles importing data from [OpenStreetMap](https://www.openstreetmap.org/) into Elasticsearch for use by Pelias.\n\nIt includes logic for filtering to select only data relevant for geocoding, transforming it to match the Pelias data model, and augmenting the data as required.\n\n## Prerequisites\n\nSee [Pelias software requirements](https://github.com/pelias/documentation/blob/master/requirements.md)\n\n## Clone and Install dependencies\n\n> For instructions on setting up Pelias as a whole, see our [getting started guide](https://github.com/pelias/documentation/blob/master/getting_started_install.md). Further instructions here pertain to the OSM importer only\n\n```bash\n$ git clone https://github.com/pelias/openstreetmap.git && cd openstreetmap;\n$ npm install\n```\n\n## Download data\n\nThe importer will accept any valid `pbf` extract you have, such as a full planet file (50GB+) from [planet.openstreetmap.org](https://planet.openstreetmap.org) or [download.geofabrik.de](https://download.geofabrik.de)\nYou can use the included download script to obtain the desired `pbf` files as follows. In the configuration file you can\nspecify which files are to be downloaded. They will all be downloaded to the `imports.openstreetmap.datapath` directory.\n\nIf no download sources are specified in the configuration, the entire planet file will be downloaded. Keep in mind this file is quite large.\n\n```bash\n$ PELIAS_CONFIG= npm run download\n```\n\n## Configuration\n\nIn order to tell the importer the location of your downloads, temp space and environmental settings you will first need to create a `~/pelias.json` file.\n\nSee [the config](https://github.com/pelias/config) documentation for details on the structure of this file. Your relevant config info for the openstreetmap module might look something like this:\n\n```javascript\n{\n \"imports\": {\n \"openstreetmap\": {\n \"download\": [{\n \"sourceURL\": \"https://s3.amazonaws.com/metro-extracts.nextzen.org/portland_oregon.osm.pbf\"\n }],\n \"datapath\": \"/mnt/pelias/openstreetmap\",\n \"leveldbpath\": \"/tmp\",\n \"import\": [{\n \"filename\": \"portland_oregon.osm.pbf\"\n }]\n }\n }\n}\n```\n\n### Configuration Settings\n\n#### `imports.openstreetmap.datapath`\n\nThis is the directory where the OSM importer will look for files to import. If configured it will also download files to this location.\n\n#### `imports.openstreetmap.download[0].sourceURL`\n\nA URL to download when the download script (in `./bin/download`) is run. Will be downloaded to the `datapath` directory.\n\n#### `imports.openstreetmap.import[0].filename`\n\nThe OSM importer will look for a file with a name matching this value in the configured `datapath` directory when importing data.\n\nIf downloading from a remote URL, the filename must match the value in `sourceURL`.\n\nA file ending in `.ndjson`, `.ldjson` or `.jsonl`, optionally `.gz` compressed,\nis read as newline delimited JSON instead of being parsed as a pbf. It is\nassumed to be the stored output of a previous `pbf2json` run over the same\nextract, and importing it produces exactly the same documents.\n\n```javascript\n{\n \"imports\": {\n \"openstreetmap\": {\n \"datapath\": \"/data/openstreetmap\",\n \"import\": [{\n \"filename\": \"pbf2json.pelias.jsonl.gz\"\n }]\n }\n }\n}\n```\n\nThe two forms can be mixed in one `import` array, and both work with\n`parallelism`. Compressed files are decompressed as they are read.\n\n#### `imports.openstreetmap.layers`\n\nThis is an object you can use to define your own layers based on OSM tags. For example, if you wanted a layer `coffee` for just coffee shops you could set:\n\n```\n{\n \"imports\": {\n \"openstreetmap\": {\n \"layers\": {\n \"coffee\": {\n \"tags\": [ \"amenity~cafe+name\" ]\n }\n }\n }\n }\n}\n```\n\n#### `imports.openstreetmap.leveldbpath`\n\nThis is the directory where temporary files will be stored in order to\ndenormalize OSM ways and relations. In the case of a planet import it is best\nto have at least 100GB free.\n\nDefaults to `tmp`.\n\n#### `imports.openstreetmap.importVenues`\n\nBy default, the OSM importer imports both venue records and addresses. If set to false, only address records will be imported.\n\nYou can also disable venue import by setting `imports.openstreetmap.layers.venue` to an empty value.\n\n#### `imports.openstreetmap.removeDisusedVenues`\n\nIf set to boolean `true`, `venue`s with popularity below `0` (as determined by OSM tags) will be\ndiscarded. In practice, this affects records with tags such as\n[`disused`](https://wiki.openstreetmap.org/wiki/Key:disused#disused:_namespace),\n[`amenity:disused`](https://wiki.openstreetmap.org/wiki/Tag:amenity%3Ddisused) and\n[`abandoned`](https://wiki.openstreetmap.org/wiki/Key:abandoned:)\n\nBy default, or if set to any other value besides `true`, these records will be imported.\n\n#### `imports.openstreetmap.parallelism`\n\nThe number of import pipelines to run in parallel. Defaults to `1`, which runs\nthe importer as a single process.\n\nReading a PBF file cannot be parallelized, but everything after the parser can\nbe. When set above `1`, a single `pbf2json` reader is started and its output is\ndistributed across that many worker processes, each running the full import\npipeline.\n\n### Administrative Hierarchy Lookup\n\nOSM records often do not contain information about which city, state (or\nother region like province), or country that they belong to. Pelias has the\nability to compute these values from [Who's on First](http://www.whosonfirst.org/) data.\nFor more info on how admin lookup works, see the documentation for\n[pelias/wof-admin-lookup](https://github.com/pelias/wof-admin-lookup). By default,\nadminLookup is enabled. To disable, set `imports.adminLookup.enabled` to `false` in Pelias config.\n\n**Note:** Admin lookup requires loading around 5GB of data into memory.\n\n## Running an import\n\nThis will start the import process. It may take a few minutes to load administrative data and begin processing the OSM PBF file, then you should see regular progress updates in the terminal.\n\n```bash\n$ npm start\n```\n\n## How long does it take?\n\nIf all goes well, you should see between 6000-7000 records imported per second on a modern machine. A full planet install will import about 80 million records, whereas most city extracts will import at most a few thousand.\n\nThese counts are of records containing valid location names to search on, data which is not directly searchable by the end user, such as fire hydrants, lamp posts, etc is not imported.\n\nIf you are looking to run a planet-wide cluster like the one we provide for [geocode.earth](https://geocode.earth) please see our documentation on [full planet builds](https://github.com/pelias/documentation/blob/master/full_planet_considerations.md#importer-machine).\n\n## Issues\n\nIf you have any issues getting set up or the documentation is missing something, please open an issue here: https://github.com/pelias/openstreetmap/issues\n\n## Contributing\n\nPlease fork and pull request against upstream master on a feature branch.\n\nPretty please; provide unit tests and script fixtures in the `test` directory.\n\n## Code Linting\n\nA `.jshintrc` file is provided which contains a linting config, usually your text editor will understand this config and give you inline hints on code style and readability.\n\nThese settings are strictly enforced when you do a `git commit`, you can execute `git commit` at any time to run the linter against your code.\n\n### Running Unit Tests\n\n```bash\n$ npm test\n```\n\n### Running End-to-End Tests\n\nThese tests run the entire pipeline against a small PBF extract to assert that the individual units work as expected when wired together.\n\n```bash\n$ npm run end-to-end\n```\n\n## Code Coverage\n\n```bash\n$ npm run coverage\n```\n\n### Continuous Integration\n\nCI tests every change against all supported Node.js versions.\n","readmeFilename":"README.md"}
![](\"https://raw.githubusercontent.com/pelias/design/master/logo/pelias_github/Github_markdown_hero.png\"){kind=logo}
\n![](\"https://img.shields.io/github/license/pelias/api?style=flat&color=orange\")
![](\"https://img.shields.io/docker/pulls/pelias/api?style=flat&color=informational\")
![](\"https://img.shields.io/gitter/room/pelias/pelias?style=flat&color=yellow\")