\n\nWhat is Pelias?\n \nPelias is a search engine for places worldwide, powered by open data. It turns addresses and place names into geographic coordinates, and turns geographic coordinates into places and addresses. With Pelias, you’re able to turn your users’ place searches into actionable geodata and transform your geodata into real places.\n
\nWe think open data, open source, and open strategy win over proprietary solutions at any part of the stack and we want to ensure the services we offer are in line with that vision. We believe that an open geocoder improves over the long-term only if the community can incorporate truly representative local knowledge.\n\n\n# Pelias Interpolation Service\n\nAn open source + open data project to perform global street address interpolation queries.\n\n\n\n## Overview\n\nThe [Openstreetmap](http://www.openstreetmap.com) and [Openaddresses](http://www.openaddresses.io) projects provide a huge cache of street address information; between them around 500 million address points are freely available to download.\n\nSome countries like Germany and the USA have dense address coverage while other have only sparse data available.\n\nThis project aims to 'fill in the gaps' in the data by intelligently estimating where the missing house numbers would lie on the road.\n\nThe service was designed for use with the [pelias geocoder](https://github.com/pelias/pelias), however it can also be used as a stand-alone application or included with other geographic software / search engines.\n\nmore info: [[design doc](docs/design-doc.md)] [[relationship to pelias](docs/introduction.md)] [[existing standards](docs/existing-standards.md)] [[conflation](docs/conflation.md)] [[database schema](docs/database-schema.md)]\n\n## Architecture\n\nThe software is written in javascript to run in nodejs, the storage engine is sqlite3.\n\nClient libraries can be written in any language which can read sqlite3. If you wish to write a client in another language please open an issue and we can explain which functions you will need to port.\n\nThe software is split in to 6 distinct parts:\n\n- the street (polyline) importer\n- the openaddresses address importer\n- the openstreetmap address + address range importer\n- the T.I.G.E.R. block range importer\n- the geometry (vertices) interpolation\n- the client APIs (the webserver and CLI interface)\n\nThe data is split in to 2 different sqlite3 databases:\n\n- street.db (holds information about streets, geometry, their names and bounding boxes)\n- address.db (holds address point data, both rooftop accuracy and pre-interpolated vertex data)\n\n## Workflow\n\n### street database\n\nFirstly you need to build the `street.db` database.\n\nYou will need a polyline data file which contains all the streets you wish to import, you can find some pre-made extracts [here](https://github.com/pelias/polylines) and there is also information on that readme about how to generate your own extracts.\n\nSee the [building the databases](https://github.com/pelias/interpolation#building-the-databases) section below for detailed information on which commands to run.\n\nThere is also a script named `./script/import.sh` in this repository which makes running this process much easier.\n\nnote: We only support the `polyline` format, you will need to format-shift data from other formats in order to import it.\n\n### address database\n\nNext you need to build the `address.db` database.\n\nYou will need to download one or more `openaddresses` or `openstreetmap` files for the addresses you wish to import.\n\nSee the [building the databases](https://github.com/pelias/interpolation#building-the-databases) section below for detailed information on which commands to run.\n\nThere are scripts named `./script/conflate_oa.sh` and `./script/conflate_osm.sh` in this repository which make running this process much easier.\n\nnote: We only support `openaddreses` and `openstreetmap` formats, you will need to create a custom importer for other sources.\n\n### precompute geometry\n\nFinally we will compute the fractional house numbers for each vertex (corner) of the street and add them to the `address.db` database.\n\nSee the [building the databases](https://github.com/pelias/interpolation#building-the-databases) section below for detailed information on which commands to run.\n\n## Building the databases\n\n### polyline\n> import road network data in the polyline format\n\nfind data here: https://github.com/pelias/polylines\n```bash\n./interpolate polyline street.db < /data/new_zealand.polylines\n```\n\n### oa\n> import openaddresses data and conflate it with the street data\n\nfind data here: https://openaddresses.io/\n```bash\n./interpolate oa address.db street.db < /data/oa/nz/countrywide.csv\n```\n\nnote: sorting the openaddresses files so that addresses on the same street are adjacent will significantly speed up imports, you can find an example of the commands required to sort the data in `./script/concat_oa.sh`.\n\n### osm\n> import openstreetmap data and conflate it with the street data\n\nfind data here: https://mapzen.com/data/metro-extracts/\n\nthe importer expects the OSM data in the JSON format exported by https://github.com/pelias/pbf2json, this format is not strictly equivalent to the http://overpass-api.de/output_formats.html#json standard, be aware.\n\nfor now it's best to use `pbf2json` to convert a `.osm.pbf` file in to json, then pipe that data in to `./interpolate osm`:\n\n```bash\n./build/pbf2json.linux-x64 -tags=\"addr:housenumber+addr:street\" london.osm.pbf > osm_data.json\n```\n\n```bash\n./interpolate osm address.db street.db < osm_data.json\n```\n\n### tiger\n> import US Census Bureau TIGER data and conflate it with the street data\n\nfind data here: https://www.census.gov/geo/maps-data/data/tiger-line.html\n\na script is provided in `./script/update_tiger.sh` which will download files for the whole of the USA, this script is safe to run multiple times as it will only update the data which has changed.\n\n```bash\n./interpolate tiger address.db street.db\n```\n\n### vertices\n> compute fractional house numbers for the street vertices\n\n```bash\n./interpolate vertices address.db street.db\n```\n\n#### logging\n\nyou can record a log of addresses which do not find a matching street. simply create an additional file descriptor, this will trigger the process to use it for logging. eg:\n\n```bash\ncat /data/oa/nz/countrywide.csv | ./interpolate oa address.db street.db 3> skip.list\n```\n\n## Using the command line\n\n### help\n> get information about which commands are available and list the arguments they accept\n\n```bash\n$ ./interpolate help\n\nUsage: interpolate [command] [options]\nNote: you will need to pipe data in to the import/conflate commands\n\n help output usage information\n search [address_db] [street_db] [lat] [lon] [house_number] [street_name] search database for specified housenumber + street\n polyline [street_db] import polyline data in to [street_db]\n oa [address_db] [street_db] conflate oa csv file in to [address_db] using [street_db]\n osm [address_db] [street_db] conflate osm file in to [address_db] using [street_db]\n tiger [address_db] [street_db] conflate tiger address range geojson file in to [address_db] using [street_db]\n vertices [address_db] [street_db] compute fractional house numbers for line vertices\n extract [address_db] [street_db] [lat] [lon] [street_name] extract street address data for debugging purposes\n server [address_db] [street_db] start a web server\n\n build run the import script\n```\n\n### search\n> search the db for an address, return an interpolated value if an exact match does not exist\n\nnote: the lat/lon values you provide are in order to disambiguate the street, they must lie within the bounding box of the desired street.\n\n```bash\n./interpolate search address.db street.db \"-41.288788\" \"174.766843\" \"16\" \"glasgow street\"\n\ntype\tinterpolated\nsource\tmixed\nnumber\t16\nlat\t-41.2886487\nlon\t174.7670925\n```\n\n### extract\n> extract address data from the db for a specific street\n\nnote: the lat/lon values you provide are in order to disambiguate the street, they must lie within the bounding box of the desired street.\n\n```bash\n./interpolate extract address.db street.db \"-41.288788\" \"174.766843\" \"glasgow street\"\n\n┌───────┬────┬────────┬─────────────┬─────────────┬─────────────┬────────┬─────────────┬─────────────┐\n│ rowid │ id │ source │ housenumber │ lat │ lon │ parity │ proj_lat │ proj_lon │\n├───────┼────┼────────┼─────────────┼─────────────┼─────────────┼────────┼─────────────┼─────────────┤\n│ 5 │ 1 │ OA │ 1 │ -41.2871999 │ 174.766753 │ R │ -41.287285 │ 174.7666662 │\n├───────┼────┼────────┼─────────────┼─────────────┼─────────────┼────────┼─────────────┼─────────────┤\n│ 23 │ 1 │ VERTEX │ 2.535 │ │ │ │ -41.287388 │ 174.766845 │\n├───────┼────┼────────┼─────────────┼─────────────┼─────────────┼────────┼─────────────┼─────────────┤\n│ 22 │ 1 │ VERTEX │ 3.376 │ │ │ │ -41.287461 │ 174.766921 │\n├───────┼────┼────────┼─────────────┼─────────────┼─────────────┼────────┼─────────────┼─────────────┤\n...\n```\n\n## Using the web server\n\n### Start the web server\n> run a web server which exposes the search APIs via an HTTP interface\n\n```bash\n./interpolate server address.db street.db\n\nserver listening on port 3000\n```\n\n#### Configuration via Environment Variables\n\nThe API supports additional environment variables that affect its operation:\n\n| Environment Variable | Default | Description |\n| -------------------- | ------- | ----------- |\n| `HOST` | `undefined` | The network address that interpolation will bind to. Defaults to whatever the current Node.js default is, which is currently to listen on `0.0.0.0` (all interfaces). See the [Node.js Net documentation](https://nodejs.org/api/net.html#net_server_listen_port_host_backlog_callback) for more information. |\n| `PORT` | `3000` | The TCP port that interpolation will use for incoming network connections |\n\n### GET /search/{format}\n> search the db for an address, return an interpolated value if an exact match does not exist\n\n`geojson`: `/search/geojson?lat=-41.288788&lon=174.766843&number=16&street=glasgow%20street`\n\n`html`: `/search/table?lat=-41.288788&lon=174.766843&number=16&street=glasgow%20street`\n\n### GET /extract/{format}\n> extract address data from the db for a specific street\n\n`geojson`: `/extract/geojson?lat=-41.288788&lon=174.766843&names=glasgow%20street`\n\n`html`: `/extract/table?lat=-41.288788&lon=174.766843&names=glasgow%20street`\n\n### GET /street/near/geojson\n> find the 100 nearest streets to a specific lat/lon pair, ordered by distance ASC\n\n`geojson`: `/street/near/geojson?lat=-41.288788&lon=174.766843`\n\n### GET /street/{id}/geojson\n> return the geometry for a specific street id\n\n`geojson`: `/street/10/geojson`\n\nsee: [source](https://github.com/pelias/interpolation/blob/master/cmd/server.js) for more information.\n\n## docker\n\n### build docker image\nThis can take some time for the first build due to installing libpostal from source.\n```bash\ndocker build -t pelias/interpolation .\n```\n\nYou can confirm that worked with:\n```bash\ndocker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\npelias/interpolation latest 7ca651b86a63 16 minutes ago 3.068 GB\n```\n\n### run docker image\nThis will run a new container based off the image created above.\n\nNotes:\n- `-p` controls port mapping (port `5000` on the host maps to `3000` in the container)\n- `-v` controls volume mapping (`/tmp` on the host maps to `/data` in the container)\n- `-d` tells docker to run the container in the background (daemonize)\n\nThis will launch the server using the databases `/tmp/address.db` and `/tmp/street.db` which must be present on the host machine\n\n```bash\ndocker run -p 5000:3000 -v /tmp:/data -d pelias/interpolation ./interpolate server /data/address.db /data/street.db\n```\n\nYou can confirm that worked with:\n```bash\n$ docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nac9c8f607b2e pelias/interpolation \"./interpolate server\" 14 minutes ago Up 14 minutes 0.0.0.0:5000->3000/tcp jolly_hamilton\n```\n\nYou should now be able to access the web server locally at `http://localhost:5000/demo/`\n\n### running scripts other than `server` in the docker container\n\nYou can run any command supported by `./interpolate` via the docker container, such as:\n\n```bash\ncat /tmp/new_zealand.polylines | docker run --rm -v /tmp:/data pelias/interpolation ./interpolate polyline /data/nz.db\n```\n\nThis will pipe the `/tmp/new_zealand.polylines` file from your host machine to the `./interpolate polyline`\nscript running in the container. The script generates an SQLite database `/data/nz.db` in the container,\nwhich is available under `/tmp/nz.db` on your host, because we use the mapping `-v /tmp:/data`.\n\n### running a build in the docker container\n\nThe build scripts are configurable via a combination of environment variables and a `pelias-config` json file.\nYou will need to download your data before running the build command.\n\nTo make use of the `pelias-config` functionality, you'll need to create a new json file called `pelias.json` for example.\nThe relevant parts of that new file should look as follows. To direct to download script to this file,\nthe `PELIAS_CONFIG` environment variable should be set. You can read more details on how to use the `pelias-config` module\n[here](https://github.com/pelias/config).\n\n```javascript\n{\n \"imports\": {\n \"interpolation\": {\n \"download\": {\n \"tiger\": {\n \"datapath\": \"/tmp/data/tiger/\",\n \"states\": [\n {\n \"state_code\": 41\n }\n ]\n }\n }\n }\n }\n}\n```\n\nNote that `datapath` will default to `./data/downloads` if not specified.\n\nTo filter the TIGER data download you can set `state_code` property in the `pelias-config` file to the 2 digit code of the state to be downloaded.\n\nNote: some state codes begin with a leading zero, you may specify a string value or omit the extra zero and provide an integer value.\n\nIn the example configuration above, the state code for Oregon, `41`, is used to limit the download.\n\nThe state code can found by referencing the table below. If no `state_code` value is found, all US data will be downloaded.\n\n| code | state |\n| --- | --- |\n| 01 | Alabama |\n| 02 | Alaska |\n| 04 | Arizona |\n| 05 | Arkansas |\n| 06 | California |\n| 08 | Colorado |\n| 09 | Connecticut |\n| 10 | Delaware |\n| 11 | District of Columbia |\n| 12 | Florida |\n| 13 | Georgia |\n| 15 | Hawaii |\n| 16 | Idaho |\n| 17 | Illinois |\n| 18 | Indiana |\n| 19 | Iowa |\n| 20 | Kansas |\n| 21 | Kentucky |\n| 22 | Louisiana |\n| 23 | Maine |\n| 24 | Maryland |\n| 25 | Massachusetts |\n| 26 | Michigan |\n| 27 | Minnesota |\n| 28 | Mississippi |\n| 29 | Missouri |\n| 30 | Montana |\n| 31 | Nebraska |\n| 32 | Nevada |\n| 33 | New Hampshire |\n| 34 | New Jersey |\n| 35 | New Mexico |\n| 36 | New York |\n| 37 | North Carolina |\n| 38 | North Dakota |\n| 39 | Ohio |\n| 40 | Oklahoma |\n| 41 | Oregon |\n| 42 | Pennsylvania |\n| 72 | Puerto Rico |\n| 44 | Rhode Island |\n| 45 | South Carolina |\n| 46 | South Dakota |\n| 47 | Tennessee |\n| 48 | Texas |\n| 49 | Utah |\n| 50 | Vermont |\n| 51 | Virginia |\n| 53 | Washington |\n| 54 | West Virginia |\n| 55 | Wisconsin |\n| 56 | Wyoming |\n\nFor more fine-grained control, you can also set the `county_code` property in the `pelias-config` file to the 3 digit code of the county to be downloaded.\n\nNote: some county codes begin with a leading zero, you may specify a string value or omit the extra zero and provide an integer value.\n\nNote: you must specify a `state_code` when specifying a `county_code`.\n\n```\n\"states\": [\n {\n \"state_code\": 41, \"county_code\": 1\n }\n]\n```\n\nCheck [the census website](https://www.census.gov/geographies/reference-files/2020/demo/popest/2020-fips.html) for a complete list of state and county FIPS codes.\n\n### docker example\n\n```bash\n# prepare a build directory and a data directory to hold the newly created database files\nmkdir -p /tmp/data/berlin\n\n# download polyline street data\ncurl -s http://missinglink.files.s3.amazonaws.com/berlin.gz | gzip -d > /tmp/data/berlin.0sv\n\n# download and extract openaddresses data\ncurl -s https://s3.amazonaws.com/data.openaddresses.io/runs/142027/de/berlin.zip > /tmp/data/berlin.zip\nunzip /tmp/data/berlin.zip -d /tmp/data\n\n# download openstreetmap data\ncurl -s https://s3.amazonaws.com/metro-extracts.mapzen.com/berlin_germany.osm.pbf > /tmp/data/berlin.osm.pbf\n\n# download tiger data (note data directory will be created if it doesn't exist)\nexport PELIAS_CONFIG=./pelias.json\nnpm run download-tiger\n```\n\nwe will mount `/tmp/data` on the local machine as `/data` inside the container, so be careful to set paths as they appear inside the container.\n\n```bash\ndocker run -i \\ # run interactively (optionally daemonize with -d)\n -v /tmp/data:/data \\ # volume mapping\n -e 'BUILDDIR=/data/berlin' \\ # location where the db files will be created\n -e 'POLYLINE_FILE=/data/berlin.0sv' \\ # location of the polyline data\n -e 'OAPATH=/data/de' \\ # location of the openaddresses data\n -e 'PBF2JSON_FILE=/data/berlin.osm.pbf' \\ # location of the openstreetmap data\n pelias/interpolation ./interpolate build\n```\n\nonce completed you should find the newly created `street.db` and `address.db` files in `/tmp/data/berlin` on your local machine.\n\n## Configuration with Pelias Geocoder API\n\nTo use Interpolation service with the Pelias API, [configure the pelias config file](https://github.com/pelias/api#pelias-config) with the port that interpolation is running on.\n\n## development\n\n### install dependencies\n\n*note:* [libpostal](https://github.com/openvenues/node-postal#troubleshooting) **must** be installed on your system before you continue!\n\nThe `Dockerfile` in this repo has complete instructions on how to install everything from scratch on Ubuntu.\n\n### TIGER dependency on GDAL\n\nThe TIGER importer requires the `ogr2ogr` binary from `gdal` version 2+ in order to extract data from the `.shp` files provided by the US Census Bureau.\n\nOn linux this you can install this with a command such as `sudo apt-get install gdal-bin`, on OSX you will need to follow [this guide](https://www.karambelkar.info/2016/10/gdal-2-on-mac-with-homebrew/).\n\n```bash\nnpm install\n```\n\n### run tests\n```bash\nnpm test\n```\n\n### run linter\nnote: if you are using the `atom` editor, we recommend the [jshint](https://atom.io/packages/jshint) plugin.\n```bash\ngit commit\n```\n\n## functional tests\n\nthis repo contains a bunch of functional end-to-end tests in the [./test/functional](https://github.com/pelias/interpolation/tree/master/test/functional) directory.\n\neach test contains a [reports](https://github.com/pelias/interpolation/tree/master/test/functional/basic/reports) directory which contains human-readable visual output of each test case, including a geojson map view showing all the point data.\n\n### create a new functional test\n\nthe easiest/best way to debug an issue is to create a new functional test case and use the test suite to assert conditions and generate visual output which you can inspect.\n\nthese are the steps I took to create the `potsdamerplatz` test case:\n\n```bash\n# copy an existing test case\ncp -r test/functional/basic test/functional/potsdamerplatz\n\n# extract the relevant polylines from a large polyline source\ngrep -Pia \"potsdamer\\s?(platz|strasse|straße)\" /data/planet.polylines > test/functional/potsdamerplatz/osm.polylines\n\n# extract the relevant address points from a large openaddresses file (header line then body)\nhead -n1 /data/oa/de/berlin.csv > test/functional/potsdamerplatz/oa.csv\ngrep -Pia \"potsdamer\\s?(platz|strasse|straße)\" /data/oa/de/berlin.csv >> test/functional/potsdamerplatz/oa.csv\n\n# extract the relevant address points from an openstreetmap PBF extract\n# see: https://github.com/pelias/pbf2json\n./build/pbf2json.linux-x64 -tags=\"addr:housenumber+addr:street\" /data/extract/greater-london-latest.osm.pbf | grep -i nevern > test/functional/nevern_square/osm.addresses.json\n```\n\nnext add that test case to `./test/_func.js` in order to it run every time anyone runs `npm test`.\n\nyou can now edit the contents of `test/functional/potsdamerplatz/run.js` to suit your needs, you should rename the text at the bottom of the file which says something like `\"functional: basic\"` to be more descriptive, in this case we will call it `\"functional: potsdamerplatz\"`. now the output from `npm test` will include that label next to each assertion run in the file.\n\ngreat! you can skip the units tests and only run the functional tests with `npm run funcs`, go ahead and do that now and you will see your new tests failing; which is good! if you are going to be running that command a lot and you don't care to wait on the other tests, you can comment them out in `./test/_func.js`.\n\nnow your test case is running it's time to have a poke around in that new directory you made.\n\nrunning the tests will produce new `street.db` & `address.db` files, you can query them directly from the command line to check what's inside them looks correct:\n\n```bash\nsqlite3 test/functional/potsdamerplatz/street.db \"SELECT * FROM polyline JOIN names ON polyline.id = names.id WHERE names.name LIKE '%platz%'\"\n```\n\nor you can start an interactive shell and have a poke around in there:\n\n```bash\nsqlite3 test/functional/potsdamerplatz/street.db\n\nSQLite version 3.8.11.1 2015-07-29 20:00:57\nEnter \".help\" for usage hints.\n\nsqlite> select count(*) from polyline;\n64\n\nsqlite> .exit\n```\n\nyou'll find a subdirectory called `./fixture` which is where all your data fixtures will live, you can query the database directly and save the response to that directory with a command such as:\n\n```bash\nsqlite3 test/functional/potsdamerplatz/address.db \"SELECT * FROM address WHERE id = 1\" > test/functional/potsdamerplatz/fixture/expected.dump\n```\n\nif you're hunting for a specific road segment to debug, you can open up your `test/functional/potsdamerplatz/osm.polylines` file and try to find the appropriate line in there, the line numbers will correspond to the ids, so the first line in that file is id=1 in the `street.db`.\n\nto visually inspect the polylines, you can cut them before the name and paste them in the search box here: http://valhalla.github.io/demos/polyline/\n\nexample:\n```\nii|ccBsskoX_@wLaAkUi@sJ{@oM??mHn@??HtL~IUvGc@zJyA|Cg@tCkAhBkA~EyDpBaDnGgIeBqEoEnGwGvEqAz@wAr@qC|@wMl@mJ^\n```\n\nlikewise if you are looking for a specific address, you can open up `test/functional/potsdamerplatz/oa.csv` and find the address in there (be aware that some very precise floating point numbers get truncated and so may not match exactly in tools like grep), you should then be able to find them in the `address.db`:\n\n```\nsqlite3 test/functional/potsdamerplatz/address.db \"SELECT * FROM address\" | grep \"52.5046774\"\n\n45|1|OA|59.0|52.5046774|13.3676731|R|52.5047673|13.3674717\n```\n\nyou'll also find a subdirectory called `./reports` which you can use to spot check the data, if you have [geojsonio](https://github.com/mapbox/geojsonio-cli) installed you can pipe the file directly to the web:\n\n```bash\ncat test/functional/potsdamerplatz/reports/preview.geojson | geojsonio\n```\n\nonce you have completed your tests and committed the files, your preview files will be visible to everyone via github.\n\nthe `./reports` directory also contains the `stdio` files from each command that was executed and a list of records which failed to conflate. these files are ignored by `.gitignore` so they don't show up on github.\n\n### Updating the mock libpostal responses\n\nBecause of the high memory and disk space requirements of libpostal, tests can be run using a list of known libpostal responses, rather than calling libpostal itself.\n\nThese mock responses are stored in `test/lib/mock_libpostal_responses.json`. They will be used by the tests as long as the `MOCK_LIBPOSTAL` environment variable is set.\n\nWhen libpostal itself changes, these mock responses need to be updated. Running the tests with `SEED_MOCK_LIBPOSTAL` set as well will cause the mock library to actually call libpostal itself, and update `mock_libpostal_responses.json`.\n","readmeFilename":"README.md"}
![](\"https://raw.githubusercontent.com/pelias/design/master/logo/pelias_github/Github_markdown_hero.png\"){kind=logo}
\n![](\"https://img.shields.io/github/license/pelias/api?style=flat&color=orange\")
![](\"https://img.shields.io/docker/pulls/pelias/api?style=flat&color=informational\")
![](\"https://img.shields.io/gitter/room/pelias/pelias?style=flat&color=yellow\")