install on server

[devinrsmith]

I'm trying to install pelias on a server (no vagrant). Even though this is the vagrant repository, it also seems like this is also the central source for bootstrapping and importing data into the server.

I was hoping that https://github.com/pelias/pelias/blob/master/INSTALL.md would help me bootstrap the system, but no such luck at the moment.

I'm not familiar enough with chef(-solo?) to figure out how to transform the Vagrantfile into a pure chef solution. Any help would be much appreciated.

[orangejulius]

Hey @devinrsmith,
We do use Chef to provision Pelias for Mapzen Search but, confusingly, it uses completely separate Chef scripts from the ones in this repository. Sadly, these scripts aren't public because they're too conflated with our own infrastructure, and aren't really general purpose enough for wide use. One thing we want to do in the near future is spin out the common parts into the open, so that they can be used by us and this vagrant image.

Also bad news: we don't have a single, comprehensive, up to date tutorial on manually setting up Pelias. What we currently have is split between the document you linked to, an old blog post, which is out of date, and the readme for one of our importers. There is useful information in all of them though.

Fortunately, consolidating those resources is now at the top of our list of things to do, and you should start to see some better setup instructions soon. In the meantime, they roughly look like this:

1.) install elasticsearch
2.) use pelias/schema to create the Elasticsearch schema
3.) Run whichever of our 4 importers are interesting to you. Running all 4 will require about 150GB of memory for your Elasticsearch cluster and take ~2 days.
4.) Start up pelias/api and you can now hit it with the same API as our official docs

As we update the documentation I'll try to update you here and we would love your feedback to continue to make our docs better.

[devinrsmith]

Thanks for the quick response.

I think I understand the large picture of how the system works (as you have laid out in yours steps). The hard part is in the details :)

I'm not sure what the status of pelias/vagrant is, but it looks like there a few key components that Vagrant bootstraps off of: the Berksfile, chef.json, and chef.run_list. I'm assuming there is some easy way to decouple these settings (specifically the chef.json and chef.run_list) from Vagrant and execute them directly. This might be an easy place to start if they existed. (While it would be easier to just use Vagrant, my local box is underpowered, and our VPS servers don't play nicely with virtualbox.)

Our dataset would require a very small subset of the data... just one osm metro extract and the corresponding openaddresses. It would be very convenient to be able to reliably reproduce the server setup for testing and development purposes.

In the very short term if we can't get a server up, hopefully we'll be able to substitute in Mapzen Search.

Looking forward to the improved documentation and setup!

[easherma]

Hey Devin, been doing some similar work lately. I'll add that, if you're setting up a local version, it might be best to use the 'production' repos instead of dev. They may not line up with the current documentation, but you'll have less problems calling and getting results from whatever data you import (IMHO).

[ghost]

Have you had any success with this, @easherma?

How did you end up, @devinrsmith?

[orangejulius]

Hey everyone, we also have brand new full installation docs. They explain how to set up a full Pelias install manually so as to cover all the pieces involved. Please take a look and if anything is confusing or omitted, let me know.

Improving the Vagrant image and breaking out the chef scripts hidden in our infrastructure and in this repo is on our radar for the near future.

[easherma]

@brianjcollins like @orangejulius mentions, I ended up doing a manual install. Ironically, I found it easier to follow and troubleshoot than the 'click and run' promise of the vagrant build, especially because it forced me to learn more about how the modules all fit together.

[ghost]

That's perfect, thanks @orangejulius! I'll build it out over the weekend, and if I have anything to add, I will let you know.

@easherma Agree 100%, never a bad idea knowing how all the pieces move.

[ghost]

I'm assuming it's a formatting issue, but I can't get any 'npm start' to function when pelias.json is in my users root folder. I've removed a lot of key/values that didn't seem necessary in pelias.json where the defaults would suffice, however, I can't seem to get much of an output to help me diagnose the problem line. If I remove pelias.json, it gives a different error, that at least is easily understood, it can't stat the location of the defaults for openstreetmap.

ubuntu@pelias-01:~/openstreetmap$ npm start

[email protected] start /home/ubuntu/openstreetmap
node index.js

/home/ubuntu/openstreetmap/node_modules/mergeable/lib/Mergeable.js:70
    throw new Error( 'failed to merge config from path:' + path );
    ^

Error: failed to merge config from path:/home/ubuntu/pelias.json
    at Config._requirePath (/home/ubuntu/openstreetmap/node_modules/mergeable/lib/Mergeable.js:70:11)
    at Config.deepMergeFromPath (/home/ubuntu/openstreetmap/node_modules/mergeable/lib/Mergeable.js:20:21)
    at generate (/home/ubuntu/openstreetmap/node_modules/pelias-config/index.js:23:32)
    at Object.<anonymous> (/home/ubuntu/openstreetmap/node_modules/pelias-logger/index.js:10:30)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at require (internal/module.js:20:19)

npm ERR! Linux 3.13.0-86-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! node v5.11.1
npm ERR! npm  v3.8.6
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: `node index.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] start script 'node index.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the pelias-openstreetmap package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node index.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs pelias-openstreetmap
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls pelias-openstreetmap
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/ubuntu/openstreetmap/npm-debug.log
`

[orangejulius]

hey @brianjcollins,
Ugh, we all run into that error and we definitely have to make it better somehow, sorry about that. It generally means that your pelias.conf file is not valid JSON. Unfortunately JSON is in fact rather strict. Be sure that all commas are in the right place (but no comma at the end of an array or property list), all quotes are double quotes, etc. http://jsonlint.com/ should also help.

[ghost]

I figured as much, however, even while using the following, it errors. I initially went to... https://github.com/pelias/config/blob/master/config/local.json to get the example and removed the data peice by peice hoping to find the problem line, however nothing allowed npm to succed.

{
   "imports": {
      "openstreetmap": {
         "datapath": "/home/ubuntu/data/openstreetmap",
         "import": [
            {
               "filename": "detroit_michigan.osm.pbf"
            }
         ]
      }
   }
}

[orangejulius]

Huh, ok. So looking at the line in question, it looks like it might be some sort of error actually loading the file. Can you make sure the config file is actually at /home/ubuntu/pelias.json, Running cd; pwd on the shell returns /home/ubuntu, and that the PELIAS_CONFIG environment variable is unset (or is pointing to your config file)?

[ghost]

Actually, your first intuition was correct. It was JSON formatting, why I don't know though... I used the following and it succeeded. Possibly due to the newlines between the brackets and braces? I don't know, but it's working...

{
  "logger": {
    "level": "debug"
  },
  "esclient": {
    "hosts": [{
      "env": "development",
      "protocol": "http",
      "host": "localhost",
      "port": 9200
    }]
  },
  "imports": {
    "openstreetmap": {
      "adminLookup": false,
      "leveldbpath": "/tmp",
      "datapath": "/home/ubuntu/data/openstreetmap",
      "import": [{
        "filename": "detroit_michigan.osm.pbf"
      }]
    }
  }
}

[ghost]

As I worked through this process, I recorded my steps and put it into a public gist. It isn't ground breaking, but it's start to finish from a clean instance to having pelias installed with some data.

I saved it for myself but I figured it might save someone some time. Obviously, improve, replicate, etc etc..

[https://gist.github.com/brianjcollins/66fefa3a08f8be355b11cb00a1c2b7d0]

[easherma]

Awesome, thanks for sharing!

On Sat, May 28, 2016 at 3:02 PM, Brian Collins [email protected]
wrote:

As I worked through this process, I recorded my steps and put it into a
public gist. It isn't ground breaking, but it's start to finish from a
clean instance to having pelias installed with some data.

I saved it for myself but I figured it might save someone some time.
Obviously, improve, replicate, etc etc..

[https://gist.github.com/brianjcollins/66fefa3a08f8be355b11cb00a1c2b7d0]

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#41 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AKDTT7osHkSMaDt1atKASpw20n6_-Xq4ks5qGJ8-gaJpZM4IFr-C
.

Eric Sherman
224-402-0004