Integrate with make_osc & deploy on osm-database #541

Hoverbear · 2017-01-10T15:45:10Z

There was an make_osc.rb and string_hstore.rb file on the deployed instances which was not in any of our repos. These are currently required for syncing between PostgreSQL and MySQL.

This PR converts configures things to appropriately talk to it.

schultyy

Found some minor things otherwise lgtm and good work to convert it into a Rake Task

schultyy · 2017-01-13T09:21:59Z

lib/tasks/make_osc.rake

+require 'pg'
+require 'builder'
+
+$fields = [ 'shop', 'office', 'aerialway', 'aeroway', 'amenity', 'tourism', 'historic', 'sport', 'leisure', 'public_transport' ]


Is there a reason this has to be a global variable? I'm asking because of the $.

I don't know. I didn't write this code.

schultyy · 2017-01-13T09:22:15Z

lib/tasks/make_osc.rake

+namespace :make_osc do
+  desc 'Make OSC files.'
+  task :full => :environment do
+    $conn = PGconn.open(:dbname => 'osm', :user => 'osm', :password => 'osm', :host => 'osm-database')


Same as above? Does this one have to be global?

I don't know. I didn't write this code.

schultyy · 2017-01-13T09:23:05Z

lib/tasks/make_osc.rake

+  end
+
+  task :diff => :environment do
+    $conn = PGconn.open(:dbname => 'osm', :user => 'osm', :password => 'osm', :host => 'osm-database')


Same as above with $conn

I don't know. I didn't write this code.

schultyy · 2017-01-13T09:27:32Z

Gemfile

@@ -3,6 +3,7 @@ source 'https://rubygems.org'
 group :default do
  gem 'rails', '~> 4.1.0'
  gem 'mysql2', '~> 0.3.18'
+  gem 'pg', '~> 0.18.4'


The pg gem requires some native extensions to be build on installation. For that it requires the Postgres dev packages. By having it in the :default group we require all devs to have Postgres installed on their machines when they want to work on wheelmap.

I'd propose to have a separate group for this we then can exclude during installation via bundle install --without.

Hm, this seems reasonable. I'll work on this.

Hoverbear · 2017-01-13T09:34:08Z

We discussed removing the $ from the variables. Will follow up with this.

Hoverbear · 2017-01-13T09:43:23Z

@schultyy Those should be resolved. Am testing...

Hoverbear · 2017-01-13T10:08:30Z

Requested changes made and work.

Xylakant · 2017-01-13T10:09:46Z

I would consider stopping this effort: What's the gain compared to the effort invested and the dependencies introduces into the main app. I'd rather see this in a small standalone script with a gemfile that's separate from the main wheelmap app.

Hoverbear · 2017-01-13T10:12:36Z

@Xylakant So a separate repo which we also deploy via capistrano or?

Xylakant · 2017-01-13T10:13:40Z

I'm uncertain if cap is not an overkill in that case. This will not see many updates, installing that on the machine via chef should be perfectly fine.

Hoverbear · 2017-01-13T10:16:33Z

@Xylakant Well, the only dependency we introduce is pg which we've moved into the production group anyways so it can be excluded. It's a rather important part of a deployment of wheelmap anyways and I'm concerned placing it somewhere else would cause it to get forgotten/lost (again).

Xylakant · 2017-01-13T10:19:55Z

The use of a group is somewhat limited, people expect bundle install to install all development dependencies and excludes should be limited to environments that are production or close to. It also requires us to install pg on all app machines that should not require it currently.

I agree that it should be committed somewhere - it can even live in a "tools" or "osm-sync" subfolder of this repo, but I don't think the effort of folding this into the rake task has any positive ROI either in time or reduced complexity.

Hoverbear · 2017-01-13T10:22:15Z

So my reasoning for making it a rake task was that it was called by a rake task already, so it seemed easy to integrate that way.

I suppose it wouldn't be much harder to drop this in a subfolder and give it it's own gemfile then modify capistrano to have the dependencies installed when it is deployed.

Xylakant · 2017-01-13T10:24:47Z

as I said - I don't think that installing it via CAP has any positive ROI either, but that's a minor point. I'd really go and move it to a dedicated repo and treat it like we treat any other dependency: We assume that it's available at the point where we install the app.

Hoverbear · 2017-01-13T10:26:28Z

Ok.

Hoverbear · 2017-01-13T12:48:22Z

Okay, this should be good to go.

Hoverbear · 2017-01-23T15:14:36Z

Rebased on top of feature/infra-rebuild

Hoverbear · 2017-04-25T13:07:43Z

Rebased.

Hoverbear added enhancement Infra-rebuild labels Jan 10, 2017

Hoverbear requested a review from schultyy January 10, 2017 15:45

holgerd added the in progress label Jan 10, 2017

Hoverbear force-pushed the feature/infra-rebuild-syncing branch from 5ddb5d6 to 3a6c0dd Compare January 13, 2017 09:18

schultyy requested changes Jan 13, 2017

View reviewed changes

Hoverbear force-pushed the feature/infra-rebuild-syncing branch 2 times, most recently from 2925593 to b692d2a Compare January 13, 2017 09:43

Hoverbear changed the title ~~Integrate make_osc from old infra into rake~~ Integrate with make_osc Jan 13, 2017

Hoverbear changed the title ~~Integrate with make_osc~~ Integrate with make_osc & deploy on osm-database Jan 13, 2017

Hoverbear force-pushed the feature/infra-rebuild branch from f97ad60 to a6ef178 Compare January 13, 2017 15:02

Hoverbear added review and removed in progress labels Jan 17, 2017

Hoverbear force-pushed the feature/infra-rebuild-syncing branch from b794f12 to 5b2d883 Compare January 23, 2017 14:50

Hoverbear force-pushed the feature/infra-rebuild branch from a6ef178 to d1e9709 Compare April 10, 2017 10:16

Hoverbear force-pushed the feature/infra-rebuild-syncing branch from 178af96 to c369343 Compare April 25, 2017 12:55

syncing changes

ca7d305

Hoverbear force-pushed the feature/infra-rebuild-syncing branch from c369343 to ca7d305 Compare April 25, 2017 13:07

Use bundle

c5bed48

Hoverbear merged commit 49ccec1 into feature/infra-rebuild May 22, 2017

Hoverbear deleted the feature/infra-rebuild-syncing branch May 22, 2017 09:03

holgerd removed the review label May 22, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate with make_osc & deploy on osm-database #541

Integrate with make_osc & deploy on osm-database #541

Hoverbear commented Jan 10, 2017 •

edited

Loading

schultyy left a comment

schultyy Jan 13, 2017

Hoverbear Jan 13, 2017 •

edited

Loading

schultyy Jan 13, 2017

Hoverbear Jan 13, 2017

schultyy Jan 13, 2017

Hoverbear Jan 13, 2017

schultyy Jan 13, 2017

Hoverbear Jan 13, 2017 •

edited

Loading

Hoverbear commented Jan 13, 2017

Hoverbear commented Jan 13, 2017 •

edited

Loading

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Hoverbear commented Jan 23, 2017

Hoverbear commented Apr 25, 2017

Integrate with make_osc & deploy on osm-database #541

Integrate with make_osc & deploy on osm-database #541

Conversation

Hoverbear commented Jan 10, 2017 • edited Loading

schultyy left a comment

Choose a reason for hiding this comment

schultyy Jan 13, 2017

Choose a reason for hiding this comment

Hoverbear Jan 13, 2017 • edited Loading

Choose a reason for hiding this comment

schultyy Jan 13, 2017

Choose a reason for hiding this comment

Hoverbear Jan 13, 2017

Choose a reason for hiding this comment

schultyy Jan 13, 2017

Choose a reason for hiding this comment

Hoverbear Jan 13, 2017

Choose a reason for hiding this comment

schultyy Jan 13, 2017

Choose a reason for hiding this comment

Hoverbear Jan 13, 2017 • edited Loading

Choose a reason for hiding this comment

Hoverbear commented Jan 13, 2017

Hoverbear commented Jan 13, 2017 • edited Loading

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Xylakant commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Hoverbear commented Jan 13, 2017

Hoverbear commented Jan 23, 2017

Hoverbear commented Apr 25, 2017

Hoverbear commented Jan 10, 2017 •

edited

Loading

Hoverbear Jan 13, 2017 •

edited

Loading

Hoverbear Jan 13, 2017 •

edited

Loading

Hoverbear commented Jan 13, 2017 •

edited

Loading