Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pre/postinstall scripts to prevent 'npm install' failure #16454

Closed
wants to merge 202 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
202 commits
Select commit Hold shift + click to select a range
ce67f3b
Improve info for :update_user command
Shylpx Oct 26, 2021
09d45b6
Update NEWS.md
Shylpx Oct 26, 2021
0ef032a
Fix 404 page
Shylpx Oct 28, 2021
b4f875c
Fix privacy dropdown behavior
moicalcob Oct 29, 2021
9c9f2d9
Update NEWS.md
moicalcob Oct 29, 2021
79d6112
Remove spaces
moicalcob Oct 29, 2021
af5f2cc
Remove spaces
moicalcob Oct 29, 2021
c8136ee
Update assets version
moicalcob Oct 29, 2021
c4966b2
Merge branch 'feature/sc-172043/reef-admin-can-t-set-map-to-private' …
moicalcob Oct 29, 2021
bd27ce1
Remove old named map when a user is reaching the limit, and notify su…
moicalcob Oct 29, 2021
a5c6c89
Update NEWS.md
moicalcob Oct 29, 2021
9870550
Change the number of named maps deleted
moicalcob Oct 29, 2021
01501fa
Merge branch 'master' into feature/sc-172043/reef-admin-can-t-set-map…
moicalcob Nov 2, 2021
b3a0263
Bump assets version
Shylpx Nov 2, 2021
048ff8a
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Nov 4, 2021
36f4089
Update NEWS.md
Shylpx Nov 4, 2021
4a99d8c
Add new rake to change usernames
moicalcob Nov 8, 2021
e51ea11
Regenerate all published maps, instead of public ones
moicalcob Nov 8, 2021
de833a7
Merge branch 'master' into feature/sc-188883/abachiller-superadmin-ch…
Shylpx Nov 8, 2021
82dd384
Refactor after code review
moicalcob Nov 8, 2021
08ec69b
Merge pull request #16370 from CartoDB/feature/sc-190856/cch-users-ch…
moicalcob Nov 9, 2021
03df04a
Merge branch 'master' into feature/sc-172043/reef-admin-can-t-set-map…
moicalcob Nov 9, 2021
ffbe5d8
Merge pull request #16367 from CartoDB/feature/sc-172043/reef-admin-c…
moicalcob Nov 9, 2021
f46b62a
Upgrade assets version
moicalcob Nov 9, 2021
ad51b28
Merge branch 'master' into feature/sc-172043/reef-admin-can-t-set-map…
moicalcob Nov 9, 2021
3d7ec5c
Update NEWS.md
moicalcob Nov 9, 2021
fd290ac
Merge branch 'feature/sc-172043/reef-admin-can-t-set-map-to-private' …
moicalcob Nov 9, 2021
2e388c2
Merge pull request #16371 from CartoDB/feature/sc-172043/reef-admin-c…
moicalcob Nov 9, 2021
d424f64
Merge branch 'master' into bug/sc-95788/404-html-has-no-body
Shylpx Nov 9, 2021
8a0b4c5
Merge pull request #16369 from CartoDB/bug/sc-95788/404-html-has-no-body
Shylpx Nov 10, 2021
37cad92
Merge branch 'master' into feature/sc-188883/abachiller-superadmin-ch…
Shylpx Nov 10, 2021
2f63abe
Merge pull request #16363 from CartoDB/feature/sc-188883/abachiller-s…
Shylpx Nov 10, 2021
d10ae12
Add new resque job to delete multiple named maps
moicalcob Nov 11, 2021
497d02b
Replace match for start_with method
moicalcob Nov 11, 2021
1193ce3
Trigger build
moicalcob Nov 11, 2021
728448e
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Nov 11, 2021
58a611b
Add SAML random usernames choice
moicalcob Nov 11, 2021
3c98a86
Update NEWS.md
moicalcob Nov 11, 2021
ce3adaa
Fixes for testing
moicalcob Nov 11, 2021
81b7684
Fix tests
moicalcob Nov 12, 2021
2d573b0
Trigger build
moicalcob Nov 12, 2021
8b7f650
Fix tests
moicalcob Nov 12, 2021
d25e519
Fix username generation method
moicalcob Nov 15, 2021
d6dc25d
Merge pull request #16372 from CartoDB/feature-flag-random-username-saml
moicalcob Nov 15, 2021
da60ec9
Increase limit to 26
Shylpx Nov 16, 2021
157b55d
Update NEWS.md
Shylpx Nov 16, 2021
359b38c
Fix tests
Shylpx Nov 16, 2021
b60c73b
Merge pull request #16374 from CartoDB/feature/sc-192476/aberran-incr…
Shylpx Nov 16, 2021
7398b06
Check empty email config
moicalcob Nov 18, 2021
51caab2
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Nov 18, 2021
31d671b
Trigger build
moicalcob Nov 18, 2021
fc958d2
Invalidate sessions on session_salt issue
Shylpx Nov 22, 2021
1ebd395
Update NEWS.md
Shylpx Nov 22, 2021
a7f683d
Merge pull request #16376 from CartoDB/bug/sc-189965/alessandro238988…
Shylpx Nov 22, 2021
8fdab16
Add export_timeout to users
Shylpx Nov 22, 2021
bad150c
Change support_email env variable name
moicalcob Nov 22, 2021
a634226
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Nov 23, 2021
df18d5e
Fix tests and update NEWS.md
Shylpx Nov 26, 2021
7f8695f
Fix call to debounce
Shylpx Nov 29, 2021
4dce50c
Fix tests and update NEWS.md
Shylpx Nov 30, 2021
f85a0f1
Fix tests (again)
Shylpx Dec 3, 2021
82f20a3
Fix Hound issues
Shylpx Dec 3, 2021
c83621b
Merge pull request #16378 from CartoDB/bug/sc-79798/westermann-gruppe…
Shylpx Dec 9, 2021
372cfc1
Add check before destroying dependent visualizations
moicalcob Dec 20, 2021
b8f628e
Update NEWS.md
moicalcob Dec 20, 2021
1dc58a2
Test rubocop 1.12.0 version
Dec 21, 2021
2c1daac
Test 1.23.0 rubocop version
Dec 21, 2021
854aa63
Test activesupport 6.1.4.4 for rubocop
Dec 21, 2021
de19a94
Revert "Test activesupport 6.1.4.4 for rubocop"
Dec 21, 2021
47f2196
Revert "Test 1.23.0 rubocop version"
Dec 21, 2021
e577b90
Revert "Test rubocop 1.12.0 version"
Dec 21, 2021
820b3ed
Fixing rubocop integration
Dec 21, 2021
464e3ff
Using ruby 2.5.7 version
Dec 21, 2021
e64dd15
Update News.md
Dec 21, 2021
b0118e8
Using rubyversion 2.5.8
Dec 21, 2021
f06c6a1
Fix failing test
moicalcob Dec 21, 2021
e7efeb7
Apply temporary patch to FileUtils copy file function
moicalcob Dec 21, 2021
c2ad50f
Fix rubocop offenses
moicalcob Dec 21, 2021
aeabb09
Ignore line for rubocop
moicalcob Dec 21, 2021
a7df0ec
Ignore other rubocop offense
moicalcob Dec 21, 2021
47fd485
Add custom redirection
Shylpx Dec 22, 2021
a46159a
Update NEWS.md
Shylpx Dec 22, 2021
ba506a1
Try fixing tests
Shylpx Dec 22, 2021
e21e6db
Minor fixes
Shylpx Dec 22, 2021
29bfbbb
Minor fixes (again)
Shylpx Dec 22, 2021
5099187
Merge pull request #16382 from CartoDB/fixing_rubocop_integration
Dec 22, 2021
d44d103
Merge branch 'master' into feature/sc-197121/coral-reef-admin-builder…
Dec 22, 2021
1d4b619
Merge branch 'master' into feature/sc-198852/update-developers-carto-…
Shylpx Dec 22, 2021
fe825a0
Fix dependencies
Shylpx Dec 22, 2021
a1caab8
Fix Rubocop offenses
Shylpx Dec 22, 2021
7425fe5
Fix rubocop ofensse
moicalcob Dec 22, 2021
092913d
Merge branch 'feature/sc-197121/coral-reef-admin-builder-map-is-delet…
moicalcob Dec 22, 2021
8729c83
Fix another rubocop offense
moicalcob Dec 22, 2021
0121d83
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Dec 23, 2021
6704b14
Merge pull request #16383 from CartoDB/feature/sc-198852/update-devel…
Shylpx Dec 23, 2021
6ea3335
Fix rubocop offenses
moicalcob Dec 23, 2021
7138f55
Fix more rubocop offenses
moicalcob Dec 23, 2021
b637dce
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Dec 23, 2021
e36cbf4
Fix another rubocop offense
moicalcob Dec 23, 2021
93ff540
Merge branch 'feature/sc-178449/notify-about-the-limit-of-4096-named-…
moicalcob Dec 23, 2021
c5dcd58
Merge branch 'master' into feature/sc-189175/sql-api-oom-errors-when-…
Shylpx Dec 28, 2021
2fd1b21
Fix Rubocop offenses
Shylpx Dec 28, 2021
b8215b1
Merge pull request #16377 from CartoDB/feature/sc-189175/sql-api-oom-…
Shylpx Dec 29, 2021
a3d67c1
Upgrade Carto.js version
moicalcob Jan 3, 2022
64d26cc
Merge branch 'master' into feature/sc-89763/carto-js-teqstream-duplic…
moicalcob Jan 3, 2022
db2c2b6
Merge branch 'master' into feature/sc-197121/coral-reef-admin-builder…
moicalcob Jan 3, 2022
034341c
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Jan 3, 2022
77861c0
Upgrade assets version
moicalcob Jan 3, 2022
1af4ff6
Merge pull request #16384 from CartoDB/feature/sc-89763/carto-js-teqs…
moicalcob Jan 3, 2022
d19cf7d
Merge branch 'master' into feature/sc-197121/coral-reef-admin-builder…
moicalcob Jan 3, 2022
0970ec4
Merge pull request #16381 from CartoDB/feature/sc-197121/coral-reef-a…
moicalcob Jan 3, 2022
2c5958f
Add autoguessing capabilities to ArcGIS connector
moicalcob Jan 14, 2022
507b775
Update NEWS.md
moicalcob Jan 14, 2022
5aa761b
Fix some rubocop ofenses
moicalcob Jan 17, 2022
46b9122
Merge pull request #16385 from CartoDB/feature/sc-201043/denver-mile-…
moicalcob Jan 17, 2022
f14947e
Add new mail sender for internal notifications
moicalcob Feb 8, 2022
1fb5fda
Merge branch 'master' into feature/sc-178449/notify-about-the-limit-o…
moicalcob Feb 8, 2022
a0fc02e
Update analyses schema after giving properly permissions
Shylpx Feb 9, 2022
5274b1c
Update NEWS.md
Shylpx Feb 9, 2022
0456ec0
Merge pull request #16368 from CartoDB/feature/sc-178449/notify-about…
moicalcob Feb 9, 2022
6c1539a
Add 'invitation_token' to signup form (for Google Signup errors)
Shylpx Feb 14, 2022
a1d007e
Update NEWS.md
Shylpx Feb 14, 2022
aba4a2c
Merge branch 'master' into feature/sc-207379/sounddiplomacy-account-u…
Shylpx Feb 15, 2022
4aca21c
Merge pull request #16391 from CartoDB/feature/sc-206349/instacart-ad…
Shylpx Feb 15, 2022
76cd08e
Merge branch 'master' into feature/sc-207379/sounddiplomacy-account-u…
Shylpx Feb 15, 2022
8bd0b09
Merge pull request #16390 from CartoDB/feature/sc-207379/sounddiploma…
Shylpx Feb 15, 2022
fcf3bce
Bump assets version from v280 to v286
aitorch Mar 3, 2022
4ff467e
Update Changelog
aitorch Mar 3, 2022
0ada566
package version lock
aitorch Mar 3, 2022
afc8c89
Merge pull request #16399 from CartoDB/bump-assets-from-280-to-286
Mar 3, 2022
c5d692a
Add notification about new platform and disable current banner
Shylpx Apr 4, 2022
2e1dc0a
Update NEWS.md and bump assets version
Shylpx Apr 4, 2022
f014f27
Fix CI and hound issues
Shylpx Apr 4, 2022
d8b1434
Update submodules to https
Shylpx Apr 4, 2022
d38a9ca
Fix typo
Shylpx Apr 4, 2022
6137103
Fix tests
Shylpx Apr 4, 2022
9fc0464
Updating github URLs to meet github security standards
Apr 5, 2022
913dabf
Merge branch 'updating_github_submodule_urls_to_meet_gh_security_stan…
Shylpx Apr 5, 2022
98537f9
Updating carto.js submodule to v3.15.17
Apr 5, 2022
9de991b
Using https for self main repository url
Apr 5, 2022
7bb8455
Merge pull request #16407 from CartoDB/updating_github_submodule_urls…
Apr 5, 2022
63d84ff
Merge branch 'master' into feature/sc-220191/notification-about-data-…
Shylpx Apr 5, 2022
c3ecb05
Merge pull request #16405 from CartoDB/feature/sc-220191/notification…
Shylpx Apr 5, 2022
36e6697
Reverse analysis order
Shylpx Apr 15, 2022
c66b18e
Update NEWS.md and package.json
Shylpx Apr 15, 2022
5fe699d
Update default value
Shylpx Apr 15, 2022
b002945
Fix tests
Shylpx Apr 18, 2022
9e9b653
Merge pull request #16412 from CartoDB/feature/sc-188915/avoid-recrea…
Shylpx Apr 19, 2022
b940abb
Update regular expression to allow Firefox/100.0
Shylpx May 9, 2022
d7c9098
Update NEWS.md
Shylpx May 9, 2022
9a29175
Merge pull request #16415 from CartoDB/bug/sc-228649/ecfiber-website-…
Shylpx May 9, 2022
5c685e9
Update banner
Shylpx Jun 13, 2022
0675937
Update NEWS.md
Shylpx Jun 13, 2022
54e0751
Fix typo
Shylpx Jun 13, 2022
e21a722
Log exception info
moicalcob Jun 13, 2022
3f042a5
Delete nil values in array
moicalcob Jun 13, 2022
52e577f
Update news
moicalcob Jun 13, 2022
31df61b
Make button non-closable
Shylpx Jun 13, 2022
7f1cd25
bug/sc-229113/all-users-add-fixed-banner-on-the-top-of
Shylpx Jun 13, 2022
e92083c
Update version
Shylpx Jun 13, 2022
10b10ba
Increase PR check timeouts
moicalcob Jun 14, 2022
dc58453
Increase CI timeout
moicalcob Jun 14, 2022
a3bb145
Merge pull request #16421 from CartoDB/feature/sc-226767/build-dedica…
moicalcob Jun 14, 2022
0bba5f0
Merge branch 'master' into bug/sc-229113/all-users-add-fixed-banner-o…
Shylpx Jun 14, 2022
64bc263
Merge pull request #16420 from CartoDB/bug/sc-229113/all-users-add-fi…
Shylpx Jun 14, 2022
0427b99
Avoid deleting a user if it has shared entities
Shylpx Jul 4, 2022
4c73412
Update NEWS.md and try to fix Rubocop
Shylpx Jul 4, 2022
aa105d2
Install rubocop-ast in advance
Shylpx Jul 4, 2022
5aacecd
Install rubocop-rails in advance
Shylpx Jul 4, 2022
a418114
Try another version
Shylpx Jul 4, 2022
c20e0ec
Try again
Shylpx Jul 4, 2022
30b4f93
Reset config
Shylpx Jul 4, 2022
9a29800
Removing the full path on the canonical tag for urls with filter para…
bdiniscia Jul 7, 2022
67aeb9b
Updated News and package.json
bdiniscia Jul 7, 2022
3bf17af
Merge pull request #16426 from CartoDB/feature/ch240818/canonical-url…
bdiniscia Jul 7, 2022
21d1dd3
Handle timeout swaping tables during sync process
moicalcob Jul 26, 2022
5e06df7
Update NEWS.md
moicalcob Jul 26, 2022
ae6c629
Merge pull request #16430 from CartoDB/bug/sc-246788/cc2-sync-failing…
moicalcob Jul 26, 2022
081212c
Add AUTODETECT_SIZE_LIMIT option when importing CSVs
moicalcob Jul 29, 2022
980cc0f
Update NEWS.md
moicalcob Jul 29, 2022
6784746
Merge pull request #16431 from CartoDB/bug/sc-240023/wscc-carto-4-car…
moicalcob Aug 1, 2022
b60ecc3
Log pg_locks when there is a timeout during a sync table importation
moicalcob Aug 1, 2022
3c64349
Merge pull request #16432 from CartoDB/bug/sc-246788/cc2-sync-failing…
moicalcob Aug 1, 2022
3c72971
Terminate locking queries when swapping tables during a synchronization
moicalcob Aug 2, 2022
2a8c5ca
Log locking queries
moicalcob Aug 2, 2022
2f80740
Merge branch 'master' into bug/sc-246788/cc2-sync-failing-randomly-3
moicalcob Aug 2, 2022
832beca
Merge pull request #16433 from CartoDB/bug/sc-246788/cc2-sync-failing…
moicalcob Aug 3, 2022
d9ef0bc
increase cloud build timeout (#16437)
alberhander Sep 12, 2022
a823924
Merge branch 'master' into bug/sc-232062/check-shared-entities-when-u…
Shylpx Nov 10, 2022
1f25b5f
Merge pull request #16424 from CartoDB/bug/sc-232062/check-shared-ent…
Shylpx Nov 10, 2022
7a713be
Add deprecation notice in docs
moicalcob Dec 16, 2022
6d6c856
Fix readthedocs sphinx open source documentation
Jan 2, 2023
a98c59f
Dummy commit
Jan 2, 2023
3e6d1b1
Dummy change
moicalcob Jan 3, 2023
ad19924
Merge pull request #16446 from CartoDB/feature/sc-277742/add-deprecat…
moicalcob Jan 3, 2023
a3a9693
Merge branch 'master' into fix_readthedocs_docs
Jan 3, 2023
6b9010e
Change popup style
moicalcob Jan 3, 2023
9518ec6
Merge pull request #16447 from CartoDB/fix_readthedocs_docs
moicalcob Jan 3, 2023
0af73ea
Add GeoParquet link to Data Catalog (#16453)
Josmorsot Sep 15, 2023
37f6003
Add pre/postinstall scripts to prevent 'npm install' command from fai…
jmgaya May 15, 2024
3ead396
Use npm config instead of GIT global config
jmgaya May 15, 2024
c3b9e5b
Revert "Use npm config instead of GIT global config"
jmgaya May 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix some rubocop ofenses
  • Loading branch information
moicalcob committed Jan 17, 2022
commit 5aa761b6ed13aef12fb46860cb3c51a540044e8f
20 changes: 10 additions & 10 deletions services/importer/lib/importer/arcgis_autoguessing.rb
Original file line number Diff line number Diff line change
@@ -5,12 +5,12 @@ module CartoDB
module Importer2
class ArcGISAutoguessing

def initialize(db, schema_name, table_name, fields_metadata)
@db = db
@schema_name = schema_name
@table_name = table_name
@fields_metadata = fields_metadata
end
def initialize(db, schema_name, table_name, fields_metadata)
@db = db
@schema_name = schema_name
@table_name = table_name
@fields_metadata = fields_metadata
end

def run
autoguess_dates
@@ -19,10 +19,10 @@ def run
def autoguess_dates
date_fields = @fields_metadata.select { |field| field['type'] == 'esriFieldTypeDate' }
date_fields.each do |field|
@db.run(%Q{
ALTER TABLE #{@schema_name}.#{@table_name} ALTER COLUMN #{field['name'].downcase} TYPE DATE
using to_timestamp(cast(#{field['name'].downcase}::bigint/1000 as bigint))::date
})
@db.run(%{
ALTER TABLE #{@schema_name}.#{@table_name} ALTER COLUMN #{field['name'].downcase} TYPE DATE
using to_timestamp(cast(#{field['name'].downcase}::bigint/1000 as bigint))::date
})
end
end

41 changes: 23 additions & 18 deletions services/importer/lib/importer/loader.rb
Original file line number Diff line number Diff line change
@@ -93,13 +93,13 @@ def streamed_run_continue(new_source_file)
run_ogr2ogr(append_mode=true)
end

def streamed_run_finish(post_import_handler_instance=nil, datasource_name)
def streamed_run_finish(post_import_handler_instance = nil, datasource_name = nil)
@post_import_handler = post_import_handler_instance

post_ogr2ogr_tasks(datasource_name)
end

def post_ogr2ogr_tasks(datasource_name)
def post_ogr2ogr_tasks(datasource_name = nil)
georeferencer.mark_as_from_geojson_with_transform if post_import_handler.has_transform_geojson_geom_column?

job.log 'Georeferencing...'
@@ -108,29 +108,27 @@ def post_ogr2ogr_tasks(datasource_name)

if post_import_handler.has_fix_geometries_task?
job.log 'Fixing geometry...'
# At this point the_geom column is renamed
begin
GeometryFixer.new(job.db, job.table_name, SCHEMA, 'the_geom', job).run
rescue StandardError => e
raise e unless statement_timeout?(e.to_s)
# Ignore timeouts in query batcher
log_warning(exception: e, message: 'Could not fix geometries during import')
job.log "Error fixing geometries during import, skipped (#{e.message})"
end
fix_geometries(job)
end

# If autoguessing is enabled, we try it on arcgis data
if datasource_name == 'arcgis' && options[:ogr2ogr_csv_guessing]
job.log 'Autoguessing ArcGIS data types...'
file = File.open @source_file.fullpath
file_content = JSON.load file
ArcGISAutoguessing.new(job.db, SCHEMA, job.table_name, file_content['fields']).run
end
autoguessing_on_arcgis_import if datasource_name == 'arcgis' && options[:ogr2ogr_csv_guessing]
rescue StandardError => e
raise CartoDB::Datasources::InvalidInputDataError.new(e.to_s, ERRORS_MAP[CartoDB::Datasources::InvalidInputDataError]) unless statement_timeout?(e.to_s)
raise StatementTimeoutError.new(e.to_s, ERRORS_MAP[CartoDB::Importer2::StatementTimeoutError])
end

def fix_geometries(job)
# At this point the_geom column is renamed
GeometryFixer.new(job.db, job.table_name, SCHEMA, 'the_geom', job).run
rescue StandardError => e
raise e unless statement_timeout?(e.to_s)

# Ignore timeouts in query batcher
log_warning(exception: e, message: 'Could not fix geometries during import')
job.log "Error fixing geometries during import, skipped (#{e.message})"
end

def normalize
converted_filepath = normalizers_for(source_file.extension)
.inject(source_file.fullpath) { |filepath, normalizer_klass|
@@ -253,7 +251,7 @@ def additional_support_tables

attr_accessor :source_file, :options


private

attr_writer :ogr2ogr, :georeferencer
@@ -404,6 +402,13 @@ def disable_autoguessing_on_wrong_column(filepath, command_output)
csv_content[line][column] = "\"#{csv_content[line][column]}\""
File.open(filepath, 'w') { |file| file.puts csv_content.to_s }
end

def autoguessing_on_arcgis_import
job.log 'Autoguessing ArcGIS data types...'
file = File.read(@source_file.fullpath)
file_content = JSON.parse(file)
ArcGISAutoguessing.new(job.db, SCHEMA, job.table_name, file_content['fields']).run
end
end
end
end