Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline optimization (Sprint 34–36) #105

Merged
merged 24 commits into from
Jun 14, 2021
Merged

Pipeline optimization (Sprint 34–36) #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
dfeb9e9
add_data: Use jq to simplify code
anthonyfok May 6, 2021
50af463
add_data: Add fetch_psra_csv_from_model and merge_csv functions
anthonyfok May 20, 2021
4fda1d5
add_data: New fetch_csv_xz function
anthonyfok May 20, 2021
ee3c8ad
add_data: Add set_synchronous_commit() for speed
anthonyfok May 20, 2021
6463dba
add_data: Add LOG and RUN functions
anthonyfok May 20, 2021
7e97869
add_data: Speed up file writes with eatmydata
anthonyfok May 21, 2021
38d59b9
add_data: Fetch pointers of CSV files for "oid sha256"
anthonyfok May 21, 2021
8491609
add_data: Clone model-factory before the wait for postgres
anthonyfok May 21, 2021
7dc525d
add_data: Do not add quotes in LOG when there is only one argument
anthonyfok Jun 2, 2021
7b0b921
add_data: Add INFO, WARN and ERROR functions for logging
anthonyfok Jun 2, 2021
8f6467e
add_data: Convert more echos and comments to LOG() etc.
anthonyfok Jun 2, 2021
12dbd02
add_data: Allow dry run for testing and debugging
anthonyfok Jun 2, 2021
313776f
Make add_data.sh and postgis/*.sh ShellCheck clean
anthonyfok Jun 2, 2021
3116de2
add_data: Hide secrets such as POSTGRES_PASS and ES_PASS in LOG()
anthonyfok Jun 3, 2021
ffd2078
add_data: Read variables from environment instead of command-line
anthonyfok Jun 3, 2021
15c2d18
add_data: Move major steps info their own functions
anthonyfok Jun 4, 2021
de8375c
add_data: Use RUN for all the main steps too
anthonyfok Jun 7, 2021
e114c19
add_data: Fix doc regarding OpenQuake CSV header stripping
anthonyfok Jun 7, 2021
60e3e57
add_data: Fix section comment and code indentation
anthonyfok Jun 10, 2021
1bfe87d
add_data: Revert change to pg_isready port setting
anthonyfok Jun 10, 2021
21fb875
add_data: Fix erroneous CSV generated by merge_csv
anthonyfok Jun 10, 2021
fd9ae1d
add_data: Do not fail if /usr/bin/time does not exist
anthonyfok Jun 11, 2021
29dec2c
add_data: Add missing YT to PT_LIST
anthonyfok Jun 12, 2021
3b9c39c
add_data: Remove double quotes around ${ES_CREDENTIALS:-}
anthonyfok Jun 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion postgis/create_db.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ CREATE DATABASE ${DB_NAME} OWNER ${POSTGRES_USER};
EOSQL

echo "Creating schema..."
psql -d ${DB_NAME} -a -U${POSTGRES_USER} -f /schema.sql
psql -d "${DB_NAME}" -a -U"${POSTGRES_USER}" -f /schema.sql
4 changes: 2 additions & 2 deletions postgis/update_db.aws.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
set -e

echo "Creating schema..."
psql -h ${POSTGRES_HOST} -d ${DB_NAME} -a -U ${POSTGRES_USER} -f /schema.sql
psql -h "${POSTGRES_HOST}" -d "${DB_NAME}" -a -U "${POSTGRES_USER}" -f /schema.sql

echo "Creating extensions..."
psql -h ${POSTGRES_HOST} -d ${DB_NAME} -a -U ${POSTGRES_USER} -f /extensions.sql
psql -h "${POSTGRES_HOST}" -d "${DB_NAME}" -a -U "${POSTGRES_USER}" -f /extensions.sql
4 changes: 2 additions & 2 deletions python/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,6 @@ COPY . .
#RUN chmod +x docker-entrypoint.sh && chmod +x add_data.sh
RUN chmod +x add_data.sh && \
apt-get update && \
apt-get install -y jq time
apt-get install -y eatmydata git git-lfs jq time
#ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
CMD ./add_data.sh ${POSTGRES_USER} ${POSTGRES_PASS} ${POSTGRES_PORT} ${DB_NAME} ${POSTGRES_HOST} ${ES_ENDPOINT} ${ES_USER} ${ES_PASS} ${KIBANA_ENDPOINT}
CMD ./add_data.sh
Loading