Merge branch 'master' into eurostat_refresh

README.md

@@ -113,7 +113,7 @@ Install requirements and setup a virtual environment to isolate python developme
 python3 -m venv .env
 source .env/bin/activate
 
-pip3 install -r requirements.txt
+pip3 install -r requirements_all.txt
 ```
 
 ##### Testing
@@ -153,8 +153,8 @@ you import modules and run tests, as below.
 
 ##### Guidelines
 
-*   Any additional package required must be specified in the requirements.txt
-    in the top-level folder. No other requirements.txt files are allowed.
+*   Any additional package required must be specified in the `requirements_all.txt`
+    file in the top-level folder. No other `requirements.txt` files are allowed.
 *   Code must be formatted according to the
     [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
     according to the [yapf formatter](https://github.com/google/yapf).

requirements.txt

@@ -1,42 +1,28 @@
-# Requirements for all Python code in this repo, except for import-automation
+# Requirements for Python scripts in this repo that have automation enabled!
 
 absl-py
 arcgis2geojson
-chembl-webresource-client>=0.10.2
 dataclasses==0.6
 datacommons==1.4.3
-deepdiff==6.3.0
-earthengine-api
-flask_restful==0.3.9
 frozendict==1.2
 func-timeout==4.3.5
 geojson==2.5.0
-geopandas==0.8.1
-geopy
 google-cloud-bigquery
 google-cloud-storage>=2.7.0
 google-cloud-logging==3.4.0
 google-cloud-scheduler==2.10.0
 gspread
 lxml==4.9.1
-matplotlib==3.3.0
-netCDF4==1.6.4
 numpy
-openpyxl==3.0.7
 pandas
 pylint
 pytest
-rasterio
-rdp==0.8
 requests==2.27.1
 retry==0.9.2
-s2sphere==0.2.5
 shapely==1.8.5
-tabula-py
 urllib3==1.26.8
 xarray==0.19.0
 xlrd==1.2.0
-yapf
 zipp
 beautifulsoup4
 ratelimit

requirements_all.txt

@@ -0,0 +1,42 @@
+# Requirements for all Python code in this repo, except for import-automation
+
+absl-py
+arcgis2geojson
+chembl-webresource-client>=0.10.2
+dataclasses==0.6
+datacommons==1.4.3
+deepdiff==6.3.0
+earthengine-api
+flask_restful==0.3.9
+frozendict==1.2
+func-timeout==4.3.5
+geojson==2.5.0
+geopandas==0.8.1
+geopy
+google-cloud-bigquery
+google-cloud-storage>=2.7.0
+google-cloud-logging==3.4.0
+google-cloud-scheduler==2.10.0
+gspread
+lxml==4.9.1
+matplotlib==3.3.0
+netCDF4==1.6.4
+numpy
+openpyxl==3.0.7
+pandas
+pylint
+pytest
+rasterio
+rdp==0.8
+requests==2.27.1
+retry==0.9.2
+s2sphere==0.2.5
+shapely==1.8.5
+tabula-py
+urllib3==1.26.8
+xarray==0.19.0
+xlrd==1.2.0
+yapf
+zipp
+beautifulsoup4
+ratelimit

run_tests.sh

@@ -28,7 +28,7 @@ function setup_python {
   if [[ "$PYTHON_REQUIREMENTS_INSTALLED" = false ]]
   then
     echo "Installing Python requirements"
-    pip3 install -r requirements.txt -q
+    pip3 install -r requirements_all.txt -q
     PYTHON_REQUIREMENTS_INSTALLED=true
   fi
 }

scripts/us_bls/jolts/bls_jolts.py

@@ -18,7 +18,7 @@
 
 Statistical Variables are generated and cleaned CSV is output.
 
-Download the requirements.txt via pip and execute the file with Python 3.
+Download the requirements_all.txt via pip and execute the file with Python 3.
 
 Dataset being processed: https://download.bls.gov/pub/time.series/jt/
 """

scripts/us_census/acs5yr/subject_tables/README.md

@@ -2,7 +2,7 @@
 
 This directory has the files and code used for importing ACS Subject Tables into DataCommons.
 
-Note: Before running the scripts here ensure the packages specified in `data/requirements.txt` is installed.
+Note: Before running the scripts here ensure the packages specified in `data/requirements_all.txt` is installed.
 
 ### Getting Started
 Before, getting started with an import of an ACS Subject Table, it is important to ensure the following are available:

scripts/us_epa/ghgrp/gen_data.sh

@@ -4,7 +4,7 @@
 pushd ../../../
 python3 -m venv .env
 source .env/bin/activate
-pip3 install -r requirements.txt -q
+pip3 install -r requirements_all.txt -q
 popd
 
 # Generate schema to import_data/