Merge branch 'datacommonsorg:master' into master

requirements.txt

@@ -25,7 +25,7 @@ retry==0.9.2
 shapely==1.8.5
 urllib3==1.26.8
 xarray==0.19.0
-xlrd==1.2.0
+xlrd
 zipp
 beautifulsoup4
 ratelimit

requirements_all.txt

@@ -36,7 +36,7 @@ shapely==1.8.5
 tabula-py
 urllib3==1.26.8
 xarray==0.19.0
-xlrd==1.2.0
+xlrd
 yapf
 zipp
 beautifulsoup4

scripts/us_census/pep/monthly_population_estimate/README.md

@@ -1,12 +1,12 @@
 # US Census PEP: National Population Count by Residential Status and Military Status
 
 ## About the Dataset
-This dataset has Population Count Estimates for the United States from the year 1980 to 2022 on a monthly basis.
+This dataset has Population Count Estimates for the United States from the year 1980 on a monthly basis till latest year.
 
 The population is categorized by residential status (resident,InArmedForcesOverseas), military status(Civilian,InArmedForces) and a combination of the same. 
 
 ### Download URL
-The data in txt/xls/xlsx formats are downloadable from within https://www2.census.gov/programs-surveys/popest/tables. The actual URLs are listed in file_urls.json.
+The data in txt/xls/xlsx formats are downloadable from within https://www2.census.gov/programs-surveys/popest/tables. The actual URLs are listed in input_url.json.
 
 #### API Output
 These are the attributes that we will use
@@ -44,12 +44,22 @@ Run the test cases
 ```/bin/python3 scripts/us_census/pep/monthly_population_estimate/preprocess_test.py
 ```
 
+### Import Procedure
 
+[Updated the script on November 11, 2024]
+Downloading input files is now integrated into preprocess.py, eliminating the need to run the separate download.sh script. 
+All source file URLs, including future URLs adhering to the same structure, are centrally managed in the input_url.json file.
+All input files required for processing should be stored within the designated "input_files" folder.
 
-### Import Procedure
+### Downloading and Processing Data
 
-The below script make a new folder named as input_data (if not already present) where the download.py script is present and will download the data into this folder.
-`/bin/python3 scripts/us_census/pep/monthly_population_estimate/download.py`
+To perform "download and process", run the below command:
+    python3 preprocess.py 
+Running this command generates input_fles and csv, mcf, tmcf files
 
-The below script will generate csv and mcf files.
-`/bin/python3 scripts/us_census/pep/monthly_population_estimate/preprocess.py`
+If you want to perform "only process", run the below command:
+    python3 preprocess.py --mode=process
+
+If you want to perform "only download", run the below command:
+    python3 preprocess.py --mode=download
+

scripts/us_census/pep/monthly_population_estimate/input_url.json

@@ -0,0 +1,12 @@
+[
+    {
+        "download_path": "https://www2.census.gov/programs-surveys/popest/tables/1990-2000/national/totals/nat-total.txt"
+    },
+    {
+        "download_path": "https://www2.census.gov/programs-surveys/popest/tables/2000-2009/state/totals/na-est2009-01.csv"
+    },
+    {
+        "download_path": "https://www2.census.gov/programs-surveys/popest/tables/2010-2019/national/totals/na-est2019-01.xlsx"
+    }
+
+]