Merge branch 'master' into population_estimate_by_asr

scripts/us_census/pep/pep_by_srh/README.md

@@ -5,9 +5,13 @@
 This import covers US Census Population estimate by SRH (Sex, Race and Hispanic / Origin).
 Data is covered for 
 
-    1990 - 2020     County
-    1980 - 2020     State
-    1980 - 2020     National
+    1990 - latest_data     County
+    1980 - latest_data     State
+    1980 - latest_data     National
+
+"latest_data" refers to the most recent population estimates for counties, states, and the nation based on the US Census Bureau's ACS Redistricting Data Summary Files.
+The provided URL points you to the source for downloading relevant data files "https://www2.census.gov/programs-surveys/popest/datasets/".
+
 
 Properties Covered - 
 
@@ -37,21 +41,43 @@ Origin
 
 ## Scripts
 ```
-download.sh
+process.py
 ```
 Above scipt creates necessary folders required for processing files and download's US 
-Census PEP files to folder download_files.
+Census PEP files to folder input_files. And reads downloaded files from input_files folder, prepares consolidated 
+(National, State, County) output files for as-is (population_estimate_by_srh.csv) and aggregated (population_estimate_by_srh_agg.csv) in output_files folder
 
+### Additional Notes
 
-```
-process.py
-```
-Reads downloaded files from download_files folder, prepares consolidated 
-(National, State, County) output files for as-is (population_estimate_by_srh.csv) and aggregated (population_estimate_by_srh_agg.csv) in output_files folder
+We have modified the script for Automation setup, now it's not required to add the future URL's in input_url.jason file. Code will take care of future URL formation automatically.
 
+### Import Procedure
 
+The below command will run process.py and generate output csv, mcf and tmcf. 
+`python3 script/us_census/pep/pep_by_srh/process.py`
+
+Execute the 'process.py' script by using the following commands:
+
+  - if you want to perform "download and process", run the below command:
+
+        `python3 process.py`
+
+  - if you want to perform "only process", run the below command:
+
+        `python3 process.py --mode=process`
+        
+  - if you want to perform "only download", run the below command:
+
+        `python3 process.py --mode=download`
+
 ```
 generate_mcf.py
 generate_tmcf.py
 ```
 Generates mcf and tmcf files for both 'as-is' data processing and 'aggregated' data processing
+
+### New Implentation:
+- [Updated the script on October 30, 2024]
+- Downloading input files is now integrated into process.py, eliminating the need to run the separate download.sh script. 
+- All source file URLs, including future URLs adhering to the same structure, are centrally managed in the input_url.json file.
+- All input files required for processing should be stored within the designated "input_files" folder.

scripts/us_census/pep/pep_by_srh/constants.py

@@ -21,7 +21,7 @@
 DC import
 """
 
-DOWNLOAD_DIR = '/download_files/'
+DOWNLOAD_DIR = '/input_files/'
 PROCESS_AS_IS_DIR = '/process_files/as_is/'
 PROCESS_AGG_DIR = '/process_files/agg/'
 OUTPUT_DIR = '/output_files/'
@@ -238,25 +238,34 @@
     '/process_files/agg/1990_2000/county/',
     '/process_files/agg/2000_2010/county/',
     '/process_files/agg/2010_2020/county/',
+    '/process_files/agg/2020_2029/county/',
     '/process_files/as_is/1990_2000/county/',
     '/process_files/as_is/2000_2010/county/',
     '/process_files/as_is/2010_2020/county/',
+    '/process_files/as_is/2020_2029/county/',
     '/process_files/as_is/1980_1990/state/',
     '/process_files/as_is/1990_2000/state/',
     '/process_files/as_is/2000_2010/state/',
     '/process_files/as_is/2010_2020/state/',
+    '/process_files/as_is/2020_2029/state/',
     '/process_files/agg/1980_1990/state/',
     '/process_files/agg/1990_2000/state/',
     '/process_files/agg/2000_2010/state/',
     '/process_files/agg/2010_2020/state/',
+    '/process_files/agg/2020_2029/state/',
     '/process_files/as_is/1980_1990/national/',
     '/process_files/as_is/1990_2000/national/',
     '/process_files/as_is/2000_2010/national/',
     '/process_files/as_is/2010_2020/national/',
+    '/process_files/as_is/2020_2029/national/',
     '/process_files/agg/1980_1990/national/',
     '/process_files/agg/1990_2000/national/',
     '/process_files/agg/2000_2010/national/',
-    '/process_files/agg/2010_2020/national/', '/output_files/'
+    '/process_files/agg/2020_2029/national/',
+    '/process_files/agg/2010_2020/national/', '/output_files/', '/input_files/',
+    '/input_files/1990_2000/county/', '/input_files/2000_2010/county/',
+    '/input_files/2010_2020/county/', '/input_files/2020_2029/county/',
+    '/input_files/1980_1990/state/', '/input_files/1990_2000/state/'
 ]
 
 MCF_SRH = '''Node: dcid:{node_name}