USCensusPEP_MonthlyPopulation-code changes for Autorefresh (#1122)

* USCensusPEP_MonthlyPopulation-code changes for Autorefresh

* SCHEDULES=scripts/us_census/pep/monthly_population_estimate:USCensusPEP_MonthlyPopulation

* added download script

* run format check

* resolve pr comment v1

* added raw data

* changes done

* changes done..

scripts/us_census/pep/monthly_population_estimate/README.md

@@ -1,12 +1,12 @@
 # US Census PEP: National Population Count by Residential Status and Military Status
 
 ## About the Dataset
-This dataset has Population Count Estimates for the United States from the year 1980 to 2022 on a monthly basis.
+This dataset has Population Count Estimates for the United States from the year 1980 on a monthly basis till latest year.
 
 The population is categorized by residential status (resident,InArmedForcesOverseas), military status(Civilian,InArmedForces) and a combination of the same. 
 
 ### Download URL
-The data in txt/xls/xlsx formats are downloadable from within https://www2.census.gov/programs-surveys/popest/tables. The actual URLs are listed in file_urls.json.
+The data in txt/xls/xlsx formats are downloadable from within https://www2.census.gov/programs-surveys/popest/tables. The actual URLs are listed in input_url.json.
 
 #### API Output
 These are the attributes that we will use
@@ -44,12 +44,22 @@ Run the test cases
 ```/bin/python3 scripts/us_census/pep/monthly_population_estimate/preprocess_test.py
 ```
 
+### Import Procedure
 
+[Updated the script on November 11, 2024]
+Downloading input files is now integrated into preprocess.py, eliminating the need to run the separate download.sh script. 
+All source file URLs, including future URLs adhering to the same structure, are centrally managed in the input_url.json file.
+All input files required for processing should be stored within the designated "input_files" folder.
 
-### Import Procedure
+### Downloading and Processing Data
 
-The below script make a new folder named as input_data (if not already present) where the download.py script is present and will download the data into this folder.
-`/bin/python3 scripts/us_census/pep/monthly_population_estimate/download.py`
+To perform "download and process", run the below command:
+    python3 preprocess.py 
+Running this command generates input_fles and csv, mcf, tmcf files
 
-The below script will generate csv and mcf files.
-`/bin/python3 scripts/us_census/pep/monthly_population_estimate/preprocess.py`
+If you want to perform "only process", run the below command:
+    python3 preprocess.py --mode=process
+
+If you want to perform "only download", run the below command:
+    python3 preprocess.py --mode=download
+

scripts/us_census/pep/monthly_population_estimate/input_url.json

@@ -0,0 +1,12 @@
+[
+    {
+        "download_path": "https://www2.census.gov/programs-surveys/popest/tables/1990-2000/national/totals/nat-total.txt"
+    },
+    {
+        "download_path": "https://www2.census.gov/programs-surveys/popest/tables/2000-2009/state/totals/na-est2009-01.csv"
+    },
+    {
+        "download_path": "https://www2.census.gov/programs-surveys/popest/tables/2010-2019/national/totals/na-est2019-01.xlsx"
+    }
+
+]

scripts/us_census/pep/monthly_population_estimate/manifest.json

@@ -0,0 +1,22 @@
+{
+    "import_specifications": [
+        {
+            "import_name": "USCensusPEP_MonthlyPopulation",
+            "curator_emails": [
+                "[email protected]"
+            ],
+            "provenance_url": "https://www2.census.gov/programs-surveys/popest/tables/",
+            "provenance_description": "The Census Bureau's Population Estimates Program (PEP) produces estimates of the population for the United States.",
+            "scripts": [
+                "preprocess.py"
+            ],
+            "import_inputs": [
+                {
+                    "template_mcf": "output/USA_Population_Count.tmcf",
+                    "cleaned_csv": "output/USA_Population_Count.csv"
+                }
+            ],
+            "cron_schedule": "0 07 * * 1"
+        }
+    ]
+}