Skip to content

Commit

Permalink
USCensusPEP_By_SexRaceHispanicOrigin changes done (#1113)
Browse files Browse the repository at this point in the history
* pep srh changes done

* pep_srh changes made

* SCHEDULES=scripts/us_census/pep/pep_by_srh:USCensusPEP_By_SexRaceHispanicOrigin

* pep_srh_changes 20241119

* pep srh change 20241120

* prp srh change 20241125

* pep srh 20241126 changes

* pep srh 20241205 changes made

* us_pep_srh 20241209 changes
  • Loading branch information
kurus21 authored Dec 12, 2024
1 parent f88e1c0 commit d2552ef
Show file tree
Hide file tree
Showing 19 changed files with 9,412 additions and 118,046 deletions.
46 changes: 36 additions & 10 deletions scripts/us_census/pep/pep_by_srh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,13 @@
This import covers US Census Population estimate by SRH (Sex, Race and Hispanic / Origin).
Data is covered for

1990 - 2020 County
1980 - 2020 State
1980 - 2020 National
1990 - latest_data County
1980 - latest_data State
1980 - latest_data National

"latest_data" refers to the most recent population estimates for counties, states, and the nation based on the US Census Bureau's ACS Redistricting Data Summary Files.
The provided URL points you to the source for downloading relevant data files "https://www2.census.gov/programs-surveys/popest/datasets/".


Properties Covered -

Expand Down Expand Up @@ -37,21 +41,43 @@ Origin

## Scripts
```
download.sh
process.py
```
Above scipt creates necessary folders required for processing files and download's US
Census PEP files to folder download_files.
Census PEP files to folder input_files. And reads downloaded files from input_files folder, prepares consolidated
(National, State, County) output files for as-is (population_estimate_by_srh.csv) and aggregated (population_estimate_by_srh_agg.csv) in output_files folder

### Additional Notes

```
process.py
```
Reads downloaded files from download_files folder, prepares consolidated
(National, State, County) output files for as-is (population_estimate_by_srh.csv) and aggregated (population_estimate_by_srh_agg.csv) in output_files folder
We have modified the script for Automation setup, now it's not required to add the future URL's in input_url.jason file. Code will take care of future URL formation automatically.

### Import Procedure

The below command will run process.py and generate output csv, mcf and tmcf.
`python3 script/us_census/pep/pep_by_srh/process.py`

Execute the 'process.py' script by using the following commands:

- if you want to perform "download and process", run the below command:

`python3 process.py`

- if you want to perform "only process", run the below command:

`python3 process.py --mode=process`
- if you want to perform "only download", run the below command:

`python3 process.py --mode=download`

```
generate_mcf.py
generate_tmcf.py
```
Generates mcf and tmcf files for both 'as-is' data processing and 'aggregated' data processing

### New Implentation:
- [Updated the script on October 30, 2024]
- Downloading input files is now integrated into process.py, eliminating the need to run the separate download.sh script.
- All source file URLs, including future URLs adhering to the same structure, are centrally managed in the input_url.json file.
- All input files required for processing should be stored within the designated "input_files" folder.
13 changes: 11 additions & 2 deletions scripts/us_census/pep/pep_by_srh/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
DC import
"""

DOWNLOAD_DIR = '/download_files/'
DOWNLOAD_DIR = '/input_files/'
PROCESS_AS_IS_DIR = '/process_files/as_is/'
PROCESS_AGG_DIR = '/process_files/agg/'
OUTPUT_DIR = '/output_files/'
Expand Down Expand Up @@ -238,25 +238,34 @@
'/process_files/agg/1990_2000/county/',
'/process_files/agg/2000_2010/county/',
'/process_files/agg/2010_2020/county/',
'/process_files/agg/2020_2029/county/',
'/process_files/as_is/1990_2000/county/',
'/process_files/as_is/2000_2010/county/',
'/process_files/as_is/2010_2020/county/',
'/process_files/as_is/2020_2029/county/',
'/process_files/as_is/1980_1990/state/',
'/process_files/as_is/1990_2000/state/',
'/process_files/as_is/2000_2010/state/',
'/process_files/as_is/2010_2020/state/',
'/process_files/as_is/2020_2029/state/',
'/process_files/agg/1980_1990/state/',
'/process_files/agg/1990_2000/state/',
'/process_files/agg/2000_2010/state/',
'/process_files/agg/2010_2020/state/',
'/process_files/agg/2020_2029/state/',
'/process_files/as_is/1980_1990/national/',
'/process_files/as_is/1990_2000/national/',
'/process_files/as_is/2000_2010/national/',
'/process_files/as_is/2010_2020/national/',
'/process_files/as_is/2020_2029/national/',
'/process_files/agg/1980_1990/national/',
'/process_files/agg/1990_2000/national/',
'/process_files/agg/2000_2010/national/',
'/process_files/agg/2010_2020/national/', '/output_files/'
'/process_files/agg/2020_2029/national/',
'/process_files/agg/2010_2020/national/', '/output_files/', '/input_files/',
'/input_files/1990_2000/county/', '/input_files/2000_2010/county/',
'/input_files/2010_2020/county/', '/input_files/2020_2029/county/',
'/input_files/1980_1990/state/', '/input_files/1990_2000/state/'
]

MCF_SRH = '''Node: dcid:{node_name}
Expand Down
77 changes: 0 additions & 77 deletions scripts/us_census/pep/pep_by_srh/download.sh

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Loading

0 comments on commit d2552ef

Please sign in to comment.