Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add EIA Provider & Weekly Petroleum Status Report #6693

Merged
merged 38 commits into from
Oct 31, 2024

Conversation

deeleeramone
Copy link
Contributor

@deeleeramone deeleeramone commented Sep 25, 2024

  1. Why?:

The EIA's Weekly Petroleum Status Report (WPSR) and Short Term Energy Outlook (STEO) are closely followed events for anyone watching the fossil fuels.

This PR represents the first integrations of open data available from the EIA.

  1. What?:

The WPSR is comprised of thirteen (excludes discontinued series) high-level categories with each containing a subset of tables. Data is from the static Excel files published here, and each file represents a single category.

All data from a single category is returned by supplying "all" to the table parameter of the WPSR endpoint.

Tables from the WPSR are returned in a flat format in the same order as presented in the Excel files. The response is suitable for pivot tables and SQL storage.

The STEO dataset is twenty-six tables of near-term projections, which is pulled from the V2 API.

Because of the structure of their API, a unified seriesID lookup similar to FRED or BLS is not possible. The symbols from within STEO tables are all contained to a single path with uniform parameters, but the symbols within WPSR are scattered throughout other segments of the API and cannot be queried in this way.

  1. Impact:

    • This package contains dependencies not included in openbb-core:

      • async-lru = "^2.0.4" (to prevent Excel files from being repeatedly downloaded)
      • openpyxl = "^3.1.5" (for working with Pandas ExcelFile)
      • xlrd = "^2.0.1" (for working with Pandas ExcelFile)
    • New endpoints:

      • obb.commodity.petroleum_status_report (WPSR - no API key required)
      • obb.commodity.short_term_energy_outlook (STEO - API key required)
    • ALRU cache (async drop-in replacement to functools.lru_cache) is used for the WPSR files that retains each "category" requested for the duration of the Python session. Setting use_cache as False will invalidate the cache and download the file again.

    • The table parameter has defined choices that represent all possible tables from all categories.

      • For WPSR, a custom validator checks this parameter against the category and the list of tables within that category; it will raise with a message indicating which choices are valid for the given category. Multiple items allowed here.
      • For STEO, choices are strings with values, such as "03d", and a map is supplied in the function's docstring.
    • STEO also accepts a symbol input - multiple items allowed - for targeted queries from the API route.

    • The response can be very large tables, 100K - 1M+ rows of data.

      • At an unknown number of rows, the interactive tables in CLI will fail and only display a white screen.
  2. Testing Done:

    • Unit/integration
Screenshot 2024-10-28 at 3 21 19 PM

@deeleeramone deeleeramone added enhancement Enhancement do not merge Label to prevent pull request merge platform OpenBB Platform v4 PRs for v4 labels Sep 25, 2024
@deeleeramone deeleeramone changed the title add eia provider and weekly petroleum status report [Feature] Add EIA Provider & Weekly Petroleum Status Report Sep 25, 2024
@deeleeramone deeleeramone removed the do not merge Label to prevent pull request merge label Oct 19, 2024
Copy link
Contributor

@piiq piiq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left a comment about inclusion of EIA into the main openbb distribution.

Additionally please have a look if any of the integrations with EIA's V2 API I've made here are reusable https://github.com/piiq/openbb-energy/tree/develop/

openbb_platform/openbb/assets/reference.json Show resolved Hide resolved
openbb_platform/pyproject.toml Show resolved Hide resolved
@deeleeramone
Copy link
Contributor Author

Additionally please have a look if any of the integrations with EIA's V2 API I've made here are reusable https://github.com/piiq/openbb-energy/tree/develop/

I was actually going to dive into what you had mapped out already, and I have some of the other routes mapped as well. That represents a bunch of work that would make this a much larger PR, so my intention was to start with this and work on the other pieces.

@piiq piiq enabled auto-merge October 31, 2024 13:45
@piiq piiq added this pull request to the merge queue Oct 31, 2024
Merged via the queue into develop with commit dd42f7a Oct 31, 2024
10 checks passed
@piiq piiq deleted the feature/eia-provider branch October 31, 2024 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking_change enhancement Enhancement platform OpenBB Platform v4 PRs for v4
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants