Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/0.2.0 #103

Merged
merged 56 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
92f9d5f
Use pydantic models, remove Metobs class
mgcth Mar 23, 2024
7d208ca
Add metobs pydantic models
mgcth Mar 23, 2024
fb6885b
Add script generating metobs pydantic models from url
mgcth Mar 23, 2024
a8fcd6a
Update Metobs and integration tests
mgcth Mar 23, 2024
09c5f26
Add lint rules and update pydantic model filed names
mgcth Mar 23, 2024
3be389b
Refactor metobs unit tests
mgcth Mar 24, 2024
b23776c
Add fixture data for metobs unit tests
mgcth Mar 24, 2024
f86614e
Lint and format
mgcth Mar 24, 2024
3437473
Use model for ParameterItems
mgcth Mar 24, 2024
b5acc24
Draft metobs
mgcth Mar 25, 2024
2c83a5d
Simplify metobs unit testing
mgcth Mar 25, 2024
5357a92
Remove banding config and add shapely and pydantic
mgcth Mar 25, 2024
6d93a91
Robustify metobs table parsing and update integration tests
mgcth Mar 25, 2024
b1bd846
Add period station_set unit test, skipping data station_set currently
mgcth Mar 25, 2024
200c6cd
Add Mesan models
mgcth Mar 26, 2024
6a67d3b
Fix dataframe parsing bug
mgcth Mar 27, 2024
6560440
Strang update, in progress
mgcth Mar 28, 2024
a4f7dee
Rename attributes and put condition check in method
mgcth Mar 29, 2024
d05391d
Update pandas to 2.2 and add panderas
mgcth Mar 29, 2024
06138c2
Update strang return types and validate strang dataframes using panderas
mgcth Mar 29, 2024
ad17dd8
Move fixtures and update strang integration tests
mgcth Mar 29, 2024
c46d93f
Partial update of strang unit tests
mgcth Mar 29, 2024
c6a21c2
Passing strang unit tests, need cleaning
mgcth Mar 29, 2024
6f8d997
Fix lint
mgcth Mar 29, 2024
a0fbd74
Update Mesan, in progress
mgcth Mar 29, 2024
375499d
Draft update of mesan and metfcts
mgcth Mar 30, 2024
21e2306
Use parametric types
mgcth Mar 30, 2024
9e5e127
Save pydantic models from smhi api
mgcth Mar 30, 2024
c8a2972
Rename model modules
mgcth Mar 30, 2024
59ae528
Fix types
mgcth Mar 30, 2024
81dc1e0
Update some tests
mgcth Mar 30, 2024
965375a
return empty on request fail
mgcth Mar 30, 2024
0de96a3
Raise on request error instead of empty data
mgcth Mar 30, 2024
8a4253c
Bump ruff in pre-commit
mgcth Mar 30, 2024
541fabd
Constrain mesan and metfcts valid times
mgcth Mar 30, 2024
5aea297
Remove mypy file
mgcth Mar 31, 2024
10602b4
Update metobs tests
mgcth Mar 31, 2024
52d8c58
Update mesan unit tests
mgcth Mar 31, 2024
bbd5f38
Fix strang unit tests and add mesan text files
mgcth Mar 31, 2024
a49cb40
Refactor unit and integration tests, remove doc args and returns from…
mgcth Mar 31, 2024
a627876
Fix intentional bug in strang url
mgcth Mar 31, 2024
5edc303
Ignore import in mypy action
mgcth Mar 31, 2024
f266e06
Simplify mesan integration tests
mgcth Mar 31, 2024
e8a72be
Simplify metobs integration tests
mgcth Mar 31, 2024
7193fe4
Simplify strang unit tests
mgcth Apr 1, 2024
db0ab01
Bump python to 3.11 in actions, add 3.12 to tests
mgcth Apr 1, 2024
9c65b6e
Fix lint
mgcth Apr 1, 2024
970f2a0
Rename strang integration test files
mgcth Apr 1, 2024
db552db
Fix mesan integration test datetime span
mgcth Apr 1, 2024
1291c22
Don't use Python 3.12 in tests
mgcth Apr 1, 2024
91b53c8
Add more attributes to return for each api type
mgcth Apr 1, 2024
60d80cd
Fix metobs unit test
mgcth Apr 1, 2024
c3492bc
Update docs
mgcth Apr 1, 2024
18bf825
Add models to docs
mgcth Apr 1, 2024
ff94feb
Update metobs models
mgcth Apr 5, 2024
8b52adc
Update docs metobs models
mgcth Apr 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .bandit

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/github-action-docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9"]
python-version: ["3.11"]

steps:
- uses: actions/checkout@v2
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/github-action-lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9"]
python-version: ["3.11"]

steps:
- uses: actions/checkout@v2
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/github-action-md-lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9"]
python-version: ["3.11"]

steps:
- uses: actions/checkout@v2
Expand Down
9 changes: 2 additions & 7 deletions .github/workflows/github-action-type.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9"]
python-version: ["3.11"]

steps:
- uses: actions/checkout@v2
Expand All @@ -19,9 +19,4 @@ jobs:
pip install -e ".[type]"
- name: Type checking with mypy
run: >
MYPYPATH=src
mypy
--namespace-packages
--explicit-package-bases
--allow-redefinition
src
mypy src --ignore-missing-imports --exclude=src/smhi/smhi.py
46 changes: 23 additions & 23 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.5
hooks:
- id: ruff
types_or: [ python, pyi, jupyter ]
args: [ --fix ]
- id: ruff-format
types_or: [ python, pyi, jupyter ]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: "586b4f0"
hooks:
- id: mypy
args: [--no-strict-optional, --ignore-missing-imports]
- repo: https://github.com/igorshubovych/markdownlint-cli
rev: v0.32.2
hooks:
- id: markdownlint
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.3.4
hooks:
- id: ruff
types_or: [python, pyi, jupyter]
args: [--fix]
- id: ruff-format
types_or: [python, pyi, jupyter]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: "586b4f0"
hooks:
- id: mypy
args: [--no-strict-optional, --ignore-missing-imports]
- repo: https://github.com/igorshubovych/markdownlint-cli
rev: v0.32.2
hooks:
- id: markdownlint
17 changes: 17 additions & 0 deletions create_pydantic_models.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metobs.smhi.se/api.json --class-name VersionModel --output metobs_versions.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metobs.smhi.se/api/version/1.0.json --class-name ParameterModel --output metobs_parameters.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metobs.smhi.se/api/version/1.0/parameter/1.json --class-name StationModel --output metobs_stations.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metobs.smhi.se/api/version/1.0/parameter/1/station/1.json --class-name PeriodModel --output metobs_periods.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metobs.smhi.se/api/version/1.0/parameter/1/station/1/period/corrected-archive.json --class-name DataModel --output metobs_data.py

datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/parameter.json --class-name DataModel --output mesan_parameters.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/approvedtime.json --class-name DataModel --output mesan_approved.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/validtime.json --class-name DataModel --output mesan_valid.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/geotype/polygon.json --class-name DataModel --output mesan_polygon.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/geotype/multipoint.json --class-name DataModel --output mesan_multipoint.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/geotype/point/lon/16/lat/58/data.json --class-name DataModel --output mesan_data_point.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url "https://opendata-download-metanalys.smhi.se/api/category/mesan2g/version/1/geotype/multipoint/validtime/20240108T060000Z/parameter/t/leveltype/hl/level/2/data.json?with-geo=true" --class-name DataModel --output mesan_data_multipoint.py

datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url "https://opendata-download-metanalys.smhi.se/api/category/strang1g/version/1/geotype/point/lon/16.158/lat/58.5812/parameter/118/data.json?from=2020-02-01&to=2020-02-02" --class-name DataModel --output strang_point.py
datamodel-codegen --output-model-type pydantic_v2.BaseModel --snake-case-field --url "http://opendata-download-metanalys.smhi.se/api/category/strang1g/version/1/geotype/multipoint/validtime/201908/parameter/118/data.json?interval=monthly" --class-name DataModel --output strang_multipoint.py
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ See [https://opendata.smhi.se/apidocs/](https://opendata.smhi.se/apidocs/)
for a complete list of available APIs.
SMHI stands for [Swedish Meteorological and Hydrological Institute](https://www.smhi.se/)
(or in Swedish: Sveriges meteorologiska och hydrologiska institut),
which is a Swedish agency under its parent department: Ministry of Climate and
which is a Swedish agency under its parent department: Ministry of Climate and
Enterprise.

Initially only these four APIs are supported
Expand Down Expand Up @@ -49,7 +49,7 @@ Use this to access data about _forecasts_, i.e. SMHI weather predictions.
## Mesan client

Client to fetch data from meteorological analysis.
Use this to access up to 24 hour predictions of weather parameters.
Use this to access the last 24 hour predictions of weather parameters.
This API is a useful complement to Metobs because the number of weather
stations are limited.

Expand Down
14 changes: 11 additions & 3 deletions docs/mesan-example.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Example of Mesan direct use

Direct usage of `Mesan`. To list approved and valid times to
Direct usage of `Mesan`. Note that `Metfcts` and `Mesan` have identical interfaces.

## Time listing

To list approved and valid times to

```python
from smhi.mesan import Mesan
Expand All @@ -14,6 +18,8 @@ Notice that `approved_time` is the time when the MESAN analysis was updated.
On the other hand, `valid_time` are valid time stamps to fetch data for,
see below.

## Parameters and geographic area

To list available parameters, geographic area as polygon and points do

```python
Expand All @@ -27,10 +33,12 @@ client.get_geo_multipoint(2)

where `get_geo_multipoint` accepts a downsample argument.

## Point and multipoint data

To get data, two methods are available.
`get_point` accepts latitude and longitude arguments.
`get_multipoint` accepts `validtime`, `parameter`,
`leveltype`, `level`, `downsample` arguments.
`leveltype`, `level`, `geo` and `downsample` arguments.
See above to acquire a valid time and parameter.

```python
Expand All @@ -39,7 +47,7 @@ from smhi.mesan import Mesan
client = Mesan()
data = client.get_point(58, 16)
data = client.get_multipoint(
"2022-11-12T23:00:00Z", "t", "hl", 2, 2
"2022-11-12T23:00:00Z", "t", "hl", 2, False, 2
)
```

Expand Down
3 changes: 3 additions & 0 deletions docs/mesan-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Mesan models

::: smhi.models.mesan_model
10 changes: 9 additions & 1 deletion docs/metfcts-example.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Example of Metfcts direct use

Direct usage of `Metfcts`. To list approved and valid times to
Direct usage of `Metfcts`. Note that `Mesan` and `Metfcts` have identical interfaces.

## Time listing

To list approved and valid times to

```python
from smhi.metfcts import Metfcts
Expand All @@ -14,6 +18,8 @@ Notice that `approved_time` is the time when the Metfcts analysis was updated.
On the other hand, `valid_time` are valid time stamps to fetch data for,
see below.

## Parameters and geographic area

To list available parameters, geographic area as polygon and points do

```python
Expand All @@ -27,6 +33,8 @@ client.get_geo_multipoint(2)

where `get_geo_multipoint` accepts a downsample argument.

## Point and multipoint data

To get data, two methods are available.
`get_point` accepts latitude and longitude arguments.
`get_multipoint` accepts `validtime`, `parameter`,
Expand Down
3 changes: 3 additions & 0 deletions docs/metfcts-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Metfcts models

::: smhi.models.metfcts_model
92 changes: 31 additions & 61 deletions docs/metobs-example.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,79 +2,49 @@

Direct usage of `Metobs`.

## Get data from known parameters

To get data from a known station (found e.g. through exploration of
[https://www.smhi.se/data](https://www.smhi.se/data)) from `Metobs` do

```python
from smhi.metobs import Metobs

client = Metobs()
data = client.get_data_stationset(
1, 192840, "corrected-archive"
)
```

## Get data by inspecting the API

*This will be deprecated in 0.2.0.*
To inspect the API, the following methods are provided
The following example is a common pattern of usage
The `Metobs` clinet consistes of 5 objects
`Versions, Parameters, Stations, Periods, Data`
that building up a chain to get data.

```python
from smhi.metobs import Metobs

client = Metobs()
client.get_parameters()

# list all parameters
client.parameters.data

# get all stations that have data from parameter 1
client.get_stations(1)
from smhi.metobs import Versions, Parameters, Stations, Periods, Data

# list all stations
client.stations.data
versions = Versions() # defaults to type = json, this step can be skipped
versions.data
# show data from this endpoint

# get all periods that have data from station 1
client.get_periods(1)
parameters = Parameters(versions) # can be called by simply Parameters()
parameters.data
# show all available parameters in that API version

# list all periods
client.periods.data
stations = Stations(parameters, 1)
stations.data
# show all available stations for parameter 1

# inspect client state
client.inspect()
periods = Periods(stations, 1)
periods.data
# show all available periods of data for station 1

# get data from parameter 1, station 1 and period corrected-archive
data = client.get_data()
data = Data(periods) # defaults to corrected-archive period
data.df
# show the station data dataframe
```

## Alternative way of using the client
Note that, the last call gives back a `MetobsDataModel` with four
dataframes when a single station is used to fetch data as above.
They are called `station`, `parameter`, `period` and `df`.

Instead of using the `Metobs` client, the objects used by that client can be
used directly. The following example is a recommended pattern of usage
It is posible to fetch a station set, that is, data from many stations for
the last hour as

```python
from smhi.metobs import Versions, Parameters, Stations, Periods, Data

versions = Versions() # defaults to type = json, this step can be skipped
versions.show
# print all available versions

parameters = Parameters(versions) # can be called by simply Parameters()
parameters.show
# print all available parameters in that API version

stations = Stations(parameters, 1)
stations.show
# print all available stations for parameter 1

periods = Periods(stations, 1)
periods.show
# print all available periods of data for station 1
periods = Periods(stations, station_set="all")
periods.data
# show all available periods of data for station set all

data = Data(periods) # defaults to corrected-archive period
data.data_header # data headers
data.data # actual data
data.df
# show
```

in which case only the `parameter` and `df` fields are populated.
3 changes: 3 additions & 0 deletions docs/metobs-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Metobs models

::: smhi.models.metobs_model
4 changes: 2 additions & 2 deletions docs/metobs-reference.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Metobs reference

::: smhi.metobs.Metobs
::: smhi.metobs.BaseMetobs

::: smhi.metobs.BaseLevel
::: smhi.metobs.Versions

::: smhi.metobs.Parameters

Expand Down
Loading
Loading