Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to production #4012

Merged
merged 15 commits into from
Dec 5, 2024
Merged

Move to production #4012

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
293aec8
Update AirQo exceedance production image tag to prod-a7e2441e-1733386719
github-actions[bot] Dec 5, 2024
3333d5b
Update auth service staging image tag to stage-73463ae7-1733386665
github-actions[bot] Dec 5, 2024
ff98911
Update KCCA exceedance production image tag to prod-a7e2441e-1733386719
github-actions[bot] Dec 5, 2024
934fcd9
Update auth service production image tag to prod-a7e2441e-1733386719
github-actions[bot] Dec 5, 2024
207a18b
Update workflows prod image tag to prod-a7e2441e-1733386719
github-actions[bot] Dec 5, 2024
b59a0c1
Update predict production image tag to prod-a7e2441e-1733386719
github-actions[bot] Dec 5, 2024
b49a82b
Update spatial production image tag to prod-a7e2441e-1733386719
github-actions[bot] Dec 5, 2024
f599d46
Harmonize data resolutions/frequencies
NicholasTurner23 Dec 5, 2024
f502aaa
Clean up low cost sensor cleaning
NicholasTurner23 Dec 5, 2024
13fd5fc
Remove old implementation of extracting device data
NicholasTurner23 Dec 5, 2024
004372e
Tenant clean up
NicholasTurner23 Dec 5, 2024
27f259d
Clean up
NicholasTurner23 Dec 5, 2024
15e753f
Merge branch 'staging' into update/Integration-iqair-devices
NicholasTurner23 Dec 5, 2024
635379e
Merge pull request #4011 from NicholasTurner23/update/Integration-iqa…
Baalmart Dec 5, 2024
d1496e5
Update workflows staging image tag to stage-635379ef-1733407940
github-actions[bot] Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion k8s/auth-service/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 3
image:
repository: eu.gcr.io/airqo-250220/airqo-auth-api
tag: prod-1c7e9a8e-1733346549
tag: prod-a7e2441e-1733386719
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/auth-service/values-stage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 2
image:
repository: eu.gcr.io/airqo-250220/airqo-stage-auth-api
tag: stage-1838aa7a-1733249623
tag: stage-73463ae7-1733386665
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/exceedance/values-prod-airqo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ app:
configmap: env-exceedance-production
image:
repository: eu.gcr.io/airqo-250220/airqo-exceedance-job
tag: prod-1c7e9a8e-1733346549
tag: prod-a7e2441e-1733386719
nameOverride: ''
fullnameOverride: ''
2 changes: 1 addition & 1 deletion k8s/exceedance/values-prod-kcca.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ app:
configmap: env-exceedance-production
image:
repository: eu.gcr.io/airqo-250220/kcca-exceedance-job
tag: prod-1c7e9a8e-1733346549
tag: prod-a7e2441e-1733386719
nameOverride: ''
fullnameOverride: ''
2 changes: 1 addition & 1 deletion k8s/predict/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ images:
predictJob: eu.gcr.io/airqo-250220/airqo-predict-job
trainJob: eu.gcr.io/airqo-250220/airqo-train-job
predictPlaces: eu.gcr.io/airqo-250220/airqo-predict-places-air-quality
tag: prod-1c7e9a8e-1733346549
tag: prod-a7e2441e-1733386719
api:
name: airqo-prediction-api
label: prediction-api
Expand Down
2 changes: 1 addition & 1 deletion k8s/spatial/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 3
image:
repository: eu.gcr.io/airqo-250220/airqo-spatial-api
tag: prod-1c7e9a8e-1733346549
tag: prod-a7e2441e-1733386719
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/workflows/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ images:
initContainer: eu.gcr.io/airqo-250220/airqo-workflows-xcom
redisContainer: eu.gcr.io/airqo-250220/airqo-redis
containers: eu.gcr.io/airqo-250220/airqo-workflows
tag: prod-1c7e9a8e-1733346549
tag: prod-a7e2441e-1733386719
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/workflows/values-stage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ images:
initContainer: eu.gcr.io/airqo-250220/airqo-stage-workflows-xcom
redisContainer: eu.gcr.io/airqo-250220/airqo-stage-redis
containers: eu.gcr.io/airqo-250220/airqo-stage-workflows
tag: stage-76ff52fe-1733336066
tag: stage-635379ef-1733407940
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
6 changes: 4 additions & 2 deletions src/workflows/airqo_etl_utils/airnow_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ def query_bam_data(

@staticmethod
def extract_bam_data(start_date_time: str, end_date_time: str) -> pd.DataFrame:
tenants = AirQoApi().get_tenants(DataSource.AIRNOW)
# TODO Update if being used.
tenants = AirQoApi().get_networks(DataSource.AIRNOW)
bam_data = pd.DataFrame()
dates = Utils.query_dates_array(
start_date_time=start_date_time,
Expand All @@ -65,11 +66,12 @@ def extract_bam_data(start_date_time: str, end_date_time: str) -> pd.DataFrame:
network_data = pd.DataFrame()

for start, end in dates:
continue
query_data = AirnowDataUtils.query_bam_data(
api_key=network_api_key, start_date_time=start, end_date_time=end
)
network_data = pd.concat([network_data, query_data], ignore_index=True)

continue
network_data["tenant"] = tenant["network"]

bam_data = pd.concat([bam_data, network_data], ignore_index=True)
Expand Down
52 changes: 0 additions & 52 deletions src/workflows/airqo_etl_utils/airqo_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -798,58 +798,6 @@ def update_sites(self, updated_sites):
params = {"tenant": str(Tenant.AIRQO), "id": site.pop("site_id")}
self.__request("devices/sites", params, site, "put")

def get_tenants(self, data_source: str) -> List[Dict[str, Any]]:
"""
Retrieve tenants given a data source.

Args:
data_source: The source of the tenant's data.

Returns:
List[Dict[str, Any]]: A list of dictionaries with tenant details.

[
{
"_id": str,
"net_status": str,
"net_email": str,
"net_phoneNumber": int,
"net_category": str,
"net_name": str,
"net_description": str,
"net_website": str,
"net_acronym": str,
"net_api_key": str,
"net_data_source": str,
"createdAt": str,
"net_users": List[Dict[str,Any]],
"net_permissions": List[Dict[str,Any]],
"net_roles": List[Dict[str,Any]],
"net_groups": List[Dict[str,Any]],
"net_departments": List[Dict[str,Any]],
"network_id": str,
"network": str,
"data_source": str,
"api_key": str"
},
]
"""
response = self.__request("users/networks")

return [
{
**network,
**{
"network_id": network.get("_id", None),
"network": network.get("net_name", None),
"data_source": network.get("net_data_source", None),
"api_key": network.get("net_api_key", None),
},
}
for network in response.get("networks", [])
if network.get("net_data_source") == str(data_source)
]

def __request(self, endpoint, params=None, body=None, method="get", base_url=None):
"""
Executes API request and returns the response.
Expand Down
Loading
Loading