Update/integration iqair devices #3981
Conversation
📝 WalkthroughWalkthroughThe pull request introduces significant modifications across several files related to data extraction, processing, and validation within the AirQo ETL utilities. Key updates include enhancements to the Changes
Possibly related PRs
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (1)
src/workflows/airqo_etl_utils/airqo_utils.py (1)
228-243: Consider optimizing the data mapping logic.The code handles nested data structures well, but there are a few opportunities for improvement:
- The
Nonedefault indata_mapping.get(key, None)is redundant as.get()returnsNoneby default.- The nested conditionals could be simplified using early returns.
Consider this refactoring:
- if isinstance(entry, dict): - for key, value_data in entry.items(): - target_key = data_mapping.get(key, None) - target_value = None - if isinstance(target_key, dict): - target_value = target_key.get("value") - target_key = target_key.get("key") - - if target_key and target_key not in row_data: - if isinstance(value_data, dict): - extracted_value = AirQoDataUtils._extract_nested_value( - value_data, target_value - ) - else: - extracted_value = value_data - row_data[target_key] = extracted_value + if not isinstance(entry, dict): + return row_data + + for key, value_data in entry.items(): + target_key = data_mapping.get(key) + if not target_key or target_key in row_data: + continue + + if isinstance(target_key, dict): + target_value = target_key.get("value") + target_key = target_key.get("key") + else: + target_value = None + + extracted_value = ( + AirQoDataUtils._extract_nested_value(value_data, target_value) + if isinstance(value_data, dict) + else value_data + ) + row_data[target_key] = extracted_value🧰 Tools
🪛 Ruff (0.8.0)
230-230: Use
data_mapping.get(key)instead ofdata_mapping.get(key, None)Replace
data_mapping.get(key, None)withdata_mapping.get(key)(SIM910)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (5)
src/workflows/airqo_etl_utils/airqo_utils.py(6 hunks)src/workflows/airqo_etl_utils/config.py(4 hunks)src/workflows/airqo_etl_utils/data_validator.py(2 hunks)src/workflows/airqo_etl_utils/schema/bam_raw_measurements.json(5 hunks)src/workflows/dags/airqo_bam_measurements.py(0 hunks)
💤 Files with no reviewable changes (1)
- src/workflows/dags/airqo_bam_measurements.py
🧰 Additional context used
🪛 Ruff (0.8.0)
src/workflows/airqo_etl_utils/airqo_utils.py
230-230: Use data_mapping.get(key) instead of data_mapping.get(key, None)
Replace data_mapping.get(key, None) with data_mapping.get(key)
(SIM910)
437-437: Local variable mapping is assigned to but never used
Remove assignment to unused variable mapping
(F841)
🔇 Additional comments (8)
src/workflows/airqo_etl_utils/schema/bam_raw_measurements.json (1)
7-11: LGTM: Network field addition is well-structured
The new "network" field is properly defined with consistent type and mode, maintaining schema consistency.
src/workflows/airqo_etl_utils/data_validator.py (2)
75-77: LGTM: Improved timestamp formatting with milliseconds handling
The regex pattern properly handles timestamps ending with 'Z' by adding milliseconds if missing, ensuring consistent datetime formatting.
213-213: LGTM: Streamlined device data column selection
The column selection is appropriately focused on essential device metadata fields.
src/workflows/airqo_etl_utils/config.py (3)
171-187: LGTM: Well-structured BAM sensor mapping
The new BAM mapping configuration properly defines all essential sensor parameters with a clear structure.
221-243: LGTM: Comprehensive gas sensor field mapping
The mapping includes all necessary gas sensor parameters with clear documentation for special fields like velocity and satellites.
317-325: LGTM: Consistent device configuration mapping
The device configuration properly references the new mapping structures while maintaining consistency across device types.
src/workflows/airqo_etl_utils/airqo_utils.py (2)
568-570: LGTM: Improved error logging.
The enhanced error logging provides clear context about device fetching failures, which will help with debugging and monitoring.
Line range hint 437-452: Remove unused variable and verify field mapping.
The variable mapping is assigned but never used, which could indicate:
- An incomplete refactoring where the old mapping was replaced with
AIRQO_LOW_COST_GAS_FIELD_MAPPING - A potential bug where the wrong mapping is being used
Please either:
- Remove the unused variable:
- mapping = configuration.AIRQO_LOW_COST_GAS_FIELD_MAPPING- Or verify if this mapping should be used instead of the field columns configuration.
Description
Updates configurations for bam and low cost gas sensors.
Summary by CodeRabbit
Release Notes
New Features
networkin the BAM raw measurements schema.Bug Fixes
Documentation
These changes aim to improve data accuracy, enhance configurability, and streamline data processing workflows for users.