Migrate metadata and variable parsing from snowex_db #6
Comments
…ible. Isolate each function and make room to expand specific campaign logic. Separate the idea of campaign, variable, and profiles
|
I have a design suggestion before we start moving all the parsing/mapping logic from the DB to insitupy. Looking at the PR #7 I think there would be value in keeping SnowEx specific mappings with the DB and making insitupy more of a Swiss army knife that knows how to parse CSV at the lowest level. Think all the logic that is under insitupy/campaigns/snowex could justifiably life with the snowex_db since it will depend on insitupy. Should we have a call to share/exchange some design ideas? |
|
Hey Joe, I think in general that makes sense. A lot of the more nuanced variables should move over. One of the hopes with insitupy is that someone could pick up single snowex pit profiles without relying on the snowex_db repo, so from that perspective I would argue for some of the common variables to stay within insitupy |
|
I did not think of that use case and a valid point, Micah. For our own sake, wouldn't we want to encourage people to use the DB instead of manually parsing and handling SnowEx CSV files themselves? For the described case, we could add a feature and document in snowex_db how to parse SnowEx CSV files and return as a GeoPandas dataframe. You could argue that a dataframe is a "light DB" and just in memory as they describe here What do you think? |
|
I'm kind of viewing insitupy as the hook for the database. Like 'here is an easy way to read in a file or two, but when you want to access a lot of data, including easy ways to compare it - here is this amazing database we built". I also think it's a nice built in example case of how people can extend insitupy for their own data files |
* jot down some notes about splitting variables and metadata * Start moving variables to the right location * comment * Issue #6 - reorganize this package to make it cleaner and more extinsible. Isolate each function and make room to expand specific campaign logic. Separate the idea of campaign, variable, and profiles * Issue #6 - lots of todos * Going after better mapping, splitting of variables, and collection work * Fixing tests * Getting through more testing * Fixing campaign test * moving over from snowex_db * Some additions to metadata * More changes * Rename metadata attributes, add timezone as kwarg for parsing * Use the configured profile data class * We need theh option to add comments * Make sure we pass the header sep in * Add the ability to have an empty profile that still contains metadata * Metadata variables auto remap please * Make sure metadata keys are checked against expected values * More metadata logic * Pass in overrides for site name and campaign name. Allow ignoring profiles that are meant to be ignored * Ability to find units from the columns * return the right variable for unit * metadata logic * Fixing tests * Fixing tests * Fixing flake8 * Clean up location parsing
Description
snowex_db handles lots of parsing of variables and metadata. Begin new structure here to house the variables and metadata for the files!
Associated conversation SnowEx/snowex_db#37
The text was updated successfully, but these errors were encountered: