Daily validation dash app

[tsmbland]

Creates a dash app to visualise data in tabular and graph form, and manually save to the Validated dataset. Replicates the original Daily Validation Javascript app (http://localhost:8000/validated/daily_validation/).

The new app is a self-contained page accessible at http://localhost:8000/validated/daily_validation_dev/

For now, the data filters are specified in the code, although eventually we will want to include these as drop downs on the page (either through the existing django forms or new dash objects).

---

I'm keeping this as a master branch for #121, and merging PRs into this branch as I add new features.

PRs already merged:
#143 Making a start on the tables, showing the appropriate data, conditional styling
#145 Callbacks for adding new data and saving to validated
#151 Arranging tables into tabs and callback for opening the detailed view table
#154 Cosmetic improvements to the plot, and radio selector for choosing which information to show
#158 Date picker for choosing date for detail table
#159 Adds data filters to the top (replacing django form functionality)

---

Notes/remaining issues:

• Daily data and detail data doesn't seem to match up. For example, the Max of maxs. for 2023-03-14 is shown as 50.34, but looking at the detailed view for that day the highest maximum value is 17.42. This is the same in the original JS app. Am I just misunderstanding something?
• The buttons are sometimes very laggy, especially the 'Save to Validated' and 'Reset Validated' buttons which require calls to the database. (This seems to vary a lot though, and could relate to developing the code while it's running.)
• Related to the above, the original app brings up loading messages while the button callbacks are running. This would be a good thing to do here given how long some of the callbacks take, but I haven't yet found an easy way to do this. Done
• I'm unsure what the 'Selected' component of the data is meant to represent. Is this just to represent which measurements are selected in the tables? If so, why does Selected deviate from Measurement in the plot (see 30th March)
• The plot only updates when buttons are pressed, so selecting rows in the tables won't change the Selected dots in the plot until a button is pressed. This is the same as the JS app.
• Why is there no data for March 31st?
• Why do days run from 5am-5am? Is this a timezone thing?
• Probably related to the above two, but opening detail view for April 1st gives 'No Rows To Show' (same in the JS app)
• Fonts: I'm defining the default font at the top as a string, and then passing this as a style attribute to every text object. This works, but presumably there's a way of globally defining the default font for the app? Edit: fixed using stylesheet
• Layout: I like being able to view the tables and the plot at the same time, but ideally they would both fit on the screen without having to scroll. I was having issues defining the height of individual divs so probably doing something wrong. I also don't know how to deal with potentially different screen sizes.
• There's a load of white space at the bottom which I don't know how to get rid of.
• We will need to add authentication somewhere so it only works for logged-in users.
• Filters at the top won't show an error message if invalid filters are selected and just won't update the tables/plot. However, this should be fixed when we address Reports should show only valid options to chose #108
• In the js app the filters automatically fill in the min/max values once the station, variable and dates are selected, however I can't find where in the code this is done, so I haven't been able to replicate it here.

[dalonsoa]

I answer here some of the remaingin quesitons:

Daily data and detail data doesn't seem to match up. For example, the Max of maxs. for 2023-03-14 is shown as 50.34, but looking at the detailed view for that day the highest maximum value is 17.42. This is the same in the original JS app. Am I just misunderstanding something?

It might be that the functions doing the post-processing after querying the database are already doing some filtering. A way of checking this would be manually have a look at the actual data as soon as it gets our of the database, before any processing.

The buttons are sometimes very laggy, especially the 'Save to Validated' and 'Reset Validated' buttons which require calls to the database. (This seems to vary a lot though, and could relate to developing the code while it's running.)

The above processing is very slow and dodgy, so I'm not surprised it is slow. Hopefully, it can be greatly simplified when we use the new model.

Related to the above, the original app brings up loading messages while the button callbacks are running. This would be a good thing to do here given how long some of the callbacks take, but I haven't yet found an easy way to do this.

No idea. Maybe @AdrianDAlessandro has a feedback on this?

I'm unsure what the 'Selected' component of the data is meant to represent. Is this just to represent which measurements are selected in the tables? If so, why does Selected deviate from Measurement in the plot (see 30th March).

I've always wondered the same thing. In principle, that should just mean that is active (i.e., not being discarded after validation), but clearly there's something else.

The plot only updates when buttons are pressed, so selecting rows in the tables won't change the Selected dots in the plot until a button is pressed. This is the same as the JS app.

Sure, but is that the behavior we want? What do you think?

Why is there no data for March 31st?

I guess that because the initial data loaded in the database, which is just some toy data used for development, does not go any further.

Why do days run from 5am-5am? Is this a timezone thing?

Most likely. The stations are in Peru, while you are in UK. The datetime objects in the database are timezone aware so they show you the information translated to your timezone.

Probably related to the above two, but opening detail view for April 1st gives 'No Rows To Show' (same in the JS app)

Most likely, that is the case.

Fonts: I'm defining the default font at the top as a string, and then passing this as a style attribute to every text object. This works, but presumably there's a way of globally defining the default font for the app?
Layout: I like being able to view the tables and the plot at the same time, but ideally they would both fit on the screen without having to scroll. I was having issues defining the height of individual divs so probably doing something wrong. I also don't know how to deal with potentially different screen sizes.
There's a load of white space at the bottom which I don't know how to get rid of.

No idea. @AdrianDAlessandro might know more about these L&F aspects of dash apps.

We will need to add authentication somewhere so it only works for logged-in users.

We will deal with this as some point. Clearly it can be done (remember the Liionsden app I showed you), so let's not worry about this, yet. Open an issue to tackle it in a future iteration.

[dalonsoa]

It's looking great! I just find a bit odd the way of selecting the detailed information about a day. Cannot be chosen based on a dropdown with the available days, a date selector or by clicking on the relevant row?

[tsmbland]

Cannot be chosen based on a dropdown with the available days, a date selector or by clicking on the relevant row?

Yeah good point. I think I'll go with a date selector as clicking rows might conflict with the checkboxes and adding buttons to each row (like the JS app) seems a bit complicated in Dash

Edit: This is now done

[tsmbland]

Daily data and detail data doesn't seem to match up. For example, the Max of maxs. for 2023-03-14 is shown as 50.34, but looking at the detailed view for that day the highest maximum value is 17.42. This is the same in the original JS app. Am I just misunderstanding something?

Why is there no data for March 31st?

Why do days run from 5am-5am? Is this a timezone thing?

These have now mysteriously fixed themselves for me. Even for old versions of the code, and for the js app, so it's not a code related issue.

@dalonsoa @CWestICL Could you have a look and let me know what times you see in the detail table (i.e. does it run from 5am to 5am or midnight to midnight), and do you see the high (50+) values in the detail table for 14th March??

[tsmbland]

@AdrianDAlessandro One other issue which we touched on before - it seems to only work when I have one callback function. Even if I define a second empty callback function that doesn't do anything the app breaks completely and none of the buttons work. It's ok as I've just been putting everything in one callback function with loads of if statements, but it's getting really large now so would be good to break up if possible.

[AdrianDAlessandro]

@AdrianDAlessandro One other issue which we touched on before - it seems to only work when I have one callback function. Even if I define a second empty callback function that doesn't do anything the app breaks completely and none of the buttons work. It's ok as I've just been putting everything in one callback function with loads of if statements, but it's getting really large now so would be good to break up if possible.

I know you've already merged, but just confirming I added a separate callback that's completely disconnected from the rest and it worked, then I tried the allow_duplicates flag and made a second one and it didn't work.

Turns out that Dash feature is broken in django_plotly_dash GibbsConsulting/django-plotly-dash#481

[tsmbland]

Thanks for looking @AdrianDAlessandro. You are indeed correct it's a specific problem with allow_duplicates and not multiple callbacks

[tsmbland]

Daily data and detail data doesn't seem to match up. For example, the Max of maxs. for 2023-03-14 is shown as 50.34, but looking at the detailed view for that day the highest maximum value is 17.42. This is the same in the original JS app. Am I just misunderstanding something?

Why is there no data for March 31st?

Why do days run from 5am-5am? Is this a timezone thing?

These have now mysteriously fixed themselves for me. Even for old versions of the code, and for the js app, so it's not a code related issue.

@dalonsoa It seems that this switch happens when I ran the scripts in #144. Flushing the database and reloading brings it back to 5am-5am (i.e. UK timezones), then running the scripts again brings me back to Peruvian(?) timezones (i.e. midnight-midnight). Haven't yet figured out what's going on, just an observation.

[dalonsoa]

The synthetic data creation is... dodgy. It is assigning UCT timezone to every station rather than leaving those stations that have a timezone as they are. As a result, it is breaking the existing data. I'll update the synthetic data creation such that stations with a proper timezone set are left untouched.