Replace JS code for validation (umbrella issue)

[dalonsoa]

Like with the reports (#120) the validation screen is currently created and displayed with a mixture of:

• Forms
• Views:
  • DailyValidation
  • DetailList
• HTML templates
• JS code to create the plots

With diverse logic here and there to deal with the data and the user interaction. While a variety of files is normally required in Django, this is a mess.

This task will have as main goal removing JS code altogether and replace it with appropriate equivalents in terms of content and user interaction using Dash/Plotly. These will need to be hooked in the correct places of the views and/or forms, and the HTML template modified accordingly.

This might be a substantial task and should be broken down into smaller issues as needed, keeping this as umbrella.

[tsmbland]

A few notes and ideas, but would definitely want to discuss first before proceeding.

---

OVERALL PLAN

1. Modify project to be compatible with dash apps using django_plotly_dash (Test this approach first using a minimal 'hello world' dash app). Consult with Callum.

   • https://django-plotly-dash.readthedocs.io/en/latest/
   • https://www.youtube.com/watch?v=psvU4zwO3Ao
   • Alternatively look here: https://dash.plotly.com/integrating-dash
   • Also look at Liionsden. This creates plots with plotly and then converts to html using plotly.offline.plot, which can then be inserted into the html for the page. Works for single plots created with plotly, but I'm not sure how well this would support more complex apps with buttons and callbacks. Could be an option for Callum's issue which is just a single plot without callbacks.

2. Replace html code for the main app (daily_validation_right_panel.html) with dash/plotly

   • single file defining app.layout, but functions building certain components can be abstracted to other files (and potentially shared with Callum's issue)
   • to start with can use daily_validation() to get the data, before integrating this into the page with the forms
   1. modify the submit button callback to launch the dash app with selected data (or include this and forms in the dash app itself)
   2. tabs
   3. graph (dispersion, bar)
   4. daily table
   5. detail table

3. Write callbacks using dash (convert from daily_validation.js), with api calls using requests module

   1. switching tabs (does this even need a callback?)
   2. opening up the detailed view
   3. filters for detailed view
   4. save to validated

Probably a few other things I'm missing

---

The following is mainly for my benefit to guide me as I work through this:

---

LAYOUT

Current approach:
Page layout defined in daily_validation.html
-> extends daily_validation_right_panel.html which defines the main app
-> extends daily_validation_tab1.html etc which define each of the three tabs
Filters on the left are dealt with using django forms.

Plan:

• Can integrate dash app into django app using django_plotly_dash. This lets you load the app into the html as a div
• Will need to convert html code for the main app (daily_validation_right_panel.html) to dash

---

FILTERS

Functionality:

• Various drop-down lists/inputs for selecting a subset of the data and relevant variable

Current approach:
DailyValidationForm: class for the form specifying fields, default values etc
-> DailyValidation: handles api calls
-> gets data using daily_validation(args) (python function)
Rendered in daily_validation.html (id: form_validation) with Submit button added below

• form data is then used by daily_query_submit accessing form_validation (ajax api call)

Plan:

• Either rewrite to be part of dash app, or leave it and get dash app to interact with the form (via api calls)

---

SUBMIT BUTTON

Functionality:

• Generates daily report table/graph based on filters above

Current approach:
Button in daily_validation.html (id: btn_submit)
-> js callback: daily_query_submit (in daily_validation.js)
-> sends ajax post request to $("#form_validation").attr('action') and triggers building tables and plots

Plan:

• this could be incorporated into the dash app and linked to a dash callback
• or keep outside the dash app and re-initialise the dash app when pressed. Then the dash app would read data from the form whenever it starts up (api call)

---

CLEAN FORM BUTTON

Functionality:
Refreshes the page, switching the filters back to default

Current approach:
Button in html (id: btn_clean)
-> navigates to validated_daily_validation (i.e. refreshes the page)

Plan:
Can probably leave this as it is

---

DAILY DATA

• daily_query_submit gets data from form_validation with the following: $("#form_validation").serialize()
• data has the following attributes (amongst others):
  • data: this provides data to bootstrapTable which creates the table
  • value_columns: list of column names. Passed to get_columns_daily to create columns for the table
  • series: contains data for plotting, passed to plotting functions. Split into series.measured, seres.validated, series.selected
  • variable: contains information about the series data. Has is_cumulative attribute which determines whether to plot the data as a bar plot or dispersion plot, and var_nombre which sets the title for the plot

---

DETAILED DATA

• data is obtained by sending a get request to '/validated/detail_list/' + station_id + '/' + variable_id + '/' + date + '/' + var_minimum + '/' + var_maximum
• relevant attributes of data:
  • series: provides data for the table
  • value_columns: provides column names for the table
  • indicators: information about the number of datapoints (I think?)

---

TABS

Functionality:

• Tabs for switching between PLOT, DAILY REPORT, (DETAIL OF SELECTED DAY)

Current approach:
Tab bars created in daily_validation_right_panel.html -> id: tab1-tab, tab2-tab, tab3-tab
-> when clicked call getTab callback (js function)
-> this hides all tabs then shows the relevant one

Plan:
Tab functionality can be built into the dash app

---

DAILY REPORT TAB

Functionality:
Table showing the selected data

• Each row is a day, which is an average of many datapoints across that day
• Colouring cells based on certain criteria
• What do the numbers in grey circles mean?
• Footer at the bottom - is this an average across rows?
• Checkboxes to select/deselect each row
• Action button for each row (magnifying glass)
  • This opens up a detailed table of datapoints for that row

Current approach:

• Columns of the table defined in get_columns_daily (js)
• Table built using daily_query_submit (js) (called by btn_submit)

Plan:

• write the table using plotly/dash
• https://plotly.com/python/table/ or https://dash.plotly.com/datatable
• callbacks to handle action buttons

---

DETAIL OF SELECTED DAY TAB

Functionality:

• Table showing data for that day
• Colouring cells based on criteria
• Footer at the bottom
• Ability to select/unselect rows in batch with a text input
• Various filters which I don't quite understand
• Action button for each row allowing user to manually modify data (Average, Maximum, Minimum) (seems dangerous?!)
• Button to add new row to data
• Button to save day to Validated

Current approach:

• table built by detail_details
  • uses get_columns_detail
  • interacts gets data from DetailList via api call
• form for adding/updating row in detail table: open_form_new, open_form_update
  -> detail_modify_row, detail_new_save: modify/add row to dataframe

Plan:
Some of this is fairly complex but should be possible in plotly/dash with some work
Use dash callback to launch the table, and api call to get the data

---

PLOT TAB

Functionality:

• Scatter plot (time vs the variable in question) (or bar chart depending on the variable)
• Three colours for Measurement, Validated, Selected

Current approach:

• bar_plot and dispersion_plot functions, which call generate_traces_bars and generate_traces_dispersion
• these are called by daily_query_submit to build the graphs

Plan:
Should be easy to replicate with plotly

---

SAVE TO VALIDATED BUTTON

Functionality:
Gathers validated data and saves by sending post request

Current approach:

• btn_daily_send, btn_detail_save
  -> calls save_daily/save_detail when pressed
  -> send ajax post request to validated/daily_save

Plan:

• this will need to be part of the dash app with callbacks
• use python requests module for api call

---