Developer Doc
Application Layout
Database Models
Fixture/Default Database Data
Dart is a standalone Django web application. It uses a python based Daphne web server to handle serving web content to load and modify data stored in a Sqlite3 database. Interactions with the web forms is primarily done though the use of HTMX which allows most of the web page updates to be done using python functions that are called using HTTP URLs. HTMX is an Ajax library that is configured to make URL calls to a web server and then swap in the response from the web server to specified parts of the webpage. This means the whole page doesn't need to be updated every time an update takes place, HTMX only changes the relevant part of the page.
When dealing with forms the typical pattern is to call a url using an hx-get="<url here>". The called function will return an alert indicating to the user something is happening, but the alert will contain hx-post="<url here>" and hx-trigger="load" attributes that will then call a function to do the actual work. Typically get requests (hx-get) are done when loading content whereas post requests (hx-post) is done when something is to be changed.
When opening file dialogs to ask a user to select a file, where possible, the file dialog should be through the users web browser. Files are typically process through the use of parsers that reside in the dart/core/parsers/ directory.
By default web browsers are designed so that web pages cannot gain access to a users file system. When requiring access to the file system, DART has the easygui package that will allow for a file/directory selection dialog to be shown to the user from the python side. However this dialog can take a moment to load and be easily pushed under other applications where the user cannot see or access it without closing the applications above the file chooser.
An example is when the user is asked to enter the directory where BTL/ROS files exist. Using a web browser based file chooser is faster and much more dependable, but only allows the user to access to individual files. In the case of reading BTL files, some of the data to identify a sensor resides in an accompanying ROS file. In order to use a browser's file chooser the user would have to select both the BTL and ROS file, both files would then have to be transferred by HTTP protocol to the application, decoded and synchronized. In this case it is easier and more efficient to use the easygui file chooser to allow the user to select a directory. Python then has direct access to work with the files.
When uploading files using the browser's file chooser from a web form, the form requires either enctype="multipart/form-data" or hx-encoding="multipart/form-data".
<form id="id_form_load_samples" enctype="multipart/form-data">
<input type="file" name="sample_file" accept=".dat,.csv,.xls,.xlsx,.xlsm" />
<!-- Form elements here -->
</form>Elog files are loaded though the dart/core/parsers/elog.py module that will break a log file into messages, add instruments/stations to the local database and then pull out required elements based on the core.models.ElogConfig settings. The default settings can be overridden by supplying a DefaultElogConfiguration.json file in the dart/settings/ directory, but settings can also be modified per mission using the Elog config form accessed from the Mission Events page next to the button used to select an Elog file.
For CTD data (loaded from SeaBird .BTL files), DART has the user point to a specified directory where the files exist. The .BTL files often have a complementary .ROS files that contains sensor data used to populate the SampleType table if the .ROS file exists. Because of the complex relationship between a .BTL and .ROS file the files cannot be easily uploaded using the standard browser file input dialog.
Bottle files are processed by the dart/core/parsers/ctd.py parser that creates global core.models.SampleType objects. DART SampleTypes are used to provide human readable labels for sensor and sample names. Later they are also linked to BioChem DataTypes. An at-sea user may not know the proper BioChem DataType to use when loading data for at-sea reporting, so this method allows data to be loaded and then linked to the BioChem DataType post-mission by a member of the data processing team.
Once sample types are created Bottle data is loaded and stored as DiscreteSampleValue objects, the same way that sample data is stored.
Discrete data refers to data loaded from a csv or excel file that contains a column representing a sample/bottle ID and a column that represents a numeric value. Comments and Data Quality flags are optional elements of a Discrete data type for a sample. Discrete data is processed using the general dart/core/parsers/SampleParser.py module that contains methods for locating header data as well as producing Pandas dataframes from the csv or xls files. Once a SampleType has been created and a file supplied a user can create a file configuration (core.models.SampleTypeConfig) that will describe how the sample parser should read data. The configs are saved in the database where they can be reused if a file of the same type is discovered in the future.
For details on loading sample data see Sample Data Loading
Plankton data for either Phytoplankton or Zooplankton is loaded using the dart/core/parsers/PlanktonParser.py module. Phytoplankton is associated with bottles linked to CTD events. Whereas Zooplankton is associated with bottles linked to RingNet events.
DART uses several error tables in the database to track errors during the file loading and validation process. Ideally file errors are attached to a file name using the core.models.FileError table so later on an error can be tracked back to a specific file where it can be corrected before reloading the data. The drawback of this method is that not all errors can be linked directly to a file. Events can be individually created by hand, rather than by bulk from an elog file, so core.models.ValidationError objects are attached to an event. More general application level errors should be attached to a core.models.Error object that links the error to the mission the error occurred within.
Errors can be downloaded as part of an errors.csv file acquired using the Downloadable reports feature.