Merge output from evolutionary search into one document

[scott-materials]

Describe the desired feature

As an evolutionary search 🎰 progresses, a series of summary files are produced, including:

convergence__staged_relaxations.html
convergence__time_vs_energy_per_atom.html
distribution_of_energy_per_atom.html
distribution_of_subworkflow_times.html
history_of_the_best_individuals.md

Rather than writing these to five different files, I think it'd be useful if they were all written to a single html file. That way I only have to bother to open/close one file, not five.

Additional context

No response

To-do items

notes by @jacksund :

• condense workflow outputs to database objects
• streamline Figure registration from a database table/object
• allow figures to be written to file and html div with single implementation
• add default method for writing summary output from db object
• link to website URL for viewing all results on a single page
• add templates and views for structure-prediction flows

[jacksund]

Agreed, I'd like to move in this direction as well. 👍

I also think the same can be requested for the other summary files too (i.e. merge all outputs into a single "report" file). If we're building a larger html file, we might as well include everything we need. Keep running with this idea, and we end up with a website/django view.

Ideally, I think the only "output" files should be (1) a URL link to the website view (even to a local server link) and (2) a "static html" of the website view. This addresses many output files and moves toward a single "report" html (which is also a view in the website).

I've planned on doing this for all workflows (not just the evo search) but haven't had much time to put towards website templates. However, I can set up the basics to address this issue. Since you've built django templates before, you'll easily be able to update things once I have the main template set up.

[jacksund]

Below are notes on how I would write a static html locally for the view

---

from django.template.loader import get_template
from django.template import Context

template = get_template(template_src)
context = Context(context_dict)
html  = template.render(context)

https://docs.djangoproject.com/en/4.1/ref/templates/api/#loading-a-template
https://docs.djangoproject.com/en/4.1/ref/templates/api/#rendering-a-context

---

Also see the render_to_string method:

from django.template.loader import render_to_string
rendered = render_to_string('my_template.html', {'foo': 'bar'})

https://docs.djangoproject.com/en/4.1/topics/templates/#module-django.template.loader
https://docs.djangoproject.com/en/4.1/topics/templates/#django.template.loader.render_to_string

[jacksund]

@scott-materials

I've streamlined how results are used when writing output files + making the web UI.

While the last few PRs might seem like overkill, they're pretty important for streamlining our code -- and not having to implement things multiple times & in multiple places.

Before these PRs, the processes were isolated for results+output files and database+webUI. This was easier when starting out Simmate, but it's now reached the point where maintaining two implementations was too difficult.

Here's how it looked before:

graph LR
  A[Calculation run] --> B[Results];
  B --> C[Save to output files];
  B --> D[Convert to database object];
  D --> E[Save to database];
  E --> F[Build into html template];

Loading

Here's how things work now:

graph LR
  A[Calculation run] --> B[Results];
  B --> C[Convert to database object];
  C --> D[Save to database];
  D --> E[Save to output files];
  D --> F[Build into html template];

Loading

adding new plots

Now if you have a script that builds a plot from a database table, I can immediately add it to both output files AND the website UI. This will be very important as Simmate grows. How to build a new plot:

from simmate.visualization.plotting import PlotlyFigure

class MyNewPlot(PlotlyFigure):

    def get_plot(table):
        data = table.filter(...).all()  # grab data from the table
        # .... make a plotly figure and return the object ....
        return plot

end goal

Ultimately, there will be a link to the website pages, but no local "report" file. In the simmary_summary.yaml file will be everything you need -- URL link, database table, database id.

Saving a local html has complications due to...

1. links and static assets will not work
2. building the html will be slow and affect workflow speed
3. building a minimal template falls back to the original issue (maintaining two implementations of the same thing)

So this issue will be closed once there is a website view for structure-prediction workflows. There won't be a single file as you originally requested, but there will be the (even better) option of viewing things in the website UI.

[scott-materials]

Impressive! I like it!

[jacksund]

@scott-materials I haven't done any html formatting, but everything is in a single page now. You can check out the html here. In the template, calcuation is the database table object (so FixedCompositionSearch.objects.get(id=123))