Allow lazy loading of DataContainers

[pmrv]

With this data containers (and potentially other data types) can be loaded "on demand" from HDF5 instead of all at once. In my simple example below this saves ~90% time on load.

The way it works is that in from_hdf values of the data container are not loading right away but just the hdf object and the path are saved in a "stub" object, that is then later transparently loaded upon item access.

Tests and design spec follow when we discussed this.

This should help with #364 and inspect mode in general. My vision would be that expensive to load objects should be using data container for all storage and can then be load()ed with lazy=True instead of inspect()ed.

Here's an example that can be run with ipython -i

# coding: utf-8
from pyiron_base import Project
import pyiron_base.generic.datacontainer as D
import numpy as np

d = D.DataContainer()
pr = Project('asdf')
h = pr.create_hdf(pr.path, 'test')

K = 'abcdefghijklmnop'
for c in K:
    dd = d.get(c, create=True)
    for c in K:
        dd[c] = np.random.rand(1024)

d.to_hdf(h, 'lazy')
# ~2s
get_ipython().run_line_magic('time', "d_normal = h['lazy'].to_object()")
# ~150ms
get_ipython().run_line_magic('time', "d_lazy = h['lazy'].to_object(lazy=True)")

assert (d_normal.a.a == d_lazy.a.a).all()

[jan-janssen]

I still have a bit of trouble to understand the full implications of this change. For example for a structure object stored in the HDF5 file, when are the positions loaded at job.structure or only when I select the positions job.structure.positions ? So I guess we should discuss this in more detail tomorrow.

[pmrv]

Currently they would be loaded when job.structure is accessed, assuming we'd have the following setup

class Job:
  ...
  def from_hdf(self, hdf, group_name):
    self._structure = HDFStub(hdf, group_name + '/structure')
  ...
  @property
  def structure(self):
    if isinstance(self._structure, HDFStub):
      self._structure = self._structure.load() # all HDF access for this structure occurs here
    return self._structure

In the current setup this change only affects how DataContainer is read from HDF, but HDFStub would be easily used to delay loading of other types as well.

However HDFStub already allows to customize this, with the register method. If the Atoms class supported to read itself lazily from HDF (which would have to be implemented separately), then you could do something like

HDFStub.register('Atoms', lambda hdf, group: Atoms.lazy_from_hdf(hdf, group))

So I guess we should discuss this in more detail tomorrow.

Yeah, will prepare a small demo.

[pmrv]

Only thing left from my side is whether I should enable lazy by default or not.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pmrv]

I rebased to fix the merge conflicts and will merge will lazy loading enabled by default, if no one objects.

[liamhuber]

Codacy actually said something meaningful! Otherwise lgtm.

[liamhuber]

Finally, codacy complains that this overrides a built-in type...do you know what it's talking about? In pycharm I don't get any complaint, and I was expecting like when you try to use "id" or "dict" as a variable.

[pmrv]

The argument type is a builtin. It is the base class of all classes and also a function to create new classes (iirc). Since I'm not using this inside this short method, I think it's ok, but I can also come up with a different name.