Save and load lattices in JSON format

[lfarv]

With this PR, AT can save and load lattices in JSON format. JSON is a text-based format with the following properties:

• JSON is a widely used format for storing structured data (ex.: Jupyter .ipynb files),
• JSON is natively supported by both Matlab and python,
• JSON files can be manipulated by external editors and viewers,
• AT files in JSON format can be used for exchanging lattices between Matlab and python, as an alternative to the proprietary .mat format.

In addition, a few modifications have been made to the global load/save package to prepare future formats:

• for file formats possibly containing several lattices (.mat, Elegant files), a specific keyword allows to select the desired lattice. A unique keyword use=lattice_name is now used for all formats, while the specific keywords (matkey=, lattice_key= etc.) are still understood for compatibility,
• similarly, in the lattice attributes, the specific attribute mat_file, elegant_file... which records the input file name is replaced by a single one: in_file. This avoids the accumulation of different attributes, with the consequence that the last loaded file could not be identified. The file name extension is enough to know which kind of file was loaded.

Finally, the file loading structure is improved to allow an easier addition of new formats.

[simoneliuzzo]

Dear @lfarv,

a few tests in matlab:
EBS lattice, no errors

FCC lattice WITH errors:

ALL looks ok here.

I will now try to load in python and save/load json in python and back to maltab

[simoneliuzzo]

Dear @lfarv,

I tried to load the json file in python.

mat_key does not work anymore

r = at.load_lattice(matfile, mat_key='betamodel')

Traceback (most recent call last):
File "/machfs/liuzzo/accelerator_toolbox_tests/matlab/jason/load_json_test.py", line 10, in
r = at.load_lattice(matfile, mat_key='betamodel')
File "/machfs/liuzzo/accelerator_toolbox_tests/matlab/jason/test_json_files_venv/lib/python3.9/site-packages/at/load/allfiles.py", line 48, in load_lattice
return load_func(filepath, **kwargs)
File "/machfs/liuzzo/accelerator_toolbox_tests/matlab/jason/test_json_files_venv/lib/python3.9/site-packages/at/load/matfile.py", line 193, in load_mat
kwargs.setdefault("use", kwargs.pop("key"))
KeyError: 'key'

the new keywork use instead is ok.
I can load the json file and plot it.

[simoneliuzzo]

The help of save_lattice and load_lattice could be updated to include save_json and load_json

[simoneliuzzo]

The JSON files saved by matlab and by python are different. In particular the R1-R2 and T1-T2 fields are not in the same order.

[simoneliuzzo]

Same for the field Energy.
At least for cross check (diff for example) it would be nice to maintain the same identical output for the json files produced by matlab and python.

[simoneliuzzo]

Dear @lfarv

globally I find this new feature very useful and working correctly!

The only issues are the mat_key (for backward compatibility) and a fixed order of the fields in json file produced by atwritejson in matlab and save_json in python

let me know and I will further test.

I have seen that many functions are impacted but I did not test that all other methods are working as before (save_lattice, load_lattice, save_mat, load_mat, etc...). May be a new test could verify that all load and save did not change their output?

[lfarv]

Hello @simoneliuzzo
Thanks a lot for testing this branch ! And nice to see that you find it useful. About your remarks:

mat_key does not work anymore

The mat_key keyword is repaired. Stupid typo in the code…

The help of save_lattice and load_lattice could be updated to include save_json and load_json

What do you mean ? Here JSON is mentioned in the help:

>>> help(at.load_lattice)
Help on function load_lattice in module at.load.allfiles:

load_lattice(filepath: str, **kwargs) -> at.lattice.lattice_object.Lattice
    Load a Lattice object from a file
    
    The file format is indicated by the filepath extension. The file name is stored in
    the *in_file* Lattice attribute. The selected variable, if relevant, is stored
    in the *use* Lattice attribute.
    
    Parameters:
        filepath:           Name of the file
    
    Keyword Args:
        use (str):          Name of the variable containing the desired lattice.
          Default: if there is a single variable, use it, otherwise select ``"RING"``
        name (str):         Name of the lattice.
          Default: taken from the file, or ``""``
        energy (float):     Energy of the lattice
          (default: taken from the file)
        periodicity (int):  Number of periods
          (default: taken from the file, or 1)
        *:                  All other keywords will be set as :py:class:`.Lattice`
          attributes
    
    Returns:
        lattice (Lattice):          New :py:class:`.Lattice` object
    
    Check the format-specific function for specific keyword arguments:
    
    .. Admonition:: Known extensions are:
        
        .mat      
            Matlab binary mat-file. See :py:func:`.load_mat`.
    
        .m        
            Matlab text m-file. See :py:func:`.load_m`.
    
        .repr     
            Text representation of a python AT Lattice. See :py:func:`.load_repr`.
    
        .lat      
            Tracy format. See :py:func:`.load_tracy`.
    
        .lte      
            Elegant format. See :py:func:`.load_elegant`.
    
        .json     
            JSON representation of a python AT Lattice. See :py:func:`.load_json`.
(END)

Anyway, I'll try and improve the help.

[lfarv]

fixed order of the fields in json file

This is more difficult. The JSON spec states that the order of fields in the structure is irrelevant. In python I may have some control on the encoding, but in Matlab, the jsonencode function does all the job without any control, except "compact or not".

[lfarv]

@simoneliuzzo
I added a signature in the JSON file, to identify the files created by AT. This will allow better error messages when trying to load a JSON file not created by AT. Please discard your existing files which are missing this signature

[simoneliuzzo]

Dear @lfarv,

I think that being able to compare to json files is important.
json file bring the advantage of being human readable, but they may be extremely big and users as me, at the very first test will want to use common tools such as diff/meld etc. to compare files and spot potential issues.
This is why I think the order of the fields in the file should be deterministic (ex alphabetic order) and identical in the output of matlab and python.

Could you test that the modifications for json files did not harm any of the other possible file load/save? repr, mat, m, lat, lte etc...
The test off all these cases starts to be complicated (many options to test) and may deserve an automated test as for other features of AT.

best regards
Simone

[lfarv]

@simoneliuzzo
Forcing the field imposes a constraint which is not in the specification. That would force me to rewrite a pseudo-JSON encoder. This is against the idea of using a standard format with standard encoders. I think it's much safer to use the standard encoders provided by python and Matlab rather than rewriting a new one.

Now to compare JSON files, you can one of the many available 3rd party JSON comparators:

• JSON diff
• JSON compare
• JSON formatter
• SemanticDiff
• and tens of others…

Otherwise, I'll add a few tests for other formats (and accordingly a few data files)