Is there a specific reason we are storing the datasets in their serialised form rather than static fields/properties in the assembly?

[epignatelli]

Ported from BHoM/BHoM_Engine#1941

@IsakNaslundBh:

Not sure I fully understand the question?

Are you asking if there is a reason we are storing json files at all? Or is the question about the inner workings of the Library_Engine?

@epignatelli

The dataset is a collection of constant objects. Why do we store this collection in a json format, in a separate file, rather than into a class (eg. public static class Gradients : Dataset)?

@IsakNaslundBh:

Some varying reasons:

1. Ease of use for less experienced developers, as well as more experienced ones. Having it in JSON files and letting the menu be dictated purely by the file structure means that all you have to do to create a dataset is to generate the objects you like, in whatever UI you like, then plug them to a Dataset obejct and push that obejct through the file adapter. Then just put the file in the correct folder and raise a PR. No need to change any bit of code etc. Easy to do, easy to understand.

2. Those datasets can be massive. If we were to put all of the SteelSections on the steel section class it would be thousands of lines. Also it would require constant updates to the specific class, requiring far more code knowledge etc.

3. IMO, keeping the schema and the data separate makes more sense to me in general. Some for the reasons above, but also as a general concept. The class is just the class definition, not a lot specific instances of it.

Saying all of this, we have a few cases of static properties on the class (as I know you know). For example the Vector class and Point class (getting X, Y, and Z vectors as well as the origin point). Something like that might make sense for the gradient case, but still think in general data is better kept separate from the class definition.

@epignatelli

I am not sure how easier and more intuitive that is. Unless the data in the dataset can be generated procedurally with an algrithm, I don't see the benefits.

The menu organization is a superstructure that we generated - we could do the same if datasets where organised in a different way, e.g. in classes (we do it already to cluster methods in components). There is a lot of knowlege in what you're describing - why should I plug all into a Dataset object? Then use a FileAdapter? What's the correct folder? Again, we data is not procedurally generated, that's not easier than writing the same thing you would write in grasshopper into a cs file.

Yep, this I understand, it makes sense. Maybe that's the case for procedurally generated data, I suppose.
It has a drawback, though. If you changed a schema and not versioned, the project compiles and you know you broke the dataset only at runtime, if you use it.

I am no proposing to mix schema and data instances - the opposite. Ideally what I would do is to add another angle and have a MachineLearning_Datasets project (as well as any other namespace) that holds the instances.

My whole point here is: I can't use datsets in their simplest form: by typing the data that's in there. I need to open the UI, create the objects, serialise them, create a file adapter and push them into the correct folder.

[IsakNaslundBh]

I am not sure how easier and more intuitive that is. Unless the data in the dataset can be generated procedurally with an algrithm, I don't see the benefits.

The menu organization is a superstructure that we generated - we could do the same if datasets where organised in a different way, e.g. in classes (we do it already to cluster methods in components). There is a lot of knowlege in what you're describing - why should I plug all into a Dataset object? Then use a FileAdapter? What's the correct folder? Again, we data is not procedurally generated, that's not easier than writing the same thing you would write in grasshopper into a cs file.

Agreed, that there is a lot of knowledge in what I am describing, and that this knowledge most probably need to be documented better. Saying this, I still think it is easier to understand for a new contributor is that all you need to do is to place a file in a folder compared to having to manage that through some other mechanism.

But would disagree that it would be as easy to write in the data in a CS file compared to doing it in something like GH. A lot of this data is more or less procedureally generated to some extent (as you are stating), generally from large tables of data, from excel for example, that are then used to generate the obejcts. Having to sit and type in all the values for all the dimensions and constants for all of our 2000+ steel sections into a cs file would be close to impossible, whilst just generating the sections you need is relatively simple.

Yep, this I understand, it makes sense. Maybe that's the case for procedurally generated data, I suppose.
It has a drawback, though. If you changed a schema and not versioned, the project compiles and you know you broke the dataset only at runtime, if you use it.

Agree on the version drawback. This requires the discipline/toolkit lead to be aware of the datasets and make sure they are being upgraded as appropriate. For the structures datasets, that are the oldest, this has had to happen a few times already, but have gone relatively smooth. Not sure having it as compiled code would help there really or make it easier, but agree, it would at least flag it.

Maybe this could be solved by some additional compliance/run check or similar @FraserGreenroyd . Something that makes sure that datasets that were previously de-serialising still are after a specific change.

I am no proposing to mix schema and data instances - the opposite. Ideally what I would do is to add another angle and have a MachineLearning_Datasets project (as well as any other namespace) that holds the instances.

Ok, sorry, misunderstood you point there :)

My whole point here is: I can't use datsets in their simplest form: by typing the data that's in there. I need to open the UI, create the objects, serialise them, create a file adapter and push them into the correct folder.

What kind of data are you talking about here, so I can understand your context a bit better?

[epignatelli]

Yes agreed, but my point is to allow doing things without reading all the wiki, but just by looking at the code that is there already and replicate. But of course a minimum level of knowledge is always necessary.

So, I have a case in which my data is just a list of metadata - the information to retrieve (either download or load from disk) a file.
That's basically the model library for machine learning.

It's much easier for me to type the information in by hand, rather than doing it in grasshopper - which goes through layers of manipulation and wrappers.

[adecler]

Agreed, that there is a lot of knowledge in what I am describing, and that this knowledge most probably need to be documented better. Saying this, I still think it is easier to understand for a new contributor is that all you need to do is to place a file in a folder compared to having to manage that through some other mechanism.

But would disagree that it would be as easy to write in the data in a CS file compared to doing it in something like GH. A lot of this data is more or less procedureally generated to some extent (as you are stating), generally from large tables of data, from excel for example, that are then used to generate the obejcts. Having to sit and type in all the values for all the dimensions and constants for all of our 2000+ steel sections into a cs file would be close to impossible, whilst just generating the sections you need is relatively simple.

I cannot agree more.
Adding to that the fact that json files don't require to recompile the project as opposed to cs files.

[al-fisher]

I get your point on ease of creation for the case of potentially small surgical datasets and for someone in the code already @epignatelli
Is perhaps a new use case slightly distinct from many of the other applications of serialising datasets, I have seen to date at least.

Although arguably the serialised labels I have created:
https://github.com/BHoM/BHoM_Datasets/blob/master/DataSets/Planning/Labels/IssueLabels.json
is a similar example.

The origins of simply storing the serialised JSON data for "datasets" is for me to make it trivial for anyone to turn any collection of BHoM objects into a "dataset" for repeated use. The same way we use the JSON files to pass around serialised analysis or BIM models, or anything - you can memorialise those exact same instances of objects for repeated use by putting the same .json in the datasets folder.

I also have historically been a fan of clear division of oM:

• schema; fixed at runtime; .cs format;
• instance; defined at runtime; .json format;

But as always if use cases are there good to enable flexibility.

Be great to see in some sense what data you are dealing with and understand your work flow? Can you share?