Load particles according to edm4hep #36

brauliorivas · 2024-06-09T20:11:40Z

BEGINRELEASENOTES

One of the principal worries of dmx is to keep up to date with the edm4hep data event model. Whenever someone wants to explore their data, this should represent the most recent model of edm4hep.
A way to keep up to date with the model, is to (periodically, under triggers, when manually run, etc) execute two scripts that will read the yaml definition file for edm4hep and create the output/datatypes.js file that will be used by dmx to detect which members/relations need to be loaded given some file i.e. reads the json data from the events, and loads everything according to output/datatypes.js.

ENDRELEASENOTES

The idea here is to follow the goal mentioned by @tmadlener, to always show the latest version. The output/datatypes.js file will be used to know which members and relations (name + type) has each type of particle. Then, when loading an event, the data will be shown according to the file. However, two things may happen:

The data from edm4hep is different from the json event, so in that case, we could ignore those values from the json.
The data from edm4hep is the same as in the json event, so everything is shown as expected.

Then, we could have a "general" particle for each type, that will access members and relations, to later have some methods that will allow to draw that particle, or draw a relation towards a type of particle.

tmadlener

I suppose this is a somewhat parallel development for #25 with a slightly more automated approach to generating the loading functionality, right?

I think it goes into a good direction already, I have a few comments inline as well. In general, this solves a few of the problems that we will be facing, and it should allow us to always read the latest format files, but I am not yet sure it will allow us to read also older files that have been written with an older version of EDM4hep. Having said that, maybe we don't need to solve that in all its glory now already and should rather try to continue with getting some of the visualization going for these with the current version of EDM4hep.

As far as I can tell, the loading is almost fully there by now, and we could start to prototyping some of the visualization for the things we can load now.

js/types/dynamic.js

tmadlener · 2024-06-10T12:03:14Z

js/types/reconstruction.js

+class Reconstruction extends Particle {
+  constructor() {
+    super();
+  }
+}
+
+export class Cluster extends Reconstruction {
+  constructor() {
+    super();
+  }
+}
+
+export class ParticleID extends Reconstruction {
+  constructor() {
+    super();
+  }
+}
+
+export class ReconstructedParticle extends Reconstruction {
+  constructor() {
+    super();
+  }
+}
+
+export class Vertex extends Reconstruction {
+  constructor() {
+    super();
+  }
+}
+
+export class Track extends Reconstruction {
+  constructor() {
+    super();
+  }
+}


I am not sure whether it's necessary to differentiate between Particle and Reconstruction in this case. In the end we want all of the datatypes that we find in the yaml file to be on sort of the same level (at least for reading in). We can then still group them in some other way, but we will probably do that manually. Additionally, since the bodies for all of these are effectively the same, I think they could also go into the file that is generated, since we will do it for (almost) all datatypes in any case.

Right. I'm going to create only a general EDMObject (as you mentioned) that will probably contain common features across all the objects. Also, I don't think it's necessary to add it to the generated file, because this EDMObject should contain more "logic" than relevant information when loading.

tmadlener · 2024-06-10T12:03:19Z

js/types/particle.js

@@ -0,0 +1,8 @@
+export class Particle {


I think this is a bit poorly named. In the end not everything in the datamodel will be something that resembles a particle. Maybe EDMObject or something like that would be a better name here.

tmadlener · 2024-06-10T12:04:37Z

js/types/units.js

+export const units = {
+  "charge": "e",
+  "mass": "GeV",
+  "time": "ns",
+  "momentum": "GeV",
+  "vertex": "mm",
+};


Does this really apply for all of the occurences? vertex in itself doesn't have a unit, the position of a vertex on the other hand probably will, so maybe position would be better here than vertex?

tmadlener · 2024-06-10T12:07:00Z

test/load.test.js

+  it("should only load particles of one type", () => {
+    const loadersConfig = "ReconstructedParticle";
+
+    const particles = loadParticles(data, "0", loadersConfig);
+
+    expect(particles.hasOwnProperty(loadersConfig)).toBe(true);
+  });


If you really want to test this, you also need to check that nothing else has been loaded. Otherwise, you are effectively just testing one half of what you want to test (according to the test title).

tmadlener · 2024-06-10T13:13:38Z

output/datatypes.js

+  "edm4hep::MCParticle": {
+    "members": [
+      {
+        "type": "int32_t",


Is the type useful in this form (i.e. targetting c++). Do we actually need the type for anything at the moment? Otherwise, we could drop it and only bring it back if we really need it.

Ok, then I will drop them.

brauliorivas · 2024-06-10T20:23:32Z

I suppose this is a somewhat parallel development for #25 with a slightly more automated approach to generating the loading functionality, right?

Indeed, this way we get the latest version.

I am not yet sure it will allow us to read also older files that have been written with an older version of EDM4hep

~~I'm not either. The option is to try to read the common things between versions, but if some particle changes too much, it isn't very helpful.~~ We could (a bit "ugly solution") include (all, some?) versions of edm4hep when generating the file with definitions. This way, we can use the latest version by default, or the one that best fits an object from edm4hep when reading a JSON file. We may also try to fetch the version according to releases but this would require a server (unless the user wants to clone the repo locally and run some sort of script). Another way is to simply read the common members/relations across versions. So for example if the latest version of MCParticle had members a and b, but the user has a file with an MCParticle from an old version with members a and c, then we only read a, b is blank and c is ignored. I think the first solution is better even though is not so orthodox, it serves well.

github-actions · 2024-06-10T22:11:40Z

PR Preview Action v1.4.7
Preview removed because the pull request was closed.
2024-06-19 14:34 UTC

tmadlener

I think for now we can focus on just getting the loading of the current / latest version work dynamically. In principle we have all the pieces in place to make this dynamic on the version as well, IIUC. Mainly (using current names not the proposed ones),

loadParticles and loadParticleType, which would need an additional version argument to then dispatch to another loading function.
A way to generate different versions of the objects to load

With this we should be able to add specific loading schemes later.

One thing that I don't yet understand is: When is the generation of the datatypes.js actually run? I assume it's done whenever necessary offline? Is this manually triggered? If that is the case, could you put just a few lines / commands into a markdown file at least minimally describing the process?

I have also left a few smaller comments. The main one I have at this point is about naming. There is still quite a bit about particle in function and variable names. However, IIUC, this could all be object or type (depending on context), as it applies to all datatypes not just particle like ones, right?

Finally (also in one of the inline comments), it should be possible to also fit MCParticle into this scheme, right? Could you do that, just to see if it is compatible with the current approach? If not, you might have to adapt either this dynamic loading or the current implementation for the MCParticle, because in that is no special datatype, it was just the one we started with.

tmadlener · 2024-06-11T12:50:09Z

js/types/load.js

+}
+
+const particlesToLoad = [
+  // subset of datatypes


I am missing the edm4hep::MCParticle here (or I am missing something entirely?). It should be possible to load that through the same mechanism, right? That would also serve as a good "real world" test to see whether the scheme works to get to a useful representation in memory.

tmadlener · 2024-06-11T12:53:31Z

model/index.js

+    if (type.includes("edm4hep::")) {
+      newMembers.push(new DataTypeMember(type, name, unit));
+    } else {
+      newMembers.push(new DataTypeMember(null, name, unit));
+    }


Sorry, I have missed this bit before when I was talking about dropping the types. Do we need the type information for members that are components? (I.e. the things starting with edm4hep::? If yes, than we might as well just keep all the type information instead of selectively dropping stuff. If we don't need the type information for the components either, I think we can drop all of them here.

Checking whether we can fit the MCParticle into this dynamic scheme should give as a good indication of what is needed, I think.

I thought to keep only types from objects that contain edm4hep:: because maybe when drawing relations, we may need to know towards what type of object the relation is. But you are right; currently it isn't necessary, and we may continue without them. Also, I've included MCParticle which loads seamlessly. Using the testing json file, I get this

when running the loading function, without error.

Is this also already used when I load a file in the viewer for displaying?

Currently, not.

js/types/load.js

m-fila · 2024-06-13T11:02:50Z

model/MODEL.md

+## Further improvements
+
+-   We could trigger the update process automatically when a new release of edm4hep is published, but for now, we will keep it simple and run the update process manually.
+-   To maitain backward compatibility, we could add a versioning system so we can later load different versions of the edm4hep data model into dmx, however we currrently load the latest version.


typos

Suggested change

- To maitain backward compatibility, we could add a versioning system so we can later load different versions of the edm4hep data model into dmx, however we currrently load the latest version.

- To maintain backward compatibility, we could add a versioning system so we can later load different versions of the edm4hep data model into dmx, however we currently load the latest version.

m-fila · 2024-06-13T11:03:53Z

model/MODEL.md

+npm run update
+```
+
+to run both processes at once and immediatly update the dmx application.


typo

Suggested change

to run both processes at once and immediatly update the dmx application.

to run both processes at once and immediately update the dmx application.

m-fila · 2024-06-13T11:04:28Z

model/MODEL.md

+```
+
+will download the latest information from the edm4hep, by using the file called `edm4hep.yaml` under `model` directory, containg up to date information about `edm4hep`. The second command
+


typo

Suggested change

will download the latest information from the edm4hep, by using the file called `edm4hep.yaml` under `model` directory, containing up to date information about `edm4hep`. The second command

tmadlener

This looks good to me. One thing that we could add as a test is to add a small JSON file that we can load and then make some assertions on to make sure that things work with "real" inputs and that the links are to the correct objects.

tmadlener · 2024-06-17T08:33:45Z

test/load.test.js

+import { objectTypes } from "../js/types/objects.js";
+
+test("load a collection of particles", () => {
+  const type = "edm4hep::Track";


I am not entirely sure why this test is here? It looks like the dynamic.test.js already does a very similar thing? What makes this test different?

load.test.js is like a more general version of dynamic.test.js. In dynamic.test.js, the 3 basic functions to load objects (loadMembers, loadOneToOneRelations and loadOneToManyRelations) are tested, meanwhile in load.test.js the function loadObjectType is tested, which uses the 3 functions. It's not exactly the same, because loadObjectType also makes some extra processing. Maybe we could test loadObjectType only because I think it may change less (the test is more result-prone than implementation-prone) and it's directly used by dmx.

brauliorivas · 2024-06-17T21:40:53Z

This looks good to me. One thing that we could add as a test is to add a small JSON file that we can load and then make some assertions on to make sure that things work with "real" inputs and that the links are to the correct objects.

Sure, I'll add this 👍🏻.

…or test suite

tmadlener

Thanks for adding the loading from json tests. This looks good to me. I have just one minor question, where I miss some bits of the testing framework.

@kjvbrt anything from your side for this? Otherwise, I would propose to merge this and then starting to work on the visualizations on top of this.

tmadlener · 2024-06-18T12:42:57Z

test/load.test.js

+    "edm4hep::ReconstructedParticle",
+  ]);
+
+  expect(objects["edm4hep::MCParticle"]).toBeDefined();


What does .toBeDefined() test here?

I thought like an "assertion". So basically, it tests if an object that contains that type of particle was created.

Ah OK. So effectively with the current setup

expect(objects["edm4hep::Cluster"]).toBeDefined();

would fail, right?

Yes, but that's an old commit. Currently, that test is not present.

brauliorivas · 2024-06-18T22:51:50Z

@kjvbrt anything from your side for this? Otherwise, I would propose to merge this and then starting to work on the visualizations on top of this.

Yeah, please 😃

tmadlener · 2024-06-19T10:15:25Z

One final thought of mine. This is able to handle multiple collections of the same type, right? For MCParticle there will usually only be one, but e.g. for Tracks or ReconstructedParticle there might be multiple collections. We can always address that in a separate PR if not yet.

brauliorivas · 2024-06-19T12:33:14Z

One final thought of mine. This is able to handle multiple collections of the same type, right? For MCParticle there will usually only be one, but e.g. for Tracks or ReconstructedParticle there might be multiple collections. We can always address that in a separate PR if not yet.

Yes, on a single event could exist multiple collections of the same type. So here all the collections are loaded and then "merged".

tmadlener · 2024-06-19T12:47:09Z

Is it then still possible to only view selected collections?

brauliorivas · 2024-06-19T12:52:00Z

Is it then still possible to only view selected collections?

Yes. But I think this should be developed when working on visualization.

kjvbrt · 2024-06-19T13:17:32Z

This looks good to me, let's continue with the visualizations... :)

brauliorivas and others added 20 commits May 29, 2024 18:12

create initial types of collections defined in gsoc

3cfbb4c

loading function to check if types are correctly working

186024e

test suite for loadParticles and buildLoader

fbfb0ea

change example data for RecoParticle test

c7b270c

Merge branch 'main' into new-types

170de18

check if previews work

27d03ec

small change

0a52737

another small change

02392eb

remove unnecessary exclamation mark

13978a4

eliminate try catch

5f48b1d

add version check + dynamic load of object

00bb203

update tests

83a9bbd

types for RecoParticle relations including 1:1 and 1:many

3e64c41

add links for types

f7a14c5

test for loading functions

904cd51

test reconstruction

ba9630c

remove a few things

69f3fd6

extract up-to-date types from edm4hep

a1afbfe

Merge branch 'new-types' into dynamic-types

1715a04

load particles according to edm4hep.yaml definition

cc58efa

brauliorivas had a problem deploying to github-pages June 9, 2024 20:11 — with GitHub Actions Failure

tmadlener reviewed Jun 10, 2024

View reviewed changes

improvements (advised) and create test a better test suite

49d5e6e

brauliorivas temporarily deployed to github-pages June 10, 2024 22:11 — with GitHub Actions Inactive

brauliorivas mentioned this pull request Jun 10, 2024

Create new types of collections #25

Closed

brauliorivas marked this pull request as ready for review June 10, 2024 22:13

tmadlener reviewed Jun 11, 2024

View reviewed changes

m-fila reviewed Jun 13, 2024

View reviewed changes

fix typos in MODEL.md

27ba065

brauliorivas had a problem deploying to github-pages June 13, 2024 13:43 — with GitHub Actions Failure

implement @tmadlener random color generator

e82ba4c

brauliorivas had a problem deploying to github-pages June 13, 2024 13:46 — with GitHub Actions Failure

brauliorivas added 6 commits June 13, 2024 16:45

minimal drawing for MCParticle

9c0ee0a

good working drawing

cd53b06

dmx works with new loading functionality

aeb04e2

dmx completely using new loading function

fc45409

remove old files + fix filter function + fix lib

bb0bc73

new loading is done. [WIP] new test suite

26597ee

brauliorivas had a problem deploying to github-pages June 14, 2024 02:22 — with GitHub Actions Failure

add very basic tests for new loading logic

0c1538b

brauliorivas temporarily deployed to github-pages June 15, 2024 04:12 — with GitHub Actions Inactive

tmadlener reviewed Jun 17, 2024

View reviewed changes

check for correct object loading and links when reading a json file f…

c37b2f3

…or test suite

brauliorivas temporarily deployed to github-pages June 17, 2024 22:36 — with GitHub Actions Inactive

tmadlener reviewed Jun 18, 2024

View reviewed changes

tmadlener merged commit fad0d00 into key4hep:main Jun 19, 2024
2 checks passed

tmadlener temporarily deployed to github-pages June 19, 2024 14:33 — with GitHub Actions Inactive

tmadlener temporarily deployed to github-pages June 19, 2024 14:34 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load particles according to edm4hep #36

Load particles according to edm4hep #36

brauliorivas commented Jun 9, 2024 •

edited

Loading

tmadlener left a comment

tmadlener Jun 10, 2024

brauliorivas Jun 10, 2024 •

edited

Loading

tmadlener Jun 10, 2024

tmadlener Jun 10, 2024

tmadlener Jun 10, 2024

tmadlener Jun 10, 2024

brauliorivas Jun 10, 2024

brauliorivas commented Jun 10, 2024 •

edited

Loading

github-actions bot commented Jun 10, 2024 •

edited

Loading

tmadlener left a comment

tmadlener Jun 11, 2024

tmadlener Jun 11, 2024

brauliorivas Jun 11, 2024 •

edited

Loading

tmadlener Jun 12, 2024

brauliorivas Jun 12, 2024 •

edited

Loading

m-fila Jun 13, 2024

m-fila Jun 13, 2024

m-fila Jun 13, 2024

tmadlener left a comment

tmadlener Jun 17, 2024

brauliorivas Jun 17, 2024 •

edited

Loading

brauliorivas commented Jun 17, 2024

tmadlener left a comment

tmadlener Jun 18, 2024

brauliorivas Jun 18, 2024

tmadlener Jun 19, 2024

brauliorivas Jun 19, 2024

brauliorivas commented Jun 18, 2024

tmadlener commented Jun 19, 2024 •

edited

Loading

brauliorivas commented Jun 19, 2024

tmadlener commented Jun 19, 2024

brauliorivas commented Jun 19, 2024

kjvbrt commented Jun 19, 2024

	- To maitain backward compatibility, we could add a versioning system so we can later load different versions of the edm4hep data model into dmx, however we currrently load the latest version.
	- To maintain backward compatibility, we could add a versioning system so we can later load different versions of the edm4hep data model into dmx, however we currently load the latest version.

	to run both processes at once and immediatly update the dmx application.
	to run both processes at once and immediately update the dmx application.

		```

		will download the latest information from the edm4hep, by using the file called `edm4hep.yaml` under `model` directory, containg up to date information about `edm4hep`. The second command

Load particles according to edm4hep #36

Load particles according to edm4hep #36

Conversation

brauliorivas commented Jun 9, 2024 • edited Loading

tmadlener left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brauliorivas Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brauliorivas commented Jun 10, 2024 • edited Loading

github-actions bot commented Jun 10, 2024 • edited Loading

tmadlener left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brauliorivas Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brauliorivas Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmadlener left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brauliorivas Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

brauliorivas commented Jun 17, 2024

tmadlener left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brauliorivas commented Jun 18, 2024

tmadlener commented Jun 19, 2024 • edited Loading

brauliorivas commented Jun 19, 2024

tmadlener commented Jun 19, 2024

brauliorivas commented Jun 19, 2024

kjvbrt commented Jun 19, 2024

brauliorivas commented Jun 9, 2024 •

edited

Loading

brauliorivas Jun 10, 2024 •

edited

Loading

brauliorivas commented Jun 10, 2024 •

edited

Loading

github-actions bot commented Jun 10, 2024 •

edited

Loading

brauliorivas Jun 11, 2024 •

edited

Loading

brauliorivas Jun 12, 2024 •

edited

Loading

brauliorivas Jun 17, 2024 •

edited

Loading

tmadlener commented Jun 19, 2024 •

edited

Loading