Generalize data source #1775

lpatiny · 2022-09-26T17:21:54Z

Currently if the data is coming from the server the only source that we allow in the .nmrium file is jcampURL.

We should make this more general and allow a FileCollection as source.

lpatiny · 2022-11-17T20:54:42Z

This is related to this issue in order to keep in cache the data (and possibly the parsed data).

Local cache of text() and arrayBuffer() filelist-utils#57

In the current situation we have:

{
  "version": 3,
  "spectra": [
    {
      "id": "53f1ff98-d146-4444-9a79-bbf8ee40102a",
      "source": {
        "jcampURL": "./data/cytisine/2d/COSY_Cytisin.dx",
        "jcampSpectrumIndex": 0
      }
    },
    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "jcampURL": "./data/cytisine/2d/COSY_Cytisin.dx",
        "jcampSpectrumIndex": 1
      }
    }
  ],

In the future we should have something like:

{
  "version": 3,
  "spectra": [
    {
      "id": "53f1ff98-d146-4444-9a79-bbf8ee40102a",
      "source": {
        "fileCollection": [
          {
            "name": "COSY_Cytisin.dx",
            "size": 1234,
            "relativePath": "data/cytisine/2d/COSY_Cytisin.dx",
            "lastModified": 234234,
            "jcampSpectrumIndex": 0
          }
        ]
      }
    },
    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "COSY_Cytisin.dx",
            "size": 1234,
            "relativePath": "data/cytisine/2d/COSY_Cytisin.dx",
            "lastModified": 234234,
            "jcampSpectrumIndex": 1
          }
        ]
      }
    }
  ]
}

The goal here is that the source could be as well a list of files for a Bruker Experiment.

Thinking about NMReData we will also need to know the index of each spectrum.

hamed-musallam · 2022-11-24T11:55:56Z

@lpatiny

This deal only with the local files which is not always the case, we should keep it general and instead of have jcampURL we have url and it accept the local or remote path

lpatiny · 2022-11-29T12:04:41Z

@jobo322 This feature will have to be implemented in nmr-load-save. Is there some obvious problems with this approach ?

hamed-musallam · 2022-11-29T12:15:49Z

@lpatiny

we need to consider if the remote file is a zip file

lpatiny · 2022-11-29T13:43:09Z

This is managed by filelist-utils. It should be transparent.

hamed-musallam · 2022-11-29T14:11:38Z

@lpatiny

what is going to be the behavior if you drag and drop a zip file, filelist-util will unzip the files and once you export as .nmrim will get the unzip files in the source object which is not available in your local machine because it is compressed

lpatiny · 2022-11-29T14:34:36Z

If you drag and drop something the source should stay empty. It is only when the data are coming from a server that we can store relative path to the '.nmrium' source file.

This could change in the future if we implement: https://developer.mozilla.org/en-US/docs/Web/API/FileSystem

lpatiny · 2022-12-01T15:33:31Z

How to deal with zip files ?

{
  "version": 3,
  "spectra": [
    {
      "id": "53f1ff98-d146-4444-9a79-bbf8ee40102a",
      "source": {
        "fileCollection": [
          {
            "name": "data.zip",
            "size": 1234,
            "relativePath": "data/data.zip",
            "lastModified": 234234,
          }
        ],
        "filter": {general: {keepFID: true, keepFT: false}, bruker: {expno:'10'}}
      }
    },
    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "data.zip",
            "size": 1234,
            "relativePath": "data/data.zip",
            "lastModified": 234234,
          }
        ],
        "filter": {general: {keepFID: false, keepFT: true}, bruker: {expno:'10'}}
      }
    },
    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "data.fid",
            "size": 1234,
            "relativePath": "data/data.fid",
            "lastModified": 234234,
          },
          {
            "name": "data.acqu",
            "size": 1234,
            "relativePath": "data/data.acqu",
            "lastModified": 234234,
          },
          {
            "name": "data.acqus",
            "size": 1234,
            "relativePath": "data/data.acqus",
            "lastModified": 234234,
          }
        ],
        "filter": {general: {keepFID:1}, bruker: {expno:'11'}}
      }
    },
    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "test.jdx",
            "size": 1234,
            "relativePath": "data/text.jdx",
            "lastModified": 234234,
          }
        ],
        "filter": {general: {keepFID:1}, jcamp: {index:'2'}}
      }
    }
  ]
}

/

lpatiny · 2022-12-01T15:49:37Z

@CS76 In the zip files you store on the server do you have one or many experiments ? Meaning for Bruker do you have 1h, cosy, tocsy in the same zip or one experiment per zip ?

CS76 · 2022-12-01T17:39:59Z

@lpatiny Currently, we are loading a single experiment at a time through one zip file via a URL provided to nmrium. Zip files are generated by nmrXiv, so we have total flexibility in generating them. Say one experiment per zip or multiple experiments (1h, cosy, tocsy) in a single zip (also, we generate these zip files on the fly).

hamed-musallam · 2022-12-01T17:43:19Z

@lpatiny

then it would be ok to go with having a file collection inside the source and it should be a role to have a single experiment per a zip file.

@CS76

what do you mean by a single experiment, does it mean one spectrum?

CS76 · 2022-12-01T17:44:22Z

what do you mean by a single experiment, does it mean one spectrum?

Yes, either 1h or 13 or cosy or tocsy

hamed-musallam · 2022-12-01T17:48:37Z

@CS76

Sorry for asking one more time, is this apply also on the Bruker ? as i know we could have multiple spectra inside the Bruker

hamed-musallam · 2022-12-01T17:52:41Z

@CS76

this will reflect on the shape of the nmrium object that you will get

  "version": 3,
  "spectra": [
    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "1h.zip",
            "size": 1234,
            "relativePath": "data/1h.zip",
            "lastModified": 234234,
          }
        ],
      }
    ,
    ranges:{values:[],options:{}},
    peaks:{values:[],options:{}},
    integrals:{values:[],options:{}},
    ...etc
    },
{
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "13c.zip",
            "size": 1234,
            "relativePath": "data/13c.zip",
            "lastModified": 234234,
          }
        ],
      }
    ,
    ranges:{values:[],options:{}},
    peaks:{values:[],options:{}},
    integrals:{values:[],options:{}},
    ...etc
    }
    ]
   }

hamed-musallam · 2022-12-01T18:16:30Z

@lpatiny

Are we going to assume that the Bruker or JCAMP should include one spectrum, whereas others who use the NMRium have to be sure that loaded file == one experiment?

lpatiny · 2022-12-02T06:42:05Z

No we can not assume this and we need this filter property.
Also the zip file contains one experiment but actually often 2 spectra (FID + FT) so even in this 'simple' case we will need the filter.

hamed-musallam · 2022-12-02T07:47:57Z

@lpatiny

but can not rely on the jcampSpectrumIndex for the jcamp files and we should move to filter otherwise it will be a missy if the order of the spectra changes in the .nmrium file, beside that, actually we change the order of the spectra from multiple spectra analysis.

and I think this change has to be listed in the migration file

CS76 · 2022-12-02T08:51:47Z

@CS76

Sorry for asking one more time, is this apply also on the Bruker ? as i know we could have multiple spectra inside the Bruker

Sorry if I wasn't clear but Bruker's output (one experiment) often contains FID and FT spectra.

lpatiny · 2022-12-02T11:34:51Z

but can not rely on the jcampSpectrumIndex for the jcamp files and we should move to filter otherwise it will be a missy if the order of the spectra changes in the .nmrium file, beside that, actually we change the order of the spectra from multiple spectra analysis.

and I think this change has to be listed in the migration file

But this was already done in the proposal I did

    {
      "id": "f1ff98d1-46e4-44da-b9bb-f8ee40102a69",
      "source": {
        "fileCollection": [
          {
            "name": "test.jdx",
            "size": 1234,
            "relativePath": "data/text.jdx",
            "lastModified": 234234,
          }
        ],
        "filter": {general: {keepFID:1}, jcamp: {index:'2'}}
      }
    }

Now the problem I see if when we have a zip file containing many jcamp that contain many spectra. We will probably need also a zip filter.

close #2021 #1976 #1775 #2013 * fix: wrong scale on fid * chore: fix migration 3 to 4 * chore: fix migration and filtering jcamp * fix: export as JSON * chore: migrate Source to nmr-load-save * fix: pass a copy to multiplet-analysis close * chore: temp fix to avoid load twice a molecule

jobo322 · 2022-12-22T14:08:29Z

done by 8aad5c6

lpatiny added the to discuss label Sep 26, 2022

lpatiny assigned jobo322 Sep 26, 2022

lpatiny added this to NMR and Cheminfo projects organisation Sep 26, 2022

lpatiny moved this to Todo in NMR and Cheminfo projects organisation Sep 26, 2022

lpatiny mentioned this issue Nov 17, 2022

Extra FileCollectionItem information cheminfo/filelist-utils#59

Open

lpatiny assigned hamed-musallam Nov 17, 2022

lpatiny mentioned this issue Nov 18, 2022

feat: fileCollectionFromFileArray cheminfo/filelist-utils#60

Closed

hamed-musallam added the priority label Nov 29, 2022

hamed-musallam mentioned this issue Nov 30, 2022

nodejs spectra-parser generating huge nmrium files NFDI4Chem/nmrxiv-nodejs-microservice#3

Closed

lpatiny mentioned this issue Dec 2, 2022

Improve source for jcamp URL #1976

Closed

4 tasks

lpatiny removed the priority label Dec 2, 2022

lpatiny added enhancement New feature or request and removed to discuss labels Dec 2, 2022

NRayya mentioned this issue Dec 12, 2022

Import CENAPT NFDI4Chem/nmrxiv#673

Closed

jobo322 moved this from Todo to In Progress in NMR and Cheminfo projects organisation Dec 16, 2022

jobo322 mentioned this issue Dec 20, 2022

feat: fileCollection as source in .nmrium files #2021 #2022

Merged

jobo322 closed this as completed Dec 22, 2022

Repository owner moved this from In Progress to Done in NMR and Cheminfo projects organisation Dec 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generalize data source #1775

Generalize data source #1775

lpatiny commented Sep 26, 2022

lpatiny commented Nov 17, 2022 •

edited

Loading

hamed-musallam commented Nov 24, 2022

lpatiny commented Nov 29, 2022

hamed-musallam commented Nov 29, 2022 •

edited

Loading

lpatiny commented Nov 29, 2022

hamed-musallam commented Nov 29, 2022

lpatiny commented Nov 29, 2022

lpatiny commented Dec 1, 2022 •

edited

Loading

lpatiny commented Dec 1, 2022

CS76 commented Dec 1, 2022

hamed-musallam commented Dec 1, 2022 •

edited

Loading

CS76 commented Dec 1, 2022 •

edited

Loading

hamed-musallam commented Dec 1, 2022

hamed-musallam commented Dec 1, 2022 •

edited

Loading

hamed-musallam commented Dec 1, 2022

lpatiny commented Dec 2, 2022

hamed-musallam commented Dec 2, 2022

CS76 commented Dec 2, 2022

lpatiny commented Dec 2, 2022

jobo322 commented Dec 22, 2022

Generalize data source #1775

Generalize data source #1775

Comments

lpatiny commented Sep 26, 2022

lpatiny commented Nov 17, 2022 • edited Loading

hamed-musallam commented Nov 24, 2022

lpatiny commented Nov 29, 2022

hamed-musallam commented Nov 29, 2022 • edited Loading

lpatiny commented Nov 29, 2022

hamed-musallam commented Nov 29, 2022

lpatiny commented Nov 29, 2022

lpatiny commented Dec 1, 2022 • edited Loading

lpatiny commented Dec 1, 2022

CS76 commented Dec 1, 2022

hamed-musallam commented Dec 1, 2022 • edited Loading

CS76 commented Dec 1, 2022 • edited Loading

hamed-musallam commented Dec 1, 2022

hamed-musallam commented Dec 1, 2022 • edited Loading

hamed-musallam commented Dec 1, 2022

lpatiny commented Dec 2, 2022

hamed-musallam commented Dec 2, 2022

CS76 commented Dec 2, 2022

lpatiny commented Dec 2, 2022

jobo322 commented Dec 22, 2022

lpatiny commented Nov 17, 2022 •

edited

Loading

hamed-musallam commented Nov 29, 2022 •

edited

Loading

lpatiny commented Dec 1, 2022 •

edited

Loading

hamed-musallam commented Dec 1, 2022 •

edited

Loading

CS76 commented Dec 1, 2022 •

edited

Loading

hamed-musallam commented Dec 1, 2022 •

edited

Loading