Non-standard _part is used within src/99-appendices/06-meg-file-formats.md example

[yarikoptic]

and nowhere described otherwise:

$> git grep _part                              
src/99-appendices/06-meg-file-formats.md:sub-control01_ses-001_task-rest_run-01_part-01_meg.fif
src/99-appendices/06-meg-file-formats.md:sub-control01_ses-001_task-rest_run-01_part-02_meg.fif

and it is to annotate two parts of the split longer file. In #371 I mention e.g. that _part was (or still is) suggested in multiple BEPs (BEP001 (multiple contrasts) and BEP004 (SWI)) but likely for a different purpose.

I am not sure if we have ATM in BIDS any agreement on annotation of multiple portions (split arbitrarily in temporal dimension) of a larger thing, besides a _run which is different semantically. That is why I was not sure how to address this issue.

attn: @bids-standard/derivatives-electrophys

[jasmainak]

ping @teonbrooks I think he might know the origins / historical background. Agree that it should be clearly described and was perhaps included a bit too hastily. But MNE now uses this convention, so changing would not be as simple as editing the specification.

[agramfort]

in current mne-python a file called 'split_raw_meg.fif' would be split in 'split_raw_part-01_meg.fif' 'split_raw_part-02_meg.fif' we must support such split files as otherwise certain MEG datasets cannot be shared. what do you suggest?

…

[teonbrooks]

there's a section in the appendix that explains the use of part. it was decided back as a bep and before merged in the main spec:
https://bids-specification.readthedocs.io/en/stable/99-appendices/06-meg-file-formats.html#neuromagelektamegin

as @agramfort mentioned, not having it would break datasets, especially raw MEG files that were saved to .fif with file size > 2GB.

back story is that .fif has a file limit constraint and the split files are a linked to one another. naming convention in the MEG bep led to use part.

[effigies]

We have a competing use for part- being proposed in BEP001/#424.

If this is just an accommodation of a file format limitation, then perhaps hacking the extension would make more sense... What about .part01.fif, .part02.fif? That more clearly associates the modification with the file format, keeps the stem the same when converting between formats, and doesn't conflict with the other proposal for part-.

GIFTI and CIFTI-2 files have double-extensions, so it would not be unique to FIF, although it would be a BIDS innovation rather than something determined by the FIF standard itself.

[yarikoptic]

I would also vote for wrapping format limitations into the format file extension. Different formats (and tools) could support different ways on how they expect split files to be annotated/specified.

is every such split .fif file is still a "legit" .fif file? then I would consider git-annex behavior of not including part* into the extension for its purposes correct:

$> git annex add balsdjflkajsdlfkjasdf.part01.fif
add balsdjflkajsdlfkjasdf.part01.fif 
ok
(recording state in git...)
(dev3) 1 32429.....................................:Tue 03 Mar 2020 10:24:44 AM EST:.
(git-annex)lena:/tmp/bu[master]
$> ls -l balsdjflkajsdlfkjasdf.part01.fif
lrwxrwxrwx 1 yoh yoh 116 Mar  3 10:24 balsdjflkajsdlfkjasdf.part01.fif -> .git/annex/objects/j7/50/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e.fif/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e.fif

But bids tools and validator should explicitly account for format-appropriate extension and allow for multiple files in such cases. Hopefully none of the compressors etc would use _part-X which would collide with BIDS.

[jasmainak]

The files are "legit" on their own. You can separate them and they can be read in separately. However, the full recording contains both the files. The first file contains a pointer to the second file.

[agramfort]

Not sure what to suggest on the git-annex side. I am not knowledgable enough

[effigies]

I think the question RE git-annex (correct me if I'm wrong @yarikoptic) is whether a tool that sees randomgarbledmess.fif (the regular file) or sub-01_task-rest_meg.part01.fif (the symlink) will work just as well, or if it needs to become randomgarbledmess.part01.fif to be interpreted correctly.

Given that it's been fine with the part-* in the stem and not the suffix, I suspect this isn't a concern.

But how would the electrophys community (or at least MNE-Python) feel about moving part* to the suffix? Would this cause any significant problems?

[teonbrooks]

I thought the suffix has to correspond to the modality. wouldn't we need to part or the part-equivalent to be an entity?

[effigies]

Oh sorry, I meant extension, not suffix...

[agramfort]

But how would the electrophys community (or at least MNE-Python) feel

about moving part* to the suffix? Would this cause any significant problems? everything is possible. We can do changes / deprecations etc. can you give me an example of what would be better?

[robertoostenveld]

for reference, from the Elekta Neuromag Data Acquisition manual:

If the recorded raw data file size becomes larger than a predefined value (in normal configuration 2GB minus 100MB), the data file is closed and a new continuation file is created. The exact size of the file can vary somewhat depending on how the internal blocks exceed the limit. When this kind of a split measurement is saved, the first segment is given the name which user writes into the saving dialog. Rest of the segments have the same name with a dash and a running number appended (numbering starts from one).

So the Neuromag software itself generates files

basename.fif
basename-1.fif
basename-2.fif

According to https://mne.tools/dev/generated/mne.preprocessing.maxwell_filter.html this is also what the MNE software supports.

An example of this with .fif, -1.fif and -2.fif is given on https://megnord.org/2018/workshop_material/preprocess_MEG.html

[agramfort]

I am fine switching MNE to _meg-%d.fif convention although I though that ending with _meg.fif was mandatory.

…

[effigies]

The suggestion above was _meg.part*.fif.

[agramfort]

sure why not. Just make sure you don't rename the files directly as file names are written in each fif file to follow to the next part.

…

[jasmainak]

yeah you'd have to IO in MNE, MNE-BIDS + any other software which writes these files + datasets which already contain these files. And update the validator (although it doesn't seem to have checks for this at the moment). It's possible but is it really necessary? Perhaps better to do it earlier than later ...

[teonbrooks]

would this be part of a major/minor version release update to BIDS?
does the validator have versions tied to the versioning of the spec?

if not, this pattern will break datasets with split files. I echo what @jasmainak says. it would be OK for just reasons, and a clear plan. let's try to mitigate breakage as much as possible.

[yarikoptic]

would this be part of a major/minor version release update to BIDS?

I hope the fix one way or another should be a part of the next release. Note that the _part is not properly introduced, described, listed in entities table. Only an example of some "convention" includes it in the file names. AFAIK it was not a properly specified/documented entity within BIDS to start with.

Proper solution would be to either accept _part-<index> as a definition to introduce such sequential files composition (<index> vs <label> choice is kinda important to guarantee that the first file is properly identified) and thus limiting any other possible utility for _part; or ... agree on some suggested above approach. Tools can adjust and keep supporting older way until a new version of BIDS, or someone introduces version-specific handling in validator.

does the validator have versions tied to the versioning of the spec?

AFAIK - not . See e.g. bids-standard/bids-validator#683 . If validation is of a concern, such files could simply be ignored via .bidsignore.

[jasmainak]

Note that the _part is not properly introduced, described, listed in entities table.

But this is a separate issue. The entity table was not updated when merging MEG/EEG and this was fixed only afterwards: #198. If documentation is an issue, we can fix this

Only an example of some "convention" includes it in the file names. AFAIK it was not a properly specified/documented entity within BIDS to start with.

No, but part was part of the MEG specification from the beginning as far as I remember.

If validation is of a concern, such files could simply be ignored via .bidsignore.

What do you mean? How can you simply ignore split files in the standard? This is a large fraction of MEG recordings ...

[yarikoptic]

If documentation is an issue, we can fix this

So you know what to do, but might need to reach out to beps mentioned above to see if harmony could be reached so it wouldn't become a stumbling point for them

[robertoostenveld]

I worry a bit that this issue is getting larger than it needs to be - or than efficient for handling it. I re-read https://bids-specification.readthedocs.io/en/stable/99-appendices/06-meg-file-formats.html#neuromagelektamegin

On the one hand (1) there is a piece of example documentation in the appendix which appears inconsistent with the actual spec, i.e. part-<number> is an as-of-yet undocumented entity. On the other hand (2) there seems to be a need to clarify how subsequent Neuromag files are to be called.

1 could be resolved by removing the specific lines from the appendix. Note that this might not solve 2. But see (*)

2 needs to be resolved by looking at the wider ecosystem and involving (perhaps) the company, since we follow the manufacturers format specification and we want to ensure that files in BIDS are compatible with all software, not only MNE-Python but also Curry, BESA, FIeldTrip, BrainStorm, etc.

Regarding (*): it is not clear to me in the current appendix why it states "In the case of data runs exceeding 2Gb, the data is stored in two separate files ..." (which appears consistent with ther manufacturers specification) and then later the same issue seems to be solved by specifying it as "Naming convention ...". Why is the latter part needed? Would (1) and (2) not both be resolved if we remove these (apparently stray) lines:

Naming convention:

sub-control01_ses-001_task-rest_run-01_part-01_meg.fif
sub-control01_ses-001_task-rest_run-01_part-02_meg.fif

?

[sappelhoff]

I also re-read the neuromag elekta megin section and also stumbled over this part:

In the case of data runs exceeding 2Gb, the data is stored in two separate files:

sub-control01_ses-001_task-rest_run-01_meg.fif
sub-control01_ses-001_task-rest_run-01_meg-1.fif

Each of these two files has a pointer to the next file. In some software applications, like MNE, one can simply specify the name of the first file, and data will be read in both files via this pointer. For this reason, it is RECOMMENDED to rename and write back the file once read, to avoid the persistence of a pointer associated with the old file name.

Naming convention:

sub-control01_ses-001_task-rest_run-01_part-01_meg.fif
sub-control01_ses-001_task-rest_run-01_part-02_meg.fif

So it seems to me like both of these would be possible:

• *meg.fif + *meg-1.fif + ... + *meg-N.fif
• OR *_part-01_meg.fif + *_part-02_meg.fif + *_part-N_meg.fif

where the latter is recommended.

Also, looking at the regexp in the validator for MEG: https://github.com/bids-standard/bids-validator/blob/1888b87268c8f4bff26e073a96191dccca137e60/bids-validator/bids_validator/rules/file_level_rules.json#L205

... it seems like *_meg-\d+.fif is not valid, but *_part-\d+_meg.fif IS valid. Can somebody confirm?

Overall I think that meg-\d+.fif is nicer than the part-\d+_meg.fif convention ... but given that we already have the part convention and that it's been implemented in a major software package for a long time ... perhaps the most we can do is to improve the docs?

And closely look at new BEPs that want to make use of "part" ... and whether that makes sense at all.

[sappelhoff]

And to answer @robertoostenveld

Why is the latter part needed? Would (1) and (2) not both be resolved if we remove these (apparently stray) lines:

1. I think the latter part is not needed, but it was judged to be more clear and thus introduced into the spec ... and the validator
2. It may seem like a nice solution to simply remove _part from the spec and the validator ... but we'd have to adjust MNE-Python and existing datasets --> and it seems to me like a backwards incompatible change (borderline, because _part was never well documented)

So as I also conclude in #429 (comment), I think improving the docs and staying with _part in MEG may be a better way forward.

[yarikoptic]

We have a competing use for part- being proposed in BEP001/#424.

With all of the above, I think the way forward is just to continue with #424 which would introduce _part into the entity table with _part-<label>, where semantic for the label would be modality specific.
Since <label> is actually a superset which includes <index>, and there from what I hear it actually did not matter what value is stored since those files are "chained" internally, after #424 is merged, subsequent PR could clarify on _part in MEG modality, and we would be all set? or am I missing some problem which would prevent it?

[robertoostenveld]

Considering the NIDM Terms project as mentioned by @dbkeator in #423 , I think it would be unfavorable to use _part-<xxx> with different meanings, and with different values (index or label). This might also set a precedent that we might not want.

The current use for MEG is to separate different parts of a long recording (that due to file size limitations is in separate files). The proposed use in #424 would be for magnitude and phase, but simultaneously acquired and each not interpretable without the other. To me those are rather different.

Just for brainstorming sake and to get a feel for this...

Consider that we would do this for combined MEG/EEG recordings

• part-MEG
• part-EEG
• part-EOG
• (part-buttonpresses, or part-trigger)

Or combined recordings in iEEG

• part-ECoG
• part-sEEG

Or combined video (does not exist in BIDS yet, but might be added at one point)

• part-video
• part-audio

Or combined eye-tracking

• part-pupildiameter
• part-gazedirection

[effigies]

I generally agree with @robertoostenveld that a modality-sensitive meaning to part is not ideal, particularly in light of the conversation in #423.

With #460, it seems we're rejecting the proposal to move the multi-part FIF hack into the extension and instead making _part- more well-defined and give less of an impression of being an afterthought. I would really prefer not to give a format-specific hack such prominent standing in the specification, although I understand if the consensus is that we're compelled by backwards compatibility.

From the above discussion, I didn't get the impression that the consensus was that this was irrevocable, but a bit painful for some specific tooling. So I just want to clarify before we push forward and make it definitely a breach of backwards compatibility to change the meaning of part.

[sappelhoff]

Note that although I proposed #460 this does not mean that I strongly favor making _part- well defined. I merely want to drive discussion and think that's easier when having concrete proposals. #460 is one such proposal

[jasmainak]

Hi guys, as mentioned by @robertoostenveld we want to ensure that this proposal is supported by all software. Yet, my impression is that part- is not supported in any of these other software except MNE even though it is in the specification. Why? It tells me that making a change in the specification is not as simple as changing one line of document. You need to change software, change old datasets, change the validator. We need to be mindful of backwards compatibility. However, for this specific change, it's perhaps worth the pain.

[yarikoptic]

Well, in case of mne support in software happened before specification in bids. #460(#424) was filled with introducing support in software (heudiconv) in mind. And in general IMHO that should be the course of action - first bids and then software. In this particular case we are now trying to accommodate existing and future software support.

edit: replaced used by mistake #460 with #424. #460 is the clarification for MEG/EEG

[jasmainak]

Well, in case of mne support in software happened before specification in bids.

not really, the part- was in the specification before we moved the specification to github. It was an error to not include them in the entity table.

[effigies]

It was an error to not include them in the entity table.

The effect of this error was to hide this consequential aspect of the BEP from all but the most diligent reviewers. BEP-001 has been around a long time, and had part- been in the entity table, I suspect we would have had this discussion back then.

[yarikoptic]

part- was in the specification

it was only in a single example without any description in the text etc. With such a presence I could call any typoed field to be a "part of the specification" ;-)
but the point is not-important -- what is important IMHO is to arrive to specification while staying compatible with what is supported in software (in #460) and which would be inline with ongoing efforts to introduce _part "properly" for other modalities (as in #424).

[jasmainak]

sounds good, +1 to clarify part and then follow up with discussion/PR for a better convention.

[yarikoptic]

re @robertoostenveld

Consider that we would do this for combined MEG/EEG recordings

• part-MEG
• part-EEG
• part-EOG
• (part-buttonpresses, or part-trigger)

Or combined recordings in iEEG

• part-ECoG
• part-sEEG

BTW, I was thinking about those while looking at #460, but those seems to would be represented as a suffix _meg, _eeg, etc, thus giving _part ability to annotate some other "parts" as different "modalities", not really parts of the same modality ... anyways, with my comment that if <index> is not strictly required to be an index, but could be a <label>, that label could incorporate index in case of some modalities with special treatment by supporting software. After all even a <label> is an "key" into an "associative array" of the entity.

re @jasmainak

+1 to clarify part and then follow up with discussion/PR for a better convention.

clarification: in my comment and != and then, it was more of at the same time, so IMHO #460 must not be considered in isolation.

[sappelhoff]

So currently I see these three options:

1. drop the current _part- from the spec, adjusting the validator, existing software, and datasets, and RECOMMEND a different way to format split .fif files that does not need _part-

   1. could be considered backward incompatible (border case)
   2. would "free up" _part- for other BEPs to use freely

2. document _part- for .fif files and thus "fix" the error that it was never properly documented in the first place.

   1. backward compatible
   2. probably destroys some BEP work that wants to introduce _part- with a different meaning (see e.g., BEP01 or [ENH] Add part entity for complex-valued data #424 )
   3. this is currently suggested as an option in [FIX] Drop _part- reference from example, introduce _split- entity #460

3. better document _part- in the whole spec, but designing it as such that other BEPs can re-use it

   1. backward compatible
   2. would probably mean that a single key-value entity takes on significantly different meanings, which would be the first of these cases in BIDS (potentially problematic, see Non-standard _part is used within src/99-appendices/06-meg-file-formats.md example #429 (comment) and Non-standard _part is used within src/99-appendices/06-meg-file-formats.md example #429 (comment))

---

Some of my thoughts:

• If there were no BEPs or PRs that wanted to use _part-, I guess that most people would be in favor of option 2
• could those BEPs or PRs that want to use _part- simply pick another word to form a new entity?

[yarikoptic]

Thank you @sappelhoff for the nice summary! Let me just comment on

If there were no BEPs or PRs that wanted to use _part-, I guess that most people would be in favor of option 2

AFAIK - most people wouldn't care enough since would not have concrete use cases. Only MEEG folks would, biased by "it is already how we do it".

could those BEPs or PRs that want to use _part- simply pick another word to form a new entity?

I guess they could. Any suggestions on a "word"? only _split- comes to mind, but then it is actually more semantically inline with use of current _part in MEEG ;) (in other cases, _parts might be coming from different files, e.g. DICOMs, thus not per se split from one larger single entity like what is happening with MEEG files; so could be adopted for the painful 1.).

Re all the pointers to #423 you reference as problematic for 3. - I am yet to understand where problem would arise.

[robertoostenveld]

thanks for the constructive discussion and good summary of viewpoints.

If there were no BEPs or PRs that wanted to use _part-, I guess that most people would be in favor of option 2

I believe this is more fundamental than "MEEG folks versus fMRI folks". Based on principles we cannot have the situation that extensions to the specification invalidate previous versions of the spec (unless we switch from 1.x to 2.x). Some people shouting louder, versus other people not (visibly) caring enough, would still not justify making such changes.

We will have to live with the situation that certain BIDS "terms" (like entities) of the specification are claimed first by one domain, causing them not to be available in another domain. Reuse of entities/terms is nice and desirable, replacing their definition is incompatible, making their definition context-dependent is undesirable.

That being said, since part-<index> is only in the "MEG examples appendix" (also in the google doc) and not in the specification chapter itself, nor in the entities table, I think that it is also fair to consider solution "1" or "3" from @sappelhoff.

Regarding solution 1: I don't see a trivial way of recommending a different way to format split other than the current one (using part).

Perhaps the best wisdom would be to go for option 3 and specify somewhere that part can relate either to different parts of simultaneously acquired data, or different parts of sequentially acquired data. Although that does leave me with this consideration about ambiguity in the extensions to the MRI parts of the specification.

[yarikoptic]

Regarding solution 1: I don't see a trivial way of recommending a different way to format split other than the current one (using part).

_split-<index>?

[sappelhoff]

Regarding solution 1: I don't see a trivial way of recommending a different way to format split other than the current one (using part).

There is also @effigies's suggestion to move if to the suffix #429 (comment)

*.part01.fif

[yarikoptic]

I guess the use of _entity-XX vs .entityXX (regardless of name) decision should be based on either such a split is documented at the level of BIDS or data file format.
As I have mentioned in a comment to #460 there is "indexable metadata entity" being WiP within BEP001 and I think it might be relevant here as well. If some _entity- to generally annotate split files was to be introduced (khe khe), then it could be nicely formalized at the level of BIDS in a sidecar .json to describe metadata related to the split (requires manual join, or chained by the file format, etc). So it sounds like a possible nice BEP (or PR) to introduce a generic support based at least on the set of use cases at hands. One other potential use-case could be .nwb (HDF5 internally) files which due to their nature of embedding lots of individual components could grow huge and someone might decide to decompose them into smaller components -- possibly not representable at BIDS level, but could be just some individual portions of the file linked into some "main" .nwb.

[jasmainak]

If you move the part* to the suffix, you might break scripts which rely on splitting filenames into the extension and the rest. It might need special handling in several different softwares. Changing part- to some other name is totally fine and might be easier to adapt.

[robertoostenveld]

split suggests an active step in the processing of splitting the data (as in break, chop, cut, hew, lop, cleave, snap, crack), i.e., as if it were one file before and multiple after. The use of part feels more neutral to me. If there were a temporal gap between the data in "file1" and "file2", part would be a better descriptor IMHO. Or perhaps we should consider section, bit, slice, chunk, lump, hunk, wedge, fragment, scrap, segment, piece; portion, share, fraction, division.

But actually, before we decide on this:

1. do we feel that it is needed to have a separate entity for splitting data over time and splitting data over another feature (e.g. mag/phase, or audio/video for a camera recording with synchronized audio that is in a separate file)? Or could a single entity (with a string as the value) be used for both?
2. should we not try to get some more Neuromag MEG users (i.e. the "wisdom of the crowd") involved in this?

If (based on "2") the Neuromag users/representatives are happy with using part-<label> where label can be a a string-based representation of something that is ordered (e.g. "01", "02"), then I think we have less reason to decide for two entities ("1").

[robertoostenveld]

BTW I don't like the idea of coding metadata in the suffix, as in meg.part01.fif or meg-1.fif. I don't think that BIDS will benefit from going in this direction in general.

[effigies]

BTW I don't like the idea of coding metadata in the suffix, as in meg.part01.fif or meg-1.fif. I don't think that BIDS will benefit from going in this direction in general.

If we are considering the general case where files might be split for space reasons, and not a format specific hack, then I agree. As currently written, this is a way to accommodate FIF limitations, which seems appropriate to the extension, IMO.

[teonbrooks]

if we were to replace part with something else, I would be in favor of split or segment. we would need to figure out how to deal with the deprecation and datasets that might have the part entity used. I think MNE might be the only package with the renaming support, and there's an issue tracking alongside this one now

[sappelhoff]

TL;DR -> let's fix this _part- "bug" pragmatically and discuss new entities when the need arises.

do we feel that it is needed to have a separate entity for splitting data over time and splitting data over another feature (e.g. mag/phase, or audio/video for a camera recording with synchronized audio that is in a separate file)? Or could a single entity (with a string as the value) be used for both?

I think it'd be best to have an entity that splits large files into several chunks and to use that entity not only for MEG, but for all kinds of files that need to be split. Practically though, the currently most used application for this would be FIFF files.

IMHO entities to split across features may be considered separately from this "bug fix" that is the topic of this issue (_part- not having been introduced properly).

At a first glance though, I don't like "overloading" the entities -> taken to the extreme, the entities lose their original purpose of being a quick to grasp and intuitive explanation of the file content. If the entities can take on several meanings depending on contexts (e.g. split over time OR features) then that'd be a step towards the extreme I mentioned.

should we not try to get some more Neuromag MEG users (i.e. the "wisdom of the crowd") involved in this?

I trust that fieldtrip or MNE-Python folks have a good grasp of the situation, so I don't think this is necessary.

[robertoostenveld]

In the case of renaming part to something else for MEG, I would prefer segment over split.

From the FieldTrip-perspective, I am the only one here in the discussion. However, MNE-Python is more used by Neuromag labs, so perhaps some MNE contributors can help to decide here. Let me ping @agramfort, @dengemann, @jasmainak, @larsoner for this.

[sappelhoff]

This issue #429 has now branched out across many repos, issues, and PRs. This post is intended as a short summary.

• THIS issue Non-standard _part is used within src/99-appendices/06-meg-file-formats.md example #429 is the main issue.
• the PR [FIX] Drop _part- reference from example, introduce _split- entity #460 is intended to close this issue
• apart from the conversation in this issue, the BIDS maintainers have discussed this in their biweekly online meeting
• further discussion has happened in the MNE Python repository under this issue: BIDS related: potential change of .fif file split naming mne-tools/mne-python#7776
• as part of the main PR intended to close this issue ([FIX] Drop _part- reference from example, introduce _split- entity #460), there are several sub-PRs necessary:
  • update BIDS validator (https://github.com/bids-standard/bids-validator/pull/960)
  • update MNE-Python ([MRG] split_naming='bids' changes from _part-%d to _split-%d mne-tools/mne-python#7794)
  • update MNE-BIDS ([MRG] _part- -> _split- to conform with BIDS  mne-tools/mne-bids#417)

[jasmainak]

you're the best @sappelhoff :)

[jasmainak]

@robertoostenveld I'm happy with the solution and the other MNE developers have also expressed their support to this proposal.

[robertoostenveld]

Very glad to see the constructive community involvement on the intersections between BIDS and analysis software. That is where the wisdom of the crowd can be found!