Suport 3D FLIM data #41
Comments
|
More thoughts. For scanning data, adding the number of rows and columns in Let's say we have a 256x512 image. Each pixel can be identified by a linear index From there, building a mask for |
|
We may want to introduce provisional support for FLIM data in Photon-HDF5 v0.5. As we gain experience with it, we can see if there are any issues and adjust the format in a subsequent version. To this goal the proposal in my second comment, i.e. using |
|
I propose adding:
Then, if data contains several Z planes, each plane will be in a different We should also encode the step in X-Y-Z, for example with the tuple:
|
|
Hello,
My conclusion on this is that one should keep the raw data as is it now (with extra metadata to guess what pixel or line each timestamp is related to), but then add a group such as "histograms" where histograms (from phu file of FLIM) could be stored (temporary or not). In the same way, I would add an analysis group where images extracted from lifetime (for instance) could be stored with the metadata of the analysis...this is maybe going too far. Finally, I'm working as well at the interface of electron microscopy and spectroscopy where datacubes are often generated (with often a spectrum (photon or electron energy) for each pixel). This exactly corresponds to the issue of FLIM data. There is an extensive project currently running on analysis of such datacubes (hyperspy) mostly devoted to electron microscopy but with numerous general tools dealing with datacubes analysis (fitting, filtering...). Maybe it could be interesting to see what could be done such as a possibility in hyperspy to load hdf5photon raw data and then to process the datacube there... |
|
Hi @seb5g, thanks for reaching out! I totally support the idea of finding synergies between different projects and applications. No, I have not worked on FLIM support except for laying down some preliminary ideas here. You are right, we designed Photon-HDF5 for storing per-photon data. So, currently, there is no place to save TCSPC histograms. Similarly, in FCS, people compute and analyze the cross-correlation (ACF/CCF) but we don't store those curves. It is certainly true that most FLIM analysis is done on histograms but you can in principle do phasor analysis directly on timestamps. On the other hand datasets can be big so some people don't store timestamps at all. We view Photon-HDF5 as a long-term storage of raw data that does not change often. For this reason we have not added support for processed data (e.g. histograms, or burst data in smFRET for example). While the per-photon data can fit in a common structure for a large range of measurements, processed data is very specific for each type of measurement and is difficult to support in a common format. So, in terms of supporting FLIM in Photon-HDF5, I think we should focus on storing the raw timestamp data with enough metadata around it, as you said. I am totally open to suggestions on how to structure the layout in the most effective way. I am not sure about the TCSPC histograms. It may be of potential interest also for other TCSPC applications. So, I may be convinced to accept a general non FLIM specific proposal to add TCSPC histograms (maybe a 2D array of "nanotimes" histograms in each photon_data group?) to Photon-HDF5. For the images, I would put them in a second HDF5 file, probably together with histograms. You can keep these two files with the same name and different suffixes (e.g. The "processed FLIM" format and Photon-HDF5 can be made inter-operable. For example they can share fields in I had a look at hyperspy, it is a very interesting project. I see how it can be very useful in many applications. As you said, hyperspy should preprocess the Photon-HDF5 data in order to make it fit in their data model. Regarding adding more file support to phconvert, I'm looking forward to your PR! 👍 |
|
I am still interested in adding support for FLIM to Photon-HDF5. I would like to participate in any discussion regarding the required metadata. With regard to the histograms, we could take the following approach. If there is no existing standard, we could develop a convenient way to organize the histograms within an HDF5 file, keeping it separate from the Photon-HDF5 specification. We could keep it in a separate file at this point. Once we feel it is ready and useful, we can determine whether it is better to keep it in a separate file or add it to the Photon-HDF5 specification. |
|
@talaurence, sounds like a good plan. If you like, we can create new a repo within the Photon-HDF5 organization to draft these specs. |
|
For a separate lifetime imaging project, I am using the HDF5 file format via PyTables. The size of the data cube is 1024X1024X4096 (x pixels, y pixels, lifetime histogram). I placed the entire data cube in a "carray", and I can access any part of it at will. I did not clip out any zeros - I just put the whole array in. Using the "blosc" compression filter, the size of the data file went from 8 Gbytes to ~900 MBytes, which is much more manageable. I think we can keep this format fairly simple: let's just make one big array, and let the HDF5 library do the hard work. We can add some metadata definitions, and leave it at that. |
|
Yes - let's create a new repo. |
|
I'm not at all sure it is wise to keep one big cube of data. The way an
analysis is done is by extracting info from the 1D traces. In previous
work, I was storing these as such with info on the related pixel in the
name of the 1d trace. That made exploration of data easier like in a tree
view such as hdfview, setting up analysis on1 trace and then looping over
the pixel to reconstruct an image.
I now think the pixel should not be absolutely coded in the name but
probably in a line and col metadata
Sébastien
Le 6 déc. 2017 01:18, "talaurence" <[email protected]> a écrit :
For a separate lifetime imaging project, I am using the HDF5 file format
via PyTables. The size of the data cube is 1024X1024X4096 (x pixels, y
pixels, lifetime histogram). I placed the entire data cube in a "carray",
and I can access any part of it at will. I did not clip out any zeros - I
just put the whole array in. Using the "blosc" compression filter, the size
of the data file went from 8 Gbytes to ~900 MBytes, which is much more
manageable. I think we can keep this format fairly simple: let's just make
one big array, and let the HDF5 library do the hard work. We can add some
metadata definitions, and leave it at that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#41 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGTcI6hqIHARr-0kxdgFAd2MtDiaBxIpks5s9d1DgaJpZM4OgBg0>
.
|
|
I am not sure I understand the problem with a large data cube. If we do not use that format, we will have hundreds or thousands of arrays and groups within the file with some sort of code for rows and columns, making it much more confusing. The HDF5 format was created in part in order to deal with large data sets such as this. Storing the entire array on disk in a single array does not mean that one has to load it into memory that way. For example, if you want a lifetime histogram of a single pixel, the following command would retrieve it from the file: If you want a whole row: If you want to get a lifetime histogram summed over a region: I think this would be a much cleaner storage format. |
|
Well, said like this, I kind of agree, but imagine now you want to scan
over a region that is not a square or rectangle but a given set of
representative pixels or an union of smaller area, then your description is
lost. In my sense, the spirit of such a thing as a unified file format is
to be able to expand on things/applications we didn't foresee at the
beginning. In the case of FLIM I do agree mostly it will be images but I
want to use that format for all hyper-spectral (or temporal) kind of signal
and applications (I have three different examples right here in my lab).
2017-12-06 14:30 GMT+01:00 talaurence <[email protected]>:
… I am not sure I understand the problem with a large data cube. If we do
not use that format, we will have hundreds or thousands of arrays and
groups within the file with some sort of code for rows and columns, making
it much more confusing. The HDF5 format was created in part in order to
deal with large data sets such as this. Storing the entire array on disk in
a single array does not mean that one has to load it into memory that way.
For example, if you want a lifetime histogram of a single pixel, the
following command would retrieve it from the file:
lifetime_hist = self.lifetime_image[row,col,:]
If you want a whole row:
row_hists = self.lifetime_image[row,:,:]
If you want to get a lifetime histogram summed over a region:
sub_array = self.lifetime_image[row_min,row_max,col_min:col_max,:]
lifetime_plot = np.sum(sub_array,axis=(0,1))
I think this would be a much cleaner storage format.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#41 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGTcI0kSVaOU12WTJheo7cjLHZgbBnX3ks5s9pbigaJpZM4OgBg0>
.
|
|
I agree with @talaurence. Once you have the array you can apply advanced indexing to select ROI in any dimensions you want (numpy advanced indexing allows using boolean masks or list on indices, for each dimension). The alternative of having milions of 1D arrays does not scale up, even performance wise. These datasets could be loaded not only as numpy arrays but also as hyperspy datasets or xray arrays so that dimensions are labeled. I have hard time imagining an application where saving the TCSPC histograms as N-D arrays (with appropriate chunking) would not be convenient. @seb5g, can you explain your application and why would not fit this model? |
|
I'm back on the subject as I finally reached the point where I will have to save custom FLIM data. Concerning the discussion above, I'm a bit lost. If we follow the first specifications you derived in this issue, only photons are saved and no (reconstructed) FLIM data. I think it to be the best. Indeed, to reconstruct FLIM one need to specify a time_bin that is not always the minimal possible. So this is up to the guy processing the data. If I summarize, you propose to add:
That said, did you specify the new array containing the linear index representing the "position" in the image. I think you said we should create a new array but then you mentioned the detector array. This last would collide with other detector definition. detectors/linear_index: linear index within the image (you get (X, Y) position from the image shape) |
|
We're working on an open source framework for laser scanning microscopy, including FLIM (not released just yet, but here is the overview: https://loci.wisc.edu/software/openscan). I agree with the view that FLIM histogram data, if stored, should go in a separate format. It has many more requirements (e.g. it may be organized in a time series and/or Z stack, or have other dimensions), and it is much more similar to microscopy image data than to (TC)SPC timestamp data. But it would be great if Photon-HDF5 could store the photon timestamp data from a FLIM experiment without losing information. The most important missing feature, if I am not mistaken, is the ability to store "marker" events. (It seems to be the only fundamental feature missing compared to BH In a typical LSM TCSPC-FLIM system where the laser scanning is performed by a separate device, the scanner sends some combination of frame, line, and pixel "clock" or "marker" signals (not necessarily all 3) to the TCSPC device, which records the timestamps of these events along with the photons. These are used (together with user-configurable delay adjustments and other parameters) to assign photon events to pixels and to exclude photon events occurring during horizontal and vertical retrace. I think it makes a lot of sense to store the hardware-generated marker timestamps, rather than interpreted pixel locations of photons, because in theory the latter can be adjusted in postprocessing (e.g. to correct for scan phase). And it's always good to keep the raw data in case one needs to investigate possible bugs in processing code: even just assigning photons to pixels has a lot of edge cases, especially when trying to support a wide range of hardware setups! I see that the "latest" documentation mentions in passing recording markers as special detectors, but as far as I can tell it is not yet specified how to indicate in the file that a given detector is a marker channel. (I would appreciate pointers if I'm missing something.) Also, I'm not sure if it is a good idea to store marker timestamps in the same array(s) as photons, because that would mean any code reading files will need to know about markers. In addition, markers don't have nanotime even if the photons do. It might be cleaner to store marker timestamps in a separate, optional, array, or set of arrays. This has the additional advantage of making it easier to write code that processes photons according to markers, especially if the time offset between markers and photons can range from positive or negative. Once we can record marker timestamps, it will probably make sense to record the necessary metadata to assign photons to FLIM pixels: this may include pixels-per-line, lines-per-frame, pixel rate (critical if pixel marker not used), line delay (or time offset between markers and photons), etc. This way, it will be possible to generate FLIM histograms without any external data. But just being able to record markers might allow me to stop writing I don't have anything against also supporting recording the pixel coordinates of each photon; it may be quite useful in some workflows. Would there be any interest if I were to come up with a more concrete proposal for how to record marker events? Or are there any existing plans for this that I didn't find? |
|
I've included picoquant hardware in an open source framework called pymodaq
http://pymodaq.cnrs.fr that stands as Modular Data Acquisition with Python.
Every hardware can also be included as a light plugin. I think lsm falls
within this category as i use pymodaq for Flim and many other type of
experiment. Maybe we could build something out of your soft and mine?
Le mer. 20 nov. 2019 20:11, Mark Tsuchida <[email protected]> a
écrit :
… We're working on an open source framework for laser scanning microscopy,
including FLIM (not released just yet, but here is the overview:
https://loci.wisc.edu/software/openscan).
I agree with the view that FLIM histogram data, if stored, should go in a
separate format. It has many more requirements (e.g. it may be organized in
a time series and/or Z stack, or have other dimensions), and it is much
more similar to microscopy image data than to (TC)SPC timestamp data.
But it would be great if Photon-HDF5 could store the photon timestamp data
from a FLIM experiment without losing information. The most important
missing feature, if I am not mistaken, is the ability to store "marker"
events. (It seems to be the only fundamental feature missing compared to BH
.spc files and PicoQuant .ptu files, aside from metadata details.)
In a typical LSM TCSPC-FLIM system where the laser scanning is performed
by a separate device, the scanner sends some combination of frame, line,
and pixel "clock" or "marker" signals (not necessarily all 3) to the TCSPC
device, which records the timestamps of these events along with the
photons. These are used (together with user-configurable delay adjustments
and other parameters) to assign photon events to pixels and to exclude
photon events occurring during horizontal and vertical retrace.
I think it makes a lot of sense to store the hardware-generated marker
timestamps, rather than interpreted pixel locations of photons, because in
theory the latter can be adjusted in postprocessing (e.g. to correct for
scan phase). And it's always good to keep the raw data in case one needs to
investigate possible bugs in processing code: even just assigning photons
to pixels has a lot of edge cases, especially when trying to support a wide
range of hardware setups!
I see that the "latest" documentation mentions in passing recording
markers as special detectors, but as far as I can tell it is not yet
specified how to indicate in the file that a given detector is a marker
channel. (I would appreciate pointers if I'm missing something.) Also, I'm
not sure if it is a good idea to store marker timestamps in the same
array(s) as photons, because that would mean any code reading files will
need to know about markers. In addition, markers don't have nanotime even
if the photons do. It might be cleaner to store marker timestamps in a
separate, optional, array, or set of arrays. This has the additional
advantage of making it easier to write code that processes photons
according to markers, especially if the time offset between markers and
photons can range from positive or negative.
Once we can record marker timestamps, it will probably make sense to
record the necessary metadata to assign photons to FLIM pixels: this may
include pixels-per-line, lines-per-frame, pixel rate (critical if pixel
marker not used), line delay (or time offset between markers and photons),
etc. This way, it will be possible to generate FLIM histograms without any
external data.
But just being able to record markers might allow me to stop writing .spc
files.
I don't have anything against *also* supporting recording the pixel
coordinates of each photon; it may be quite useful in some workflows.
Would there be any interest if I were to come up with a more concrete
proposal for how to record marker events? Or are there any existing plans
for this that I didn't find?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#41?email_source=notifications&email_token=ABSNYI7DJPLEH6IOBNHDERLQUWDVRA5CNFSM4DUADA2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEET6MIQ#issuecomment-556262946>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSNYI3T4PE3ODL6FTSVNITQUWDVRANCNFSM4DUADA2A>
.
|
I would be interested in a proposal for adding marker arrays to the photon-hdf5 format. I agree with your assessments. I have also been developing software for FLIM applications using Picoquant hardware. So far we have been performing sample scanning, but we plan to move on to laser scanning soon. I would be interested in comparing your software with mine. I can put mine on GitHub, but there is an approval process I need to go through here. I will start that process and put it on GitHub as soon as I can. There is a "ScopeFoundry" microscope development software based on Python from LBNL. I believe they have some options for Picoquant hardware as well. That may be of interest to all of our efforts as well. |
|
Glad to hear I'm not the only one interested in this. Our code is in C and C++ and we've only used a Becker & Hickl device so far, although I have studied the PicoQuant API and data format. I am contemplating a separate library for streaming TCSPC data handling (including live histogramming), but I don't know how far I'll pursue that. I'll open a separate issue with my proposal for storing marker timestamps, since it feels a little diverged from the title of this issue. |
As for @talaurence request, we should think of how to handle 3D FLIM data.
Initially we thought that this case could be covered by a single
photon_datagroup and then having (x, y, z) labels.But in the 3D case having the whole 3D data in a single array is not convenient. I think it would make sense to have a
photon_datagroup for each z, and then add an array of (x, y) positions to reconstruct a slice from a single array of timestamps/nanotimes.We could use the
detectorsarray to give the (x, y) coordinates or we can define a new array (coordinates,positions?). I tend to prefer the latter option of adding a new array. But we have to think if this case needs some special configuration in/setup.The text was updated successfully, but these errors were encountered: