highlighted pixel display error

[FrancaCassol]

Hi,
I have a strange phenomenon by displaying highlight pixels: the pixels that are highlighted do not corresponds to the given mask. This is strange because in other entries of my notebook , it works very well. I do not really understand what is happening

Thanks

[maxnoe]

Without the data, we will not be able to reproduce this.

Can you at least print image and mask or provide it as files?

Unrelated: This does not look like the LST Camera Geometry you sue is in the coordinate frame ctapipe expects for coordinate transformation (HESS / SIMTEL), but in the MAGIC / FACT reference frame.

[FrancaCassol]

Hi @maxnoe,
you find the file in https://www.cppm.in2p3.fr/~cassol/pedestal.hdf5

I read it as following (let me know if there is a better way):

from ctapipe.io.containers import PedestalContainer
from ctapipe.io.hdf5tableio import HDF5TableWriter, HDF5TableReader

data = PedestalContainer()
with HDF5TableReader('/astro/users/cassol/soft/python/lstchain-test/pedestal.hdf5') as h5_table:
    
    assert h5_table._h5file.isopen == True

    count = 0
    ped_charge = [] 
    ped_std = []
    for ped_data in h5_table.read('/tel_0/pedestal', ped_data):

        print(count)
        
```

[FrancaCassol]

Unrelated: This does not look like the LST Camera Geometry you sue is in the coordinate frame ctapipe expects for coordinate transformation (HESS / SIMTEL), but in the MAGIC / FACT reference frame.

Unfortunately I have not idea of the difference, this is the geometry I created for LST1. What do you see wrong with that?

[maxnoe]

In the hess / simtel reference frame, the flat side should be up.

[maxnoe]

Coordinate transformations from Camera to Sky and from Sky to Camera will probably be wrong.

[maxnoe]

@FrancaCassol I'm sorry, but could you expand the code snippet, so that it reproduces the plot?

[FrancaCassol]

Yes, sorry.

chan =0
image = ped_data.charge_std
mask=  ped_data.charge_std_outliers 

fig = plt.figure(300,figsize=(16, 5))

# plot charge with higlight mask
ax = plt.subplot(1, 2, 1)
disp = CameraDisplay(event.inst.subarray.tels[0].camera)
disp.image = image[chan] 
disp.highlight_pixels(mas[chan],color='g', linewidth=2, alpha=0.75)
disp.cmap = plt.cm.coolwarm
disp.axes.text(2.4, 0, 'charge std', rotation=90)

disp.add_colorbar()

## plot mask
ax = plt.subplot(1, 2, 2)
disp = CameraDisplay(event.inst.subarray.tels[0].camera)
disp.image = mask[chan] 
disp.highlight_pixels(mask[chan],color='w', linewidth=2, alpha=0.75)
disp.cmap = plt.cm.coolwarm
disp.axes.text(2.4, 0, 'mask', rotation=90)
disp.add_colorbar()

[maxnoe]

Please: Provide a minimal example, that I can just copy / paste that runs and produces the plot!

E.g. I have no idea, where event comes from.

The script should "just work" and produce the plot, where you suspect the error.

[thomasgas]

Quick look:

disp.highlight_pixels(mas[chan],color='g', linewidth=2, alpha=0.75)

mas[chan] should be mask[chan]?

[FrancaCassol]

Coordinate transformations from Camera to Sky and from Sky to Camera will probably be wrong.

So, there is a wrong global rotation?

[maxnoe]

No, the transform is a reflection. I think it is x -> -y, y -> -x.

See discussion here: #255

Compare e.g. with CameraGeometry.from_name("LSTCam")

[FrancaCassol]

Yes

The script should "just work" and produce the plot, where you suspect the error.

you find a it in the notebook in https://www.cppm.in2p3.fr/~cassol/Test_code.ipynb

[maxnoe]

The problem is, that you mask has dtype uint8, not bool.

Integers will be interpreted as pixel ids, not as boolean mask. This is not a bug in ctapipe, you must make sure, that the mask has the correct type (boolean).

I don't know, why it was saved as uint8 and not as bool

[maxnoe]

This is the image when I add .astype(bool):

[maxnoe]

Also numpy will interprete ints as indices, not a mask. So instead of all the values where you have True or False, you get the index 0 or 1:

In [3]: a = np.arange(5, 10)

In [4]: a
Out[4]: array([5, 6, 7, 8, 9])

In [5]: mask = np.array([0, 1, 0, 1,  0])

In [6]: a[mask]
Out[6]: array([5, 6, 5, 6, 5])

[FrancaCassol]

Wonderful, an other mystery solved :-)

I write them with the HDF5TableWriter, it seems it does not preserve the bool ...

[maxnoe]

ctapipe/ctapipe/io/hdf5tableio.py

Lines 17 to 31 in e4030e3

	PYTABLES_TYPE_MAP = {
	"float": tables.Float64Col,
	"float64": tables.Float64Col,
	"float32": tables.Float32Col,
	"int": tables.IntCol,
	"int8": tables.Int8Col,
	"int16": tables.Int16Col,
	"int32": tables.Int32Col,
	"int64": tables.Int64Col,
	"uint8": tables.UInt8Col,
	"uint16": tables.UInt16Col,
	"uint32": tables.UInt32Col,
	"uint64": tables.UInt64Col,
	"bool": tables.UInt8Col,
	}

It seems bool is explicitly converted to uint8

[FrancaCassol]

I see, is there a specific reason for that? I see that the BoolCol exists (@kosack ?)

[maxnoe]

See proposed solution here #1083

[maxnoe]

@FrancaCassol the fix is now merged, can you try again?

[FrancaCassol]

It works! :-)