PicketFence on Elekta Agility MLC

[calculusbook]

I am having troubles trying to interpret the following results produced by the PicketFence module.
The code in Snippet 1 is to perform the combined analysis on a PicketFence pattern

from pylinac.picketfence import PicketFence, MLC
pf_img = r"data/23321312323213.PicketFence.dcm"
pf = PicketFence(pf_img, mlc = MLC.AGILITY)
pf.analyze(tolerance=1.5, action_tolerance=1.0)
pf.plot_analyzed_image()
print(pf.results())

Here is the result:

Picket Fence Results:
Gantry Angle (°): 0.0
Collimator Angle (°): 0.0
Tolerance (mm): 1.5
Leaves passing (%): 100.0
Absolute median error (mm): 0.028mm
Mean picket spacing (mm): 29.8mmn
Picket offsets from CAX (mm): 88.9 59.3 29.4 -0.4 -30.1 -59.9 -89.7
Max Error: 0.134mm on Picket: 5, Leaf: 60

First of all, what is the absolute median error? Why is it so small (less than 0.1mm) ? With the EPID image resolution being 0.25mm/pixel, how is it possible to achieve that high sub-pixel precision?

In the second snippet, the code performs the individual leaf detection/analysis on the same EPID image

from pylinac.picketfence import PicketFence, MLC
pf_img = r"data/23321312323213.PicketFence.dcm"
pf = PicketFence(pf_img, mlc = MLC.AGILITY)
pf.analyze(tolerance=1.5, action_tolerance=1.0, separate_leaves=True, nominal_gap_mm=3.0)
pf.plot_analyzed_image()
print(pf.results())

Picket Fence Results:
Gantry Angle (°): 0.0
Collimator Angle (°): 0.0
Tolerance (mm): 1.5
Leaves passing (%): 100.0
Absolute median error (mm): 0.628mm
Mean picket spacing (mm): 29.8mmn
Picket offsets from CAX (mm): 88.9 59.3 29.4 -0.4 -30.1 -59.9 -89.7
Max Error: 1.310mm on Picket: 3, Leaf: A29

Now the median error is 0.628mm, which seems to be a reasonable estimate, but it is about 20 times larger than the median error from Snippet 1. Why is the difference so drastic? “Max Error” came out quite different as well, and it is pointing at a different MLC leaf in a different picket now.

[jrkerns]

"First of all, what is the absolute median error?"

It's the median value of all the absolute distances between each MLC kiss and the idealized picket. A perfect delivery would have 0 median error.

"Why is it so small (less than 0.1mm)?"

On Varian machines this is pretty typical. IDK about Elekta.

"With the EPID image resolution being 0.25mm/pixel, how is it possible to achieve that high sub-pixel precision?"

Interpolation.

"Now the median error is 0.628mm, which seems to be a reasonable estimate, but it is about 20 times larger than the median error from Snippet 1. Why is the difference so drastic? “Max Error” came out quite different as well, and it is pointing at a different MLC leaf in a different picket now."

These are doing completely separate analyses. See this section of the docs. As it says there "The gap value is the combined values of the planned gap, MLC DLG, and EPID scatter effects. This is required since the expected position is no longer at the center of the MLC kiss, but offset to the side and depends on the above effects. You will likely have to determine this for yourself given the different MLCs and EPID combinations make a dynamic computation difficult." If you look at the image closely, there is likely a systematic offset of the expected positions vs the measured positions, either consistently too far in or out. This indicates the nominal gap parameter needs tweaking.

I can't tell you the ideal nominal_gap_mm. I don't like the separate leaf analysis approach for that reason. Start with whatever the planned gap is and tweak from there (usually you have to go up a little due to the rounded leaf edge and is energy-specific, but again, that's only for Varian). I would start with combined analysis (snippet 1) until you were comfortable with the expected results and only then experiment with the separate analysis.

[calculusbook]

My understanding, and please correct me if I am wrong, is that the picket fence test is to quantify the discrepancies between the estimated position of an MLC leaf and the expected position of the same MLC leaf. My emphasis here is that the term “an estimated/expected position” is defined with respect to an MLC leaf tip which is the object of interest, so to speak.

When you say “the expected position is no longer at the center of the MLC kiss but offset to the side", you apparently define the expected (reference) position with respect to something else than an MLC leaf tip. It is “the center of the MLC kiss” for the combined analysis and “an offset to the side” for the individual leaf analysis. Why not to use the expected position of the very leaf tip as the reference position? Also what confuses me is that you suggest that I have to tweak “the offset” to get good looking results where the MLC errors would be, for example, centered around zero. Perhaps, there are some other criteria for “tweaking” nominal_gap_mm ?

There is another issue I'd like to discuss. Since these two different types of analysis have two different definitions of the expected (reference) position, the combined and individual tests produce different sets of MLC positional errors.

I would expect them (the test results) to be strongly correlated when performed on the same image. I got correlation coefficients of 0.57 for the combined analysis MLC errors versus the individual Side A errors, and 0.43 for the combined MLC errors versus the individual Side B errors. I don't see much correlation between the results.

[jrkerns]

"When you say “the expected position is no longer at the center of the MLC kiss but offset to the side", you apparently define the expected (reference) position with respect to something else than an MLC leaf tip."

It's the position of the center of the MLC kiss +/- half the nominal gap. So if the center is at -50mm and a 10mm gap, the expected positions will be at -45 and -55mm.

"Why not to use the expected position of the very leaf tip as the reference position?"

Because I don't have that information. People do 10 pickets, 5 pickets, 15 pickets. They can be 1mm wide, 50mm wide. They can be offset; you never know. It would certainly be possible to have an analysis where the expected offsets are passed in and the MLC positions are compared to an absolute line in space. However, in my experience, there's often small twists in the EPID or head that cause the angle to not be exactly 0. Finally, I come from Varian, and as I said multiple times, the rounded leaf tips of Varian means that the leaf tip position is not the same as the effective radiation field edge since there is some penetration through the very edge of the leaf. This is referred to as the dosimetric leaf gap.

The implicit assumption with picket fence is that most leaves are accurate, or at least the error is randomly distributed about the ideal position. If they are, then the fit through the centers of all the leaf kisses is a valid approach. If the leaves have a consistent offset (e.g. they are all 3mm to the right) this could be detected via the offset from CAX, which is reported for each picket. If there is indeed a skew to the MLCs, this is also reported as the angle of the fitted picket vs a vertical/horizontal line. This itself can be a valuable check.

So, it's not that I don't want to use the leaf tip but that it's far more generalizable to use the center of the MLC kiss. I'll have to consider what an absolute offset looks like, as there is angle and orientation to worry about.

"Perhaps, there are some other criteria for “tweaking” nominal_gap_mm ?"

That's exactly what I tried to cover here. I don't have better answers for you.

"I don't see much correlation between the results."

I wouldn't expect there to be. As I describe in the difference between the analyses, it's very possible to have very different error values. Specifically, see the "both leaves offset" error. In such a case, the combined analysis reports 0 while the individual error is very large.

[mchamberland]

For what it's worth, I found it useful to create a picket fence with known errors to test and validate the picket fence leaf pair analysis. I am satisfied with its results and its ability to pick up errors above our tolerance value. 

[calculusbook]

Now I have a purely technical question about the following code:

picketfence.py Line 1242
mag_factor = self._image.sid / 1000
picket_pos += (
sign * self._nominal_gap_mm * mag_factor / 2 * self._image.dpmm
)

What does mag_factor do here?

It seems unnecessary since image.dpmm and nominal_gap_mm are both defined at the ISO.

What's interesting is that images acquired on the Truebeam in my institution have SID=SAD=1000mm which makes mag_factor to be 1, thus nullifying an effect of this seemingly unnecessary multiplier. EPID images coming from the Elekta VersaHD have SID=1600mm and mag_factor=1.6 creates a bit of a mess in the results of the individual leaf analysis. To be precise, the mag_factor in this computation magnifies the amplitude of the errors which might be not a big deal since this test is to identify outliers and the effect of mag_factor could be mitigated by tweaking nominal_gap_mm.

[jrkerns]

Yeah this might be a double correction. I'll take a look. Thanks

[calculusbook]

Here are a couple of findings I came across when testing the code on real-world EPID images. What I found may seem obvious to the author, but I could not find any mention of it in the documentation which is, by the way, comprehensive, detailed and overall excellent. I appreciate it.
Thought my findings could be of interest to someone who uses/wants to use this code for PicketFence testing.

1. The optimal value for the nominal_gap_mm parameter ends up being the estimated average of the picket's half-width. Only when nominal_gap_mm equals the average half-width of the pickets, the individual analysis errors become zero centered. My reasoning here is that individual MLC positional errors have to be zero-centered because otherwise you cannot meaningfully apply a threshold to detect outliers. Zero-centricity of the errors is the only criterion for the tweaking nominal_gap since there is “an implicit assumption that most leaves are accurate or the error is randomly distributed about the ideal position”.

The combined analysis error for an MLC leaf relates to the individual MLC leaf error as follows:

Combined_Error(leaf_pair_k) = Individual_Error(SideB_leaf_k) + 0.5 * ( Side_A_Pos(leaf_k) - Side_B_Pos(leaf_k) + nominal_gap)
Combined_Error(leaf_pair_k) = Individual_Error(SideA_leaf_k) + 0.5 * ( Side_B_Pos(leaf_k) - Side_A_Pos(leaf_k) - nominal_gap)

The following code illustrates the first equation.

import numpy as np
import matplotlib.pyplot as plt
from pylinac.picketfence import PicketFence, MLC

pf_img = "your_picket_fence_image.dcm"
mlc_head = MLC.MILLENNIUM
mag_factor = 1.0

pf = PicketFence(pf_img, mlc = mlc_head) # the combined analysis
pf2 = PicketFence(pf_img, mlc = mlc_head) # the individual leaf analysis
optimal_gap_mm = 1234.5 # An arbitrary number. Does not matter in this example
pf.analyze(tolerance=1.5, action_tolerance=1.0)
pf2.analyze(tolerance=1.5, action_tolerance=1.0, separate_leaves=True, nominal_gap_mm=optimal_gap_mm)

picket_idx = 0
p = pf.pickets[picket_idx]
p2 = pf2.pickets[picket_idx]

comb_errors = np.asarray([l.error[0] for l in p.mlc_meas])
individ_errors_a = np.asarray([l.error[0] for l in p2.mlc_meas])
individ_errors_b = np.asarray([l.error[1] for l in p2.mlc_meas])

half_gap = [(mlc.position[0] - mlc.position[1])/(2*p2.image.dpmm) + mag_factor*optimal_gap_mm/2 for mlc in p2.mlc_meas]

fig,ax = plt.subplots(2,1, figsize=(12,6))
ax[0].plot((comb_errors - individ_errors_b), label='Diff. between Combined Analysis Error and Individual Error (Side B), mm')
ax[0].plot(half_gap, 'o', label='Estimated Picket Width + half nominal_gap, mm')
ax[1].plot(half_gap - (comb_errors - individ_errors_b), label='difference')
_= [a.legend() for a in ax]

My understanding is that nominal_gap_mm is not needed if the individual leaf analysis makes use of two separate linear models that would fit the MLC Bank A and B separately for a given picket. The current code makes a linear fit on the mid points between the leafs (sort of introducing an aggregate metric) and then, to perform the individual analysis the code goes sort of backwards shifting the linear fit by nominal_gap_mm which seems counterintuitive to me. Instead, I would make two linear models for Bank A and Bank B separately and that would produce individual leaf errors and only then, I will get to the combined analysis averaging the two linear models into one, which effectively corresponds to a linear fit on the picket mid points.

2. I found the individual analysis was able to detect something that the combined analysis could have missed which, of course, just confirms what the author had to say in the docs:
   “Clearly, the separate analysis is advantageous here in terms of detecting the error. The chance of MLC leaves being off by the same amount in opposite directions seems extraordinarily rare.”

I just don't agree with the sentiment about “the chance being extraordinary rare” in the latter sentence. The very first PicketFence image from one of our linacs showed the following result:

The individual errors for the rightmost MLC leafs seem much more pronounced in comparison to their combined counterparts. Also for some reason, the positional errors show some degree of nonlinearity. It could be due to some effect of the imager calibration or the MLC banks themselves were not linearly aligned. In any case, the currently employed linear model does not take into account the non-linearity, thus introducing a bias into the estimated errors. I was not buying the PicketFence results reported with a 0.01mm accuracy, and now I am convinced that tight accuracy is false. I say this just to point out that the current PicketFence code does not quantify any uncertainty associated with the results.

My other case is rather simple. Two pickets on the EPID image unexpectedly came out a little wider than the other ones. I am not sure why. It could be the Elekta imager software that unevenly combined the separate picket images.

The combined analysis was not able to catch the two deviated pickets, apparently because it looked for the mid-point of each MLC kiss.

This is likely going to be my last post in this discussion. I got what I wanted to know about the internals of this test. I greatly appreciate what James has been doing for this excellent project. Thank you much!