Add instructions example for FC 2D #921

nucleosynthesis · 2024-03-20T19:44:19Z

As in the title - added a nice example for running and plotting a FC calculation (here is the 2D Hgg toy card) in the docs using combineTool

Also removed output for 2D NLL scan (not sure its very helpful).

kcormi · 2024-03-21T14:04:18Z

docs/part3/commonstatsmethods.md

+
+which will produce a file `FeldmanCousins.root` that contains a `TGraph2D` which stores the calculated value of $p_{x}$ for each point in the grid. The script below can be used to draw these values and extract the contour corresponding to 68% CL ($\alpha=0.32$).
+
+/// details | **Show script**


I wonder if its worth adding this as directly as an executable script to the repo? Or is there a reason to prefer having them copy the code and run it that way themselves?

Personally I think that for relatively simple scripts like this (i.e not official things like the combineTool scripts), its better not to have an actual script to maintain. This way, the code is instructive rather than something to just use and not think about (at least I think)

Fair enough, I was considering that point of view too and saw both sides of it a little bit. We can leave this one as is for now.

kcormi · 2024-03-21T14:04:21Z

docs/part3/commonstatsmethods.md

@@ -1015,11 +1007,15 @@ Imposing physical boundaries (such as requiring $\mu>0$ for a signal strength) i

 The boundary is imposed by **restricting the parameter range(s)** to those set by the user, in the fits. Note that this is a trick! The actual fitted value, as one of an ensemble of outcomes, can fall outside of the allowed region, while the boundary should be imposed on the physical parameter. The effect of restricting the parameter value in the fit is such that the test statistic is modified as follows ;

-$q(x) = - 2 \ln \mathcal{L}(x,\hat{\hat{\theta}}(x))/\mathcal{L}(\hat{x},\hat{\nu})$, if $\hat{x}$ in contained in the bounded range
+$$q(x) = - 2 \ln \mathcal{L}(x,\hat{\hat{\theta}}(x))/\mathcal{L}(\hat{x},\hat{\nu}),$$


I didn't notice this before, but I think $\theta$ should be $\nu$ here, right? And While we're at it, perhaps we should also change this notation to be more consistent with the paper? i.e. using $\mu$ where currently $x$ is being used? (Here and on a few other lines in this section)

You're right, should be \nu instead of \theta and we can use \vec{\mu} as elsewhere instead of "x" - will fix this

kcormi · 2024-03-21T14:05:42Z

docs/part3/commonstatsmethods.md


-There is a tool for extracting *2D contours* from the output of `HybridNew` located in `test/makeFCcontour.py`. This can be used provided the option `--saveHybridResult` was included when running `HybridNew`. It can be run with the usual <span style="font-variant:small-caps;">Combine</span> output files (or several of them) as input,
+For *two-dimensional* models, or if the parameter does not behave like a cross section, you will need to extract the contours from the output of `HybridNew` and plot them yourself. We will use the `toy-hgg-125.txt` datacard in the example below to demonstrate how this can be done.


I think it would be good to explicitly include the path to toy-hgg-125.txt datacard and to also provide the explicit text2workspace command used here (since they have to also include the multiparameter options), just to save people the time of having to figure out these details when they are try to check this example.

right, the instructions are already further up the page as its the same as for the multidimfit 2D example, but can repeat the instructions here.

True, I think either repeating them, or linking to those instructions would be fine. Though maybe repeating is a bit easier in the end.

Agree - done (repeated instructions)

kcormi · 2024-03-21T14:45:23Z

docs/part3/commonstatsmethods.md

-To extract 2D contours, the names of each parameter must be given `--xvar poi_x --yvar poi_y`. The output will be a ROOT file containing a 2D histogram of value of $p_{x,y}$ for each point $(x,y)$ which can be used to draw 2D contours. There will also be a histogram containing the number of toys found for each point.
+Once this is done, we extract the contours using
+```sh
+combineTool.py -M HybridNewGrid  ./grid.json -d toy-hgg-125.root --task-name fc2d --job-mode 'interactive' --cycles 0 --output


what about adding, here or above, a brief description of what --cycles 1 and --cycles 0 are doing? It was a little cryptic to me at first.

Me too :) @ajgilbert might be able to suggest something, but once the combineTool moves into combine I hope it comes with more documentation (and cycles would be included there I assume)

kcormi · 2024-03-21T14:47:13Z

docs/part3/commonstatsmethods.md

 ```

-To extract 2D contours, the names of each parameter must be given `--xvar poi_x --yvar poi_y`. The output will be a ROOT file containing a 2D histogram of value of $p_{x,y}$ for each point $(x,y)$ which can be used to draw 2D contours. There will also be a histogram containing the number of toys found for each point.
+Once this is done, we extract the contours using
+```sh


I'm getting an error when I try to run this about 'plot_settings' not being a key in config and then, if I fix that manually, another error about indexing opts. Both are coming from PlotTestStat in

/work/kcormier/CAT/combine_tutorial/run_preexercices/CMSSW_11_3_4/python/CombineHarvester/CombineTools/combine/LimitGrids.py

Is it me, did I make a mistake in the setup?

I ran this again and seems to work fine (we don't need the line about making plots in the JSON so i removed it but I don't think it will make a difference)

Ok, found it - needed to specifically include the line "make_plots": false, in the json. Just checked out a clean area and ran the commands to verify everything works now.

nucleosynthesis · 2024-03-22T13:34:13Z

Possibly want to tie this to #902 and #908 since its related to using combineTool in stats calculations

nucleosynthesis · 2024-04-06T11:12:23Z

@kcormi - are we ok to merge this now? There's no change to the combine code. These instructions work with v9.2.0 but are improved wrt the original script as they make use of combineTool

kcormi · 2024-04-08T07:49:31Z

Yes, thanks Nick! Sorry for being a little slow on the merge, this is a nice improvement.

nckw added 2 commits March 20, 2024 19:42

Add instructions example for FC 2D

bb65332

nicer figure for LH Scan

5e411cf

kcormi reviewed Mar 21, 2024

View reviewed changes

nckw added 3 commits March 21, 2024 15:26

fix notation for FC

54a7f13

fix notation for FC

28848e2

small fix in description

49306a0

nucleosynthesis added the documentation Updates for the documentation label Mar 22, 2024

nucleosynthesis added 2 commits March 26, 2024 17:12

few more details for 2D FC

456b0f9

need to specify make_plots false!

887d6e6

kcormi merged commit 896e024 into main Apr 8, 2024
6 checks passed

nucleosynthesis deleted the nckw_add_example_in_FC_docs branch April 8, 2024 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add instructions example for FC 2D #921

Add instructions example for FC 2D #921

nucleosynthesis commented Mar 20, 2024

kcormi Mar 21, 2024

nucleosynthesis Mar 21, 2024

kcormi Mar 21, 2024

kcormi Mar 21, 2024

nucleosynthesis Mar 21, 2024 •

edited

Loading

kcormi Mar 21, 2024

nucleosynthesis Mar 21, 2024

kcormi Mar 21, 2024

nucleosynthesis Mar 21, 2024

kcormi Mar 21, 2024

nucleosynthesis Mar 21, 2024

kcormi Mar 21, 2024

nucleosynthesis Mar 21, 2024

nucleosynthesis Mar 26, 2024

nucleosynthesis commented Mar 22, 2024

nucleosynthesis commented Apr 6, 2024

kcormi commented Apr 8, 2024


		which will produce a file `FeldmanCousins.root` that contains a `TGraph2D` which stores the calculated value of $p_{x}$ for each point in the grid. The script below can be used to draw these values and extract the contour corresponding to 68% CL ($\alpha=0.32$).

		/// details \| Show script


		There is a tool for extracting 2D contours from the output of `HybridNew` located in `test/makeFCcontour.py`. This can be used provided the option `--saveHybridResult` was included when running `HybridNew`. It can be run with the usual <span style="font-variant:small-caps;">Combine</span> output files (or several of them) as input,
		For two-dimensional models, or if the parameter does not behave like a cross section, you will need to extract the contours from the output of `HybridNew` and plot them yourself. We will use the `toy-hgg-125.txt` datacard in the example below to demonstrate how this can be done.

Add instructions example for FC 2D #921

Add instructions example for FC 2D #921

Conversation

nucleosynthesis commented Mar 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nucleosynthesis Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nucleosynthesis commented Mar 22, 2024

nucleosynthesis commented Apr 6, 2024

kcormi commented Apr 8, 2024

nucleosynthesis Mar 21, 2024 •

edited

Loading