Allow explicit assumptions in the theory

[scarlehoff]

Just the same as it is allowed with convolute.
We need it for the theory that allows for c-cbar.

Btw, convolute doesn't work for me.

pineko convolute data/grids/200/E906deut_bin_09.pineappl.lz4 data/ekos/200/E906deut_bin_09.tar test 0 2                                                                           
┌───────────────┐
│ Computing ... │
└───────────────┘
   data/grids/200/E906deut_bin_09.pineappl.lz4
 + data/ekos/200/E906deut_bin_09.tar
 = test
 with max_as=0, max_al=2, xir=1.0, xif=1.0
Traceback (most recent call last):
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/bin/pineko", line 8, in <module>
    sys.exit(command())
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/pineko/cli/convolute.py", line 52, in subcommand
    _grid, _fk, comp = evolve.evolve_grid(
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/pineko/evolve.py", line 133, in evolve_grid
    alphas_values = [op["alphas"] for op in operators["Q2grid"].values()]
  File "/home/juacrumar/.cache/pypoetry/virtualenvs/pinefarm-HwI1B8mU-py3.10/lib/python3.10/site-packages/pineko/evolve.py", line 133, in <listcomp>
    alphas_values = [op["alphas"] for op in operators["Q2grid"].values()]
KeyError: 'alphas'

[scarlehoff]

Question @andreab1997 @alecandido

is it better to have an --assumptions flag or rather have an assumptions key in the theory?

This should be override whatever assumptions are automatically deduced here

Another option is to have only two situations: "no assumptions" or "automatically deduced", in which case the flag should be --no-assumptions or --no-optimization or whatever.

@cschwan which of the assumptions in pineappl would be the right one for ccbar asymmetry (but everything else equal) ?

[andreab1997]

Question @andreab1997 @alecandido

is it better to have an --assumptions flag or rather have an assumptions key in the theory?

This should be override whatever assumptions are automatically deduced here

Another option is to have only two situations: "no assumptions" or "automatically deduced", in which case the flag should be --no-assumptions or --no-optimization or whatever.

@cschwan which of the assumptions in pineappl would be the right one for ccbar asymmetry (but everything else equal) ?

I would say the second option to be better. By default the assumptions should be automatically deduced and then you can disable the deduction with a flag.

[cschwan]

That should be Nf4Ind; 4 flavours, all of them (including anti-particles) are independently chosen. See also the documentation which explicitly mentions NNPDF's, MSHT's and CT's choices.

[alecandido]

is it better to have an --assumptions flag or rather have an assumptions key in the theory?

I would rather always specify the assumption, and use deduction only as a fall-back.
The theory DB entry describes the theory itself, so all the tables should be consistent with that choice (and for this reason I would not allow for an explicit --assumptions flag, if you really want to, change your theory file), and it should be advertised in the global description.

Deduction is still useful for old theories, and it works as it is, compliant with @cschwan docs: even NNPDF did and does fits with Nf3Ind, when using perturbative charm.

[scarlehoff]

I would say the second option to be better. By default the assumptions should be automatically deduced and then you can disable the deduction with a flag.

Then we need to create a new theory option for ccbar asymmetry as @felixhekhorn said here

NNPDF/nnpdf#1627 (comment)

Although after rereading that message I'm starting to think that we should instead have an explicit assumptions line in the theory runcard...

[alecandido]

Although after rereading that message I'm starting to think that we should instead have an explicit assumptions line in the theory runcard...

I would say this is sufficient for all purposes.
However,

Then we need to create a new theory option for ccbar asymmetry as @felixhekhorn said here

This I don't understand, Nf4Ind is what you need, and if you're fitting above charm matching scale is already what you're getting.
The only thing for which we would need such an option would be if we were fitting below charm matching, and you also want the intrinsic ccbar asymmetry. But this is not what NNPDF is doing, charm is always fitted in the perturbative region.

[scarlehoff]

pineko/src/pineko/theory_card.py

Line 80 in f7164fa

if ic:

In this branch Nf4Sym is the only option available. We want to have also Nf4Ind here.

I would say this is sufficient for all purposes.

Yes, with this nothing extra is needed.

[alecandido]

In this branch Nf4Sym is the only option available. We want to have also Nf4Ind here.

You never enter that branch in any NNPDF fit, since it is activated by this other branch:

pineko/src/pineko/theory_card.py

Line 78 in f7164fa

if Q0 < match_scales["c"]:

that is never true when ic is true.

In NNPDF IC fits you always enter here:

pineko/src/pineko/theory_card.py

Line 76 in f7164fa

if Q0 < match_scales["b"]:

and resolve to Nf4Sym.
(even if we are not fitting ccbar asymmetry, we need it in the FkTables, since it could be perturbatively generated, in general)

[scarlehoff]

Wait, I'm confused. The only way we get Sym is entering on that branch otherwise mod == Ind. But you are telling me we never enter there. So all theory 400 datasets are alrady Nf4Ind? But then there's no problem.

So what is the problem for fitting theory 400 with a ccbar basis?

[alecandido]

So all theory 400 datasets are alrady Nf4Ind?

Yes

But then there's no problem.

So what is the problem for fitting theory 400 with a ccbar basis?

Indeed, there is no problem :)

[scarlehoff]

Then we are done.

@tgiani you can run the fit with theory 400 and the current code already for ccbar.

[alecandido]

I would keep this, even though you don't need for the current purpose

[scarlehoff]

But then the title should be changed to whatever the scope we want. In this case I think what @andreab1997 suggested is enough (just have a no-assumptions)

[alecandido]

The title I already changed before reopening, and it reflects exactly what I want: the assumptions should be allowed to be explicitly set in the theory.

Deduction might fail in some cases (i.e. assumptions is not always a function of the other values), so it should be possible to set it.
At the moment, it is always setting the most aggressive optimization available, but you might want to retain more entries in some cases (e.g. there is no option for intrinsic charm asymmetry in 3 flavors, nor intrinsic bottom at all)

[tgiani]

@scarlehoff ok great thank you, I ll run it

[cschwan]

So all theory 400 datasets are alrady Nf4Ind?

Yes

Doesn't that mean theory 400 could be smaller with Nf4Sym, so isn't that a performance 'bug'?

[alecandido]

Doesn't that mean theory 400 could be smaller with Nf4Sym, so isn't that a performance 'bug'?

It could be smaller, but you want to tell explicitly that c- is going to be zero, that is theoretically wrong, since being in the 4FNS it can be perturbatively generated by other valence partons, so you should not be able to set it to 0.

More than a "performance bug" I'd call it a "design choice": we decided not to do anything theoretically wrong (at the level of FkTables), that was not explicitly chosen and documented.
But you can consider it to be both of course: having an assumptions key, you could declare that you assume c- to be 0, and further optimize your theory (however, best case is 1 distribution in 9, so I guess you can get a 10-20% gain - not negligible, but not incredibly relevant).

[cschwan]

What I meant to ask is: with Nf4Sym V15 is a separate entry in FK tables (it's not added to V), but before this exercise it isn't fitted. But was/is it properly treated anyway? What's the difference between fitted and non-fitted PDFs in FK tables for say n3fit?

[alecandido]

But was/is it properly treated anyway? What's the difference between fitted and non-fitted PDFs in FK tables for say n3fit?

Nope: it was arbitrarily assumed that $V_{15} = V$, i.e. $c^- = 0$ at fitting scale. This is an arbitrary choice, because if you start from the 3 flavor patch, c- will be already generated by matching conditions.
In this sense, this choice is as arbitrary as CT's $s^- = 0$ choice. The only, practically humongous, difference is the order of magnitude of the two objects.

Of course, if c- is null, then you don't need a V15 entry, you have one less element in flavor basis.

[cschwan]

But how are the FK tables generated? With Nf4Ind or with Nf4Sym? Because in the first case V and V15 are independent and only in the second case V15 is added to V.

[alecandido]

I explained in details in this answer: #64 (comment)

If you ask for theory 400 in particular, it is Nf4Ind, because it branches in "below bottom matching" but not into "below charm matching".

And we do this, because if you put there you're fitting scale, it is impossible from the theory perspective to determine if you want to fit $V_{15}$ or not. You need further assumptions, as we know ;)

[felixhekhorn]

@scarlehoff I think it hasn't been already explicitly said here since the discussion went else where: the problem you experienced with convolute in the very first message is most likely due to mismatched versions: i.e. you're using an eko with an incompatible eko data version (so the actual object you're trying to load is outdated). Unfortunately, it is very hard at the moment to know this before hand (you need to check the eko version accompanying the pineko version and then "guess" the eko version of the file) - we need to improve this as well ...

[alecandido]

I think it hasn't been already explicitly said here since the discussion went else where: the problem you experienced with convolute in the very first message is most likely due to mismatched versions

@felixhekhorn this has already been extensively discussed here NNPDF/pineline#17