pgf will need adaption to the new hook management of LaTeX.

[u-fischer]

We just released a new development version of the next LaTeX version,
In this dev release we implemented a new hook system and extended the existing set of hooks.

Details can be found in the announcement:
https://www.latex-project.org/news/2020/07/22/latex-dev-2020-10-1b/

Beside other we implemented a number of shipout hooks which were previously provided by various external packages.
The new implementation clashes with pgf as it uses in pgfutil-latex.def the shipout box directly. As a consequence e.g. shadings are suddenly lost if you compile with xelatex:

%!TEX xelatex-dev
\documentclass{book}
\usepackage{tikz}

\begin{document}
\showoutput
\begin{tikzpicture}
  \shade[shading=axis,
       top color=yellow,
       bottom color=blue ] (0,0) rectangle (10,10);
\end{tikzpicture}
\end{document}

Source of the problem is the use of everyshi in

pgf/tex/generic/pgf/utilities/pgfutil-latex.def

Lines 123 to 177 in bbe2b56

	\EveryShipout{%
	% Add at begin page stuff
	\setbox\@cclv=\vbox{%
	\setbox\z@=\hbox{%
	% the boxes \pgfutil@abe ("every page") and \pgfutil@abb ("current page")
	% are used to generate pdf objects / dictionaries which are
	% required for the graphics which are somewhere in the "real"
	% page content.
	% BUT: these pdf objects MUST NOT be affected by text layout
	% shifts! Consequently, we have to undo \hoffset and \voffset
	% (which are h/v shifts to the page layout).
	%
	% Note that this of importance for shadings. To be more
	% specific: try out shadings with standalone (which uses
	% \hoffset) and with xdvipdfmx (which appears to be more
	% fragile than pdflatex) - they break unless we undo \hoffset
	% and \voffset.
	\ifdim\hoffset=0pt \else \hskip-\hoffset\fi
	\pgfutil@abe\unhbox\pgfutil@abb\pgfutil@abc\global\let\pgfutil@abc\pgfutil@empty
	\ifdim\hoffset=0pt \else \hskip+\hoffset\fi
	}%
	\wd\z@=\z@%
	\ht\z@=\z@%
	\dp\z@=\z@%
	% ... see remark above regarding \hoffset/\voffset:
	\ifdim\voffset=0pt \else \vskip-\voffset\fi
	\box\z@%
	\ifdim\voffset=0pt \else \vskip+\voffset\fi
	%
	% if TeX changes into vertical mode, it inserts \parskip and
	% \lineskip. Disable it here:
	\nointerlineskip%
	\ifvbox\@cclv \unvbox\@cclv \else\hsize=\wd\@cclv \noindent\unhbox\@cclv \fi%
	%\box\@cclv
	% using \box instead of \unhbox or \unvbox
	% has the advantage that glue settings won't be
	% altered (as for \unhbox/\unvbox). But \box breaks compatibility
	% with the 'remember picture' feature.
	}%
	}
	% Preview hack: preview.sty hacks into \shipout (which is ok), but
	% does not honour everyshi.sty (which is not ok). This causes everyshi
	% material to get lost.
	\AtBeginDocument{
	\@ifpackageloaded{preview}{%
	% Ok, package loaded. Swap definitions of everyshi.sty's shipout
	% and preview.sty's shipout:
	\ifPreview
	\let\shipout\@EveryShipout@Org@Shipout%This is the null version of \shipout, created by preview and saved by everyshi
	\let\@EveryShipout@Org@Shipout\pr@shipout% This is the original shipout
	\let\pr@shipout\@EveryShipout@Shipout%
	\fi
	}{}%
	}

It uses the hard coded number \@cclv:

\EveryShipout{%
  % Add at begin page stuff
 \setbox\@cclv=\vbox{%

Changing all the \@cclv to the new \ShipoutBox should work. As would using the new hook system instead. An ad-hoc implementation is (it will need a bit polishing for pgf, as I used expl3, and the first command name that came into my mind). Instead of testing for \AddToHook you could also test with \@ifl@t@r\fmtversion{2020/10/01}{}{}.

\ExplSyntaxOn
\newcommand\use@hook@version{%
  \AddToHook{shipout/background}{\put(\dim_to_decimal:n{\hoffset-1in},\dim_to_decimal:n{\voffset-1in})
     {\pgfutil@abe\unhbox\pgfutil@abb\pgfutil@abc\global\let\pgfutil@abc\pgfutil@empty}}}
\ExplSyntaxOff

\@ifundefined{AddToHook}{%
  \RequirePackage{everyshi}
  \EveryShipout{%
    % Add at begin page stuff
   \setbox\@cclv=\vbox{%
      \setbox\z@=\hbox{%
          % the boxes \pgfutil@abe ("every page") and \pgfutil@abb ("current page")
          % are used to generate pdf objects / dictionaries which are
          % required for the graphics which are somewhere in the "real"
          % page content.
          % BUT: these pdf objects MUST NOT be affected by text layout
          % shifts! Consequently, we have to undo \hoffset and \voffset
          % (which are h/v shifts to the page layout).
          %
          % Note that this of importance for shadings. To be more
          % specific: try out shadings with standalone (which uses
          % \hoffset) and with xdvipdfmx (which appears to be more
          % fragile than pdflatex) - they break unless we undo \hoffset
          % and \voffset.
          \ifdim\hoffset=0pt \else \hskip-\hoffset\fi
          \pgfutil@abe\unhbox\pgfutil@abb\pgfutil@abc\global\let\pgfutil@abc\pgfutil@empty
          \ifdim\hoffset=0pt \else \hskip+\hoffset\fi
      }%
      \wd\z@=\z@%
      \ht\z@=\z@%
      \dp\z@=\z@%
      % ... see remark above regarding \hoffset/\voffset:
      \ifdim\voffset=0pt \else \vskip-\voffset\fi
      \box\z@%
      \ifdim\voffset=0pt \else \vskip+\voffset\fi
      %
      % if TeX changes into vertical mode, it inserts \parskip and
      % \lineskip. Disable it here:
      \nointerlineskip%
      \ifvbox\@cclv \unvbox\@cclv \else\hsize=\wd\@cclv \noindent\unhbox\@cclv \fi%
      %\box\@cclv
      % using \box instead of \unhbox or \unvbox
      % has the advantage that glue settings won't be
      % altered (as for \unhbox/\unvbox). But \box breaks compatibility
      % with the 'remember picture' feature.
    }%
  }}
  {\use@hook@version}

For any question, comment or problem please open an issue at https://github.com/latex3/latex2e.

[ilayn]

Does this mean every project that uses everyshi or atbegshi is affected? Are these backported or only with the new version?

[u-fischer]

@ilayn The new latex format provides a reimplementation of the commands of these packages and suppresses the loading of the package themselves. If you try e.g. this with the new pdflatex-dev

\documentclass{article}
\usepackage{atbegshi}
\AtBeginShipout{\typeout{hello}}
\begin{document}
abc

\end{document}

you can see in the log that atbegshi is not loaded, but the command works fine. Packages that uses atbegshi or everyshi in a "normal way" should continue to work fine. They can if they want remove atbegshi and switch to the new hooks, but there is no need to hurry. Only packages like pgf here which do some low-level stuff need direct adaption.

[ilayn]

Thanks @u-fischer . What I am trying to figure out is, how to detect which LaTeX version user has so that proper fallback is obtained. If they are not going to be backported then there will(must) be a detection of the new hooks. Otherwise old stuff will break. Hence my question. But got my answer thanks.

[u-fischer]

As I mentioned in the issue you can test for the format with \@ifl@t@r\fmtversion{2020/10/01}{}{} , or you can simply test as I do in the code if one the new commands is defined, e.g. \AddToHook. And no, one can't backport this, this quite a large change. If someone wants to use the new hooks, they will have to update their LaTeX.

[josephwright]

I'd use \ifdefined\AddToHook or similar, dependining of course on if one requires e-TeX; testing the LaTeX version is in some senses safer, but to my knowledge there is no other \AddToHook command to fall over.

[hmenke]

The new implementation clashes with pgf as it uses in pgfutil-latex.def the shipout box directly. As a consequence e.g. shadings are suddenly lost if you compile with xelatex:

I'm happy to adapt PGF to your requirement but this sound like a massive regression to me. Are you sure that you want to break everyone's usage of the shipout box?

[ilayn]

Indeed I would expect some sort of a graceful deprecation cycle since most users will be baffled by this when they have a new vanilla TikZ but an old format. I don't know who is maintaining the ??shi?? packages but it would be the best option if new PGF has an obligatory new dependency. Otherwise maybe we need to write an in-house version pgfshipout or something and put it on CTAN. There are probably too many shipout hacks out in the wild.

If it was to me I am always in favor of crashing and burning without any backwards compatibility but being a maintainer requires some straightjacket etiquette in this regard.

[FrankMittelbach]

The new implementation clashes with pgf as it uses in pgfutil-latex.def the shipout box directly. As a consequence e.g. shadings are suddenly lost if you compile with xelatex:

I'm happy to adapt PGF to your requirement but this sound like a massive regression to me. Are you sure that you want to break everyone's usage of the shipout box?

@hmenke We don't want to break everyone's usage of shipout and the code carefully works around various cases of using \shipout that could in fact be taken as non-supported usage from a LaTeX perspective. But if there is any progress then some code/hacks that have been necessary in the past have need adjustments. The main idea here is to get to a stage where covering core mechanisms isn't any longer necessary but instead offered hooks suffice in a controllable manner.

[FrankMittelbach]

Indeed I would expect some sort of a graceful deprecation cycle since most users will be baffled by this when they have a new vanilla TikZ but an old format.

A new TikZ should work with an old format for sure but that should be fairly simple to achieve I would hope.

If it is just the everyshi usage then wouldn't it be possible to use atbegshi (which is 100% supported and is basically an improved version of evershi (which is 99% as long as you you don't try to use it and alter 255 also directly)

The alternative is to detect if you are in a new format and use new code otherwise keep the old

Otherwise maybe we need to write an in-house version pgfshipout or something and put it on CTAN. There are probably too many shipout hacks out in the wild.

the goal should really be to stop using private package shipout changes and instead use the machinery provided: only then you can rely on different packages working together. I might be mistaken but my guess is that if you load atbegshi on top right now you are also also end up being in trouble and that means at the moment there are unnecessary incompatibilities ( as some packages load that)

If it was to me I am always in favor of crashing and burning without any backwards compatibility but being a maintainer requires some straightjacket etiquette in this regard.

yes this is unfortunately true and what we are trying here is to balance this. On one hand we do in fact ask a handful of package owners to adjust but with the hope that this way all the current struggles and efficient advices (load this package after that but this before ...) stop being necessary in the not too distantant future.

[ilayn]

I fully agree. Moreover, I would expect either one of the following consequences as version clashes:

• The package properly dies and rejects moving further due to a missing dependency (or requires the format and force installs)
• The package keeps legacy behavior completely hidden from the user (even ships out the legacy package) and figures things by itself. It seems like the job falls on PGF here depending on what it discovers as the installed format version.

The limbo state is the worst one if the user needs to figure out which version works with what. That's practically the DLL hell of windows in the 90s and 00s. My vote goes for the former since then the users are forced to evolve their system to the latest & greatest.

[u-fischer]

@ilayn I'm not quite sure what you worry about. The code I suggested should allow tikz to work with older latex (as then the current code is used) and with newer latex (where it uses a new hook). Imho it won't lead to compatibility problems. And other package should be able to do the same.

[ilayn]

What I am worried about is the downstream packages introducing further hacks appending into shipout routines. TikZ itself might be safe with this but anything that relies on TikZ, I didn't check but say tcolorbox, circuitikz or whatever, will cryptic errors and strange issues. So we can't break downstream like that unless everybody moves together or at least gets to know about this. Hence my vote for the loud death.

[FrankMittelbach]

* The package properly dies and rejects moving further due to a missing dependency (or requires the format and force installs)

this is now available on the hook level so you can state that two packages writing to the same hook are in conflict and can't be used together. That can be done in advance ie if package A knows it really doesn't work with B it can state this without having to worry whether or not B is loaded before or after or not at all

it can als ask for hook code to be ordered (again without having to worry in which order the packages get loaded). But most of the time the issues are simply the ordering of code inside hooks not that two packages cn't be used to together and here again the hook machinery will help to sort this out statically

* The package keeps legacy behavior completely hidden from the user (even ships out the legacy package) and figures things by itself. It seems like the job falls on PGF here depending on what it discovers as the installed format version.

if you mean by that that something like PGF detects in what environment it runs then yes to some extend the bigger packages have to (at least when it comes down to knowing what functionality is available).

[FrankMittelbach]

What I am worried about is the downstream packages introducing further hacks appending into shipout routines. TikZ itself might be safe with this but anything that relies on TikZ, I didn't check but say tcolorbox, circuitikz or whatever, will cryptic errors and strange issues. So we can't break downstream like that unless everybody moves together or at least gets to know about this. Hence my vote for the loud death.

well first of all hacks into shipout are really really rare. It is not as we haven't studied all of CTAN to see what we are likely to break. Anyway, with what was suggested was to get rid of most of the hacks that you had to do and (at least for newer LaTeX rely on standard interface which makes the new code much smaller and cleaner). For the downstream packages that should normally be transparent --- if not they would need to move with it, yes but this is why we have 2-3 month to ensure that we caught the (few) cases.

[u-fischer]

Hence my vote for the loud death.

sorry I don't know what that means in this case. You can't break tikz for the current latex, so you have to keep the old code for now, and if you want to test against the next version you will need the new code. If this lead to incompabilities with other packages we will hopefully be able to detect this in the next months. But I'm now using the new code and the pgf patch locally for quite some time, and imho I would have seen obvious problems - I run lots of different documents everyday.

[ilayn]

sorry I don't know what that means in this case.

I'm sorry too because I don't know which part is not clear. Let me try again

Instead of introducing switches for every major change and leading to basically an if this version then do this spaghetti, I am suggesting that the new version of TikZ requires the new format and doesn't work with old stuff. Because the new format doesn't let me choose legacy behavior and offers no compatibility layer and passes this responsibility downstream. Because if PGF doesn't do this someone else needs to do that down the stream.

Is it clear enough? This is standard practice in every major language package management system.

[FrankMittelbach]

I'm not a fan of spagetti either (well I am but only in pasta form) and this is certainly a reasonable approach. A little less drastic with less issues for user in the transition period is to freeze the current version in some file foo-<date>.sty and have a single switch at the top that detects and older format for which there are interfaces and simply loaded the old format then. In that case the spagetti is confined a very small place and easy to understand (and no maintenance).

A variant approach of that is the rollback functonality for packages, but in this case I would probably go for the above.

[hmenke]

Wow, this thread got rapidly derailed into some massive bike shedding. While the discussion is interesting, could you please continue this on the LATEX-L mailing list?

[ilayn]

@hmenke What?

[hmenke]

@u-fischer @josephwright I want to keep the patch minimal. Is this sufficient?

--- a/tex/generic/pgf/utilities/pgfutil-latex.def
+++ b/tex/generic/pgf/utilities/pgfutil-latex.def
@@ -119,10 +119,17 @@
 }
 \let\pgfutil@abe\pgfutil@empty%
 
-\RequirePackage{everyshi}
-\EveryShipout{%
+\@ifl@t@r\fmtversion{2020/10/01}{%
+    \def\pgfutil@everyshipout{\AddToHook{shipout/before}}%
+    \def\pgfutil@shipoutbox{\ShipoutBox}%
+}{%
+    \RequirePackage{everyshi}%
+    \def\pgfutil@everyshipout{\EveryShipout}%
+    \def\pgfutil@shipoutbox{\@cclv}%
+}
+\pgfutil@everyshipout{%
   % Add at begin page stuff
- \setbox\@cclv=\vbox{%
+ \setbox\pgfutil@shipoutbox=\vbox{%
     \setbox\z@=\hbox{%
         % the boxes \pgfutil@abe ("every page") and \pgfutil@abb ("current page")
         % are used to generate pdf objects / dictionaries which are
@@ -152,8 +159,8 @@
     % if TeX changes into vertical mode, it inserts \parskip and
     % \lineskip. Disable it here:
     \nointerlineskip%
-    \ifvbox\@cclv \unvbox\@cclv \else\hsize=\wd\@cclv \noindent\unhbox\@cclv \fi%
-    %\box\@cclv
+    \ifvbox\pgfutil@shipoutbox \unvbox\pgfutil@shipoutbox \else\hsize=\wd\pgfutil@shipoutbox \noindent\unhbox\pgfutil@shipoutbox \fi%
+    %\box\pgfutil@shipoutbox
     % using \box instead of \unhbox or \unvbox
     % has the advantage that glue settings won't be
     % altered (as for \unhbox/\unvbox). But \box breaks compatibility

[u-fischer]

@hmenke I'm not really the right person to comment if a shipout box manipulation is working on not. @FrankMittelbach could say more. But in general I would avoid it if possible and rely on the hooks of the kernel. Imho it is safer if less packages do such low-level stuff as understanding the order of boxing/unboxing can get tricky.

Beside this your patch doesn't do the same as mine. For example (with a current xelatex-dev):

\documentclass{book}

\usepackage{tikz}

\begin{document}
\AddToHook{shipout/background}{\put(5cm,-5cm){%
  xxxx%
  \begin{tikzpicture}
  \shade[shading=axis,
       top color=yellow,
       bottom color=blue ]
       (0,0) rectangle (3,3);
   \end{tikzpicture}xxxx
}}

\begin{tikzpicture}
  \shade[shading=axis,
       top color=yellow,
       bottom color=blue ]
       (0,0) rectangle (1,1);
\end{tikzpicture}       
\end{document}

output with my patch

output with your patch

(using the shading on the "main" page is needed to get it at all, also with my patch).

[FrankMittelbach]

Am 04.09.20 um 06:31 schrieb Henri Menke:

@u-fischer <https://github.com/u-fischer> @josephwright <https://github.com/josephwright> I want to keep the patch minimal. Is this sufficient?

to me this looks right in the sense that it should mimic the code the way it was, i.e., going through the before hook and adding the code manually. But if I understand Ulrike's remark correctly then the old code had a problem with shading and you then preserve that while using the forground/background hooks directly instead of manually adding stuff to the \ShipoutBox.