PGFMath not parsing {"two"}[0] with \usetikzlibrary{babel}

[ilyaza]

The example from the manual for {x} (p.1033 in TeXLive2021):

\def\myarray{{1,"two",2+1,"IV","cinq","sechs",sin(\i*5)*14}}
\foreach \i in {0,...,6}{\pgfmathparse{\myarray[\i]}\pgfmathresult, }

does not work with babels present in TexLive2021. It was working fine (even without \usetikzlibrary{babel}) in TeXLive2018.

Minimal working example (MWE)

\documentclass{article}
\usepackage[russian,english]{babel}
\usepackage{tikz}
\usetikzlibrary{babel}

\begin{document}

\def\myarray{{1,"two",2+1,"IV","cinq","sechs",sin(\i*5)*14}}
\foreach \i in {0,...,6}{\pgfmathparse{\myarray[\i]}\pgfmathresult, }

\end{document}

The error message is

! Package PGF Math Error: Unknown function `two' (in '{1,"two",2+1,"IV","cinq",
"sechs",sin(6*5)*14}[6]').

[muzimuzhi]

Firstly, tikz package library babel won't affect the behavior of pgfmath.

On the babel side, With babel-russian, " is made an active character whose definition is equivalent to

\def\normal@char@quote{"}
\catcode`\"=\active
\gdef"{%
  \ifx\protect\@typeset@protect % \edef case
  \else
    \ifx\protect\@unexpandable@protect % \protected@edef case
      \noexpand "
    \else % case 3
      \protect "
    \fi
    \expandafter\@gobble
  \fi
  \normal@char@quote
}

With the above definition,

• \edef\x{"} will store a normal, catcode 12 (other) " in \x, while
• \protected@edef\x{"} will store an active character, catcode 13 (active) " in \x.

On pgfmath's side, after some setup, \pgfmathparse will first expand then store its argument in \pgfmath@expression and the math parser expects a catcode 12 " to mark the beginning and ending of a string. Commit 9d18b41 changed the expansion from a simple \edef to \protected@edef, which caused the reported difference. This change is contained in v3.1.4, 2019 Jul. That's why texlive 2018 works.

[muzimuzhi]

I'm experienced in neither handling active characters nor using babel. As a temp workaround, one may restore the catcode of " only for \pgfmathparse:

\usepackage{etoolbox}
% after both babel and tikz are loaded
\apptocmd\pgfmath@catcodes{\catcode`\"=12 }{}{\fail}

[hmenke]

You just can't win. Reverting to \edef from \protected@edef would break things like

\pgfmathparse{width("\textbf{Hello}")}

[hmenke]

Resetting the quote catcode might work but then this will stop working

% \usepackage[ngerman]{babel}
\pgfmathparse{width(\string"compound"=word\string")}

although that is a very niche use-case at best. Probably not worth supporting.

[ilyaza]

% \usepackage[ngerman]{babel}
\pgfmathparse{width(\string"compound"=word\string")}

In principle, could one code \pgfmathrestorecatcode (which is “opposite” to \string above) so that

% \usepackage[ngerman]{babel}
\pgfmathparse{width(\string"compound\pgfmathrestorecatcode"=word\string")}

works? (Of course, then \strings become superfluous.)

[muzimuzhi]

For \edef, \catcode is non-expandable, hence it's impossible to change catcode(s) in an expandable manner.

When pgfmath supports lazy evaluation (#1006), a function string(<string without quotes>) would be helpful.

[muzimuzhi]

For \edef, \catcode is non-expandable, hence it's impossible to change catcode(s) in an expandable manner.

Ah, maybe

% \usepackage[ngerman]{babel}
% \usepackage{etoolbox}
\apptocmd\pgfmath@catcodes{%
  \scantokens{\makeatletter
    \def\pgfmath@char@user@quote{"}%
  \makeatother\noexpand}%
  \catcode`\"=12
}

\def\pgfmathrestorecatcode#1{%
  \if#1"\pgfmath@char@user@quote
  \else % other cases avaliable
    #1
  \fi
}

\pgfmathparse{width("compound\pgfmathrestorecatcode"=word")}

[ilyaza]

    \def\pgfmath@char@user@quote{"}%
 …
  \if#1"\pgfmath@char@user@quote

I had in mind slightly different (pseudo)code:

    \expandafter\def\csname pgfmath-prechar\string#1\endcsname{#1}%
 …
  \ifDefined{pgfmath-prechar\string#1}{\csname pgfmath-prechar\string#1\endcsname}{#1}

(I do not know whether \strings are (strictly) necessary.

Updated: the first line above should be good when used “on the outside”, but for pgfmath inside tikzpicture this may be too late. So tikzpicture should do something similar, and pgfmath should not redefine {pgfmath-prechar\string#1} if it is already defined.

[ilyaza]

Firstly, tikz package library babel won't affect the behavior of pgfmath.

This was not clear at all from the babel section in the manual. This may need to be clarified there…

[muzimuzhi]

the first line above should be good when used “on the outside”, but for pgfmath inside tikzpicture this may be too late.

agree

[ilyaza]

Should I put the suggestion for implementing \pgfmathrestorecatcode into a separate issue?

[hmenke]

I've already decided not to implement it, because it's too niche of a use-case. It also won't work in many cases, mostly for content that is already tokenized and I refuse to put \scantokens anywhere into pgfmath.

[ilyaza]

Now I’m completely lost! (After I run \tracingall with the initial example.) In it “the content … is already tokenized” (via \def) — as in your objection.

How can changing \catcodes inside \pgfmathparse fix this and make it parse?

Furthermore, I see that \pgfmathsetmacro first scans its second argument, then resets \catcodes (via calling \pgfmathparse). It seems that it should be too late, after #2 is already scanned?

[davidcarlisle]

rather than fiddle with catcodes (which would only work at the top level anyway) I'd define the active quote to expand to the non active one, something like

\documentclass{article}
\usepackage[russian,english]{babel}
\usepackage{tikz}
\usetikzlibrary{babel}

\makeatletter
{\catcode`\"\active
\gdef\pgfmath@catcodes{% Maybe unnecessary.
  \catcode`\==12 %
  \catcode`\,=12 %
  \catcode`\|=12 %
  \catcode`\&=12 %
  \edef"{\string"}%
}}
\makeatother

\begin{document}


\def\myarray{{1,"two",2+1,"IV","cinq","sechs",sin(\i*5)*14}}
\foreach \i in {0,...,6}{\pgfmathparse{\myarray[\i]}\pgfmathresult, }

\end{document}

although possibly re-arranged a bit so that it doesn't do a global definition and the \string at run time.