LaTeX build log still contains Overfull \hbox reports although verbatimforcewraps is active and has actually solved the PDF output problem

[jfbu]

Describe the bug

$ grep -n 'Overfull \\hbox' _build/latex/foo.log
893:Overfull \hbox (60.49344pt too wide) in paragraph at lines 99--99

but the output is absolutely fine:

because verbatimforcewraps solved the problem of digits overflowing into margin.

How to Reproduce

index.rst:

FOO documentation
=================

Add your content using ``reStructuredText`` syntax. See the
`reStructuredText <https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html>`_
documentation for details.

SymPy can evaluate floating point expressions to arbitrary precision.  By
default, 15 digits of precision are used, but you can pass any number as the
argument to ``evalf``.  Let's compute the first 100 digits of `\pi`.

    >>> pi.evalf(100)
    3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068

conf.py contains:

latex_elements = {
    'sphinxsetup': 'verbatimforcewraps',
}

then make latexpdf and check the TeX log.

Environment Information

Python version:        3.12.3 (v3.12.3:f6650f9ad7, Apr  9 2024, 08:18:48) [Clang 13.0.0 (clang-1300.0.29.30)])
Python implementation: CPython
Sphinx version:        8.1.0+/7875ec862
Docutils version:      0.20.1
Jinja2 version:        3.1.4
Pygments version:      2.18.0

Sphinx extensions

No response

Additional context

With verbatimforcewraps Sphinx uses a "measuring phase" when rendering code-blocks to detect overflows. The Overfull \hbox originates in this measuring phase.

Having it in log file gives false positives which complicates search for other overflowing not originating in code-blocks. On the other hand it gives a way to locate where the verbatimforcewraps kicked in...

relates sympy/sympy#26867

[picnixz]

Just wondering but can't we use lstlisting environments instead of verbatim? (unless we already use them). The environment supports automatic line wrapping so it shouldn't be an issue. But I'm not sure ot supports stylings (how do we actually render those boxes? or even create them?)

[jfbu]

Just wondering but can't we use lstlisting environments instead of verbatim? (unless we already use them). The environment supports automatic line wrapping so it shouldn't be an issue. But I'm not sure ot supports stylings (how do we actually render those boxes? or even create them?)

I will try to answer concisely (which is tough for me!). We use a combination of framed with fancyvrb. Plus some core TeX to solve certain problems. This being said, here is now my original concise (ha! ha!) reply:

We do not use the LaTeX "verbatim" but an extension provided by fancyvrb package, but with some deep core TeX changes to automatically wrap long codelines in a way which allows pagebreaks in-between the created lines and avoid color leaks (for this last aspect this means that the macros for framing and adding background color can not use the core, or xcolor, \colorbox/\fcolorbox from upstream LaTeX kernel but need our own authored ones, but I promised to be concise and this is already drifting away from main topic; only mentioning that tcolorbox can solve such color leaks in similar situations only by an approach which excludes xelatex). Then there is a longstanding core question: why use a "verbatim-like" environment at all? Indeed, "verbatim" basically means to be able to handle characters which are special to LaTeX, but in our case they come back to us already escaped and transformed into Pygments mark-up. Indeed LaTeX uses the exact same calls to Pygments as HTML does. So the fancyvrb Verbatim and not the LaTeX verbatim is used because fancyvrb has an option to keep \ , {, and } act as usual in LaTeX; which is of course fundamental if we want Pygmentize mark-up (which is via LaTeX commands) have any effect. Certainly fancyvrb was used intially by Sphinx in order to have tools for example to add line-numbers, to make sure lines all have the same height, and to have line-breaks matching those of source code. (which meant that long codelines for many years overflowed in page margin, even beyond page limits). When I first started contributing to this project I took care of aspects such as adding caption in a non-separable way, either before or after code-blocks, add continuation hints at page breaks, which are related to framed package and of course handle the problem of long code-lines, which required hacking into some fancyvrb innards. I also added highlighting etc..., and finally also added the verbatimforcewraps which is object of this issue regarding TeX's report of Overfull \hbox. I raised this issue only to better explain what the PR which I will soon push solves, because it will necessarily be (probably a single line of) core TeX (not even LaTeX, deeper than LaTeX). But I am glad I did because your question prompts me to stress that actually we need almost nothing of fancyvrb. The advantage of fancyvrb is that this is a very old LaTeX package which has a current maintainer so that it is in no danger of being moved by TeXLive or MikTeX into "obsolete" section, and at same time is very stable so that we do not have to worry too much that something upstream might break us (although as I hinted to we do some rather deep hacks). And, by default Pygments returns LaTeX embedded into Verbatim from fancyrb (we started using a separately named wrapper sphinxVerbatim and not directly Verbatim at 1.5 of December 2016), so the origins of using fancyvrb lie deep even though it depends on the fact that when Pygments started providing LaTeX it of course looked for some LaTeX support package and fancyvrb fit the bill but it could have been possible to write a specific small LaTeX package. Am I concise? No of course and I am not even sure I said what I wanted to say, which is, that we don't actually really need a true "verbatim" environment, but a much more minimal one which will obey codelines+wrapping them indivdually if needed.

As per lstlisting it was never (and will probably never be) considered for the simple reason that all our highlighting is provided by Pygments, and I see very strong reasons to have exact same highlighting tools for both HTML and LaTeX. The minted package is a LaTeX Pygments-wrapper, but it requires shell-escape which is absolutely not something to allow for security reasons. Besides some problems I briefly alluded too are not really solved by minted (or were partially solved only in relatively recent iteration).

So mid-term or long-term we could drop entirely having our sphinxVerbatim be an extension of fancyvrb's Verbatim, but this is a time-costly, requires maintaining a compatibility layer with projects having used 'fvset' key, or extensions such as nbsphinx which is the one I know with some specific sphinxVerbatim related needs, although if I remember correctly probably more on the framed aspects than the fancyvrb ones.