`\MT@varwidth@setup` doubly subtracts protrusion under LuaTeX

[ponte-vecchio]

G'day,

We have identified an issue where \MT@varwidth@setup doubly counts protrusion under LuaTeX, causing varwidth, and downstream consumers such as tabularray, to measure content as narrower than its true natural width and break lines prematurely on the second pass. We believe the cause is an engine-level sign difference, where \rightmarginkern and \leftmarginkern return negative values in LuaTeX (cf. positive in pdfTeX).

In microtype, the varwidth patch applies \kern\MT@vwid@rightmargin (and the left equivalent) to compensate for protrusion.¹ Using the code made available from an email at dev-luatex shows that \rightmarginkern returns a negative value in LuaTeX.²³ The resulting consequence is that the compensating kern subtracts protrusion from a box whose width already reflects the protrusion, producing a measured width narrower than the natural text width. Downstream packages such as varwidth then set \hsize to this incorrect value, causing premature line breaks on the second pass (see e.g., TeXackers/tabularray#616).

The potential reason for this behaviour is because the two engines, pdfTeX and LuaTeX, implement protrusion through different mechanisms. In pdfTeX, margin kerns do not alter the natural width of the box they live in, rather, protrusion is applied virtually at shipout. \rightmarginkern in this context returns the absolute protrusion amount as a positive dimension, and adding a positive kern of that magnitude correctly restores the pre-protrusion width.

By contrast, in LuaTeX, margin kerns are real nodes of type 29 injected into the line.⁴ Specifically, the kern carries a negative width specifically so that when hpack walks the list, the resulting box width is reduced by the protrusion amount. When \unhbox exposes the box to varwidth, the measured natural width already has protrusion subtracted. \rightmarginkern returns the width of this very node, which is the same negative value.

Coming back to microtype - because \MT@varwidth@setup assumes the pdfTeX behaviour/semantics, the compensating \kern\MT@vwid@rightmargin adds a negative kern to a box whose width is already shrunk, thus double-counting the protrusion. On the second pass, varwidth sets \hsize to this artificially narrow measurement, and the line-breaking algorithm responds by breaking earlier than the visual width would warrant.

The assumption applied to \MT@varwidth@setup also has had consequences in downstream packages as well. Take tabularray - one of the packages we now maintain. With varwidth module enabled, tabularray uses varwidth inside \__tblr_get_cell_size_with_vbox: to measure natural width, then sets the final cell width in \__tblr_make_vcell_text:N via \dim_set:Nn \tex_hsize:D {#2}. That width is passed to a paragraph builder invoked inside \l_tmpa_box (which is a vbox) , hence protrusion is fully enabled during final typesetting. There's no missing vbox state.
What in fact occurs is that the measurement pass inside varwidth returns a width that's smaller than the text's true natural width which tabularray faithfully uses. That is, the typesetting context within tabularray was fine, only that the number it is given was wrong.

Since the sign difference is engine-level and unlikely to change upstream, we believe accommodating it within \MT@varwidth@setup seems like the best way forward. We will follow up with a PR, and of course welcome feedback on the directionality of the fix.

Regards,
@ponte-vecchio and @peaR-red

Footnotes

1. https://github.com/schlcht/microtype/blob/5d3516ab2b239d3d249f30096e70cc28867e84c4/microtype.dtx#L6388-L6404 ↩

2. More precisely, both L and R return -5.0pt ↩

3. See also: https://chat.stackexchange.com/transcript/41?m=68921483#68921483 ↩

4. https://github.com/TeX-Live/texlive-source/blob/trunk/texk/web2c/luatexdir/tex/packaging.c#L763 ↩

[peaR-red]

Here is a demonstration of the patch. Example is taken from the original issue post in tabularray.

\documentclass{article}
\usepackage{varwidth}
\usepackage{microtype}
\makeatletter
  \def\MT@varwidth@setup{%
    \ifMT@protrusion
      \MT@warning@nl{Cannot patch varwidth without etex extensions}%
      \MT@info@nl{Patching varwidth to enable character protrusion}%
      \newdimen\MT@vwid@leftmargin
      \newdimen\MT@vwid@rightmargin
      \patchcmd\@vwid@sift{\sift@deathcycles\z@}
        {\ifhbox\z@ \MT@vwid@leftmargin\leftmarginkern\z@
         \MT@vwid@rightmargin\rightmarginkern\z@\fi \sift@deathcycles\z@}{}{}%
      \patchcmd\@vwid@resetb{\kern\@vwid@loff \unhbox\z@}
        {\kern\@vwid@loff \ifdim\MT@vwid@leftmargin=\z@\else\kern-\MT@vwid@leftmargin\fi
         \unhbox\z@ \ifdim\MT@vwid@rightmargin=\z@\else\kern-\MT@vwid@rightmargin\fi}{}{}%
      \patchcmd\@vwid@measure{\kern\@vwid@loff \unhbox\z@}
        {\kern\@vwid@loff \ifdim\MT@vwid@leftmargin=\z@\else\kern-\MT@vwid@leftmargin\fi
         \unhbox\z@ \ifdim\MT@vwid@rightmargin=\z@\else\kern-\MT@vwid@rightmargin\fi}{}{}%
    \fi
  }
\makeatother
\begin{document}
\newcommand\mytext{H )}
\newlength\mylen
\settowidth\mylen{\begin{varwidth}{\textwidth}\mytext\end{varwidth}}
\setlength\parindent{0pt}

Default:
\fbox{\begin{minipage}{\mylen}\mytext\end{minipage}}
\fbox{\vbox{\hsize=\mylen\mytext}}

Raggedright:
\fbox{\begin{minipage}{\mylen}\raggedright\mytext\end{minipage}}
\fbox{\vbox{\hsize=\mylen\raggedright\mytext}}
\end{document}

Caution

For end users applying the patch, it should be pasted in the preamble, not in the main document. \MT@varwidth@setup initializes at the beginning of the document, so even if you redefine it inside the document, the damage has already been done.

It would make sense to do something like this in microtype.dtx.

%<*pdf-|lua->
%<pdf->\MT@requires@pdftex5{
  \def\MT@varwidth@setup{%
    \ifMT@protrusion
      \MT@info@nl{Patching varwidth to enable character protrusion}%
      \newdimen\MT@vwid@leftmargin
      \newdimen\MT@vwid@rightmargin
      \patchcmd\@vwid@sift{\sift@deathcycles\z@}
        {\ifhbox\z@ \MT@vwid@leftmargin\leftmarginkern\z@
         \MT@vwid@rightmargin\rightmarginkern\z@\fi \sift@deathcycles\z@}{}{}%
      \patchcmd\@vwid@resetb{\kern\@vwid@loff \unhbox\z@}
        {\kern\@vwid@loff \ifdim\MT@vwid@leftmargin=\z@\else\kern\MT@vwid@leftmargin\fi
         \unhbox\z@ \ifdim\MT@vwid@rightmargin=\z@\else\kern\MT@vwid@rightmargin\fi}{}{}%
      \patchcmd\@vwid@measure{\kern\@vwid@loff \unhbox\z@}
        {\kern\@vwid@loff \ifdim\MT@vwid@leftmargin=\z@\else\kern\MT@vwid@leftmargin\fi
         \unhbox\z@ \ifdim\MT@vwid@rightmargin=\z@\else\kern\MT@vwid@rightmargin\fi}{}{}%
    \fi
  }
%<pdf->}{
%<pdf->  \def\MT@varwidth@setup{%
%<pdf->    \ifMT@protrusion
%<pdf->      \MT@warning@nl{... upgrade pdftex ...}%
%<pdf->    \fi
%<pdf->  }
%<pdf->}
%</pdf-|lua->
%<*lua->
\MT@requires@luatex3{
  \def\MT@varwidth@setup{%
    \ifMT@protrusion
      \MT@info@nl{Patching varwidth to enable character protrusion}%
      \newdimen\MT@vwid@leftmargin
      \newdimen\MT@vwid@rightmargin
      \patchcmd\@vwid@sift{\sift@deathcycles\z@}
        {\ifhbox\z@ \MT@vwid@leftmargin\leftmarginkern\z@
         \MT@vwid@rightmargin\rightmarginkern\z@\fi \sift@deathcycles\z@}{}{}%
      \patchcmd\@vwid@resetb{\kern\@vwid@loff \unhbox\z@}
        {\kern\@vwid@loff \ifdim\MT@vwid@leftmargin=\z@\else\kern-\MT@vwid@leftmargin\fi
         \unhbox\z@ \ifdim\MT@vwid@rightmargin=\z@\else\kern-\MT@vwid@rightmargin\fi}{}{}%
      \patchcmd\@vwid@measure{\kern\@vwid@loff \unhbox\z@}
        {\kern\@vwid@loff \ifdim\MT@vwid@leftmargin=\z@\else\kern-\MT@vwid@leftmargin\fi
         \unhbox\z@ \ifdim\MT@vwid@rightmargin=\z@\else\kern-\MT@vwid@rightmargin\fi}{}{}%
    \fi
  }
}{}
%</lua->

Alas, I will leave the final PR to @ponte-vecchio to address this issue.

[schlcht]

Thanks, but I think your analysis is wrong. The problem with varwidth and, by extension, tabularray (which has already been reported as #57) does not result from a difference in how protrusion is implemented in general but rather what \rightmarginkern reports for lines that end with a fil: pdftex reports zero (which IMO is the correct result, as there is indeed no protrusion), while luatex reports the margin kern that would occur, were there not a fil).

While your patch would fix the MWE at hand, it gives wrong results for all lines that do not end with a fil. (Try a varwidth environment containing several fully justified lines, where protrusion will be reversed.)

I think this diverging behaviour of \rightmarginkern should be fixed in luatex -- Luigi has just responded to my report, so let's see what happens...