Buffer capture bug on Ubuntu 22.04.5 LTS

[kpinc]

I'm only guessing at the source of the problem because reproducing the bug is tricky, but perhaps there's something you can do with this report anyway. Apologies also for not working this down to a minimal example. I spent too much time trying to reproduce this on a Debian system, and finding out what is really happening, and just don't have time right now to do more. I am filing this report to have a record of the issue and something to work from.

I've a HTML <span> element that contains "custom tags", %defs called from a namespace, which utilize ${capture(caller.body) | n,trim}. When the template is rendered the result of the custom tags is placed after the closing </span> element -- but only on Ubuntu 22.04.5 LTS. This is especially strange because this did not happen when the custom tags were enclosed in a <div>, adding the <span> between the <div> and the custom tag brought out the behavior.

I'm supplying code by URL below. The domain will soon change from "papio-test" to "papio", so readers may need to adjust the URLs given.

The defs are lines 41 through 55 of https://papio-test.biology.duke.edu/gitweb/?p=pgwui_common;a=blob;f=src/pgwui_common/templates/lib.mak;h=266b16a786c0bb016b0ce0fd743d3f7ece1dff5d;hb=2a8e2abe9c6af2b679fb5e8f494ad5d82e693ed5

Their usage is lines 171 through 197 of
(https://papio-test.biology.duke.edu/gitweb/?p=pgwui_sql;a=blob;f=src/pgwui_sql/templates/sql.mak;h=bbe9cb927a40b8ef9108c39fc2a1e3ff3a9d85dd;hb=3028886cf78bd3a2c1333167f18e57e82c7071bc

The rendered result (or portion thereof) is:

    <div class="stickyfooting">
      <hr>
      <span class="stickyrightbar">
        
        </span><td class="label"><label for="show_rules_id">Show rules:</label></td>

        
        <td colspan="1" class="">
          <input name="show_rules" id="show_rules_id" tabindex="1" type="checkbox" onchange="rulesDisplay(this.checked)">
        </td>
        


        
        <td class="label"><label for="show_spaces_id">Show spaces:</label></td>

        
        <td colspan="1" class="">
          <input name="show_spaces" id="show_spaces_id" tabindex="2" type="checkbox" onchange="whitespaceDisplay(this.checked)">
        </td>
        

      
    </div>

Note how the rendered span contains only whitespace, and is then followed by the tag output. What's expected, and what happens on Debian, is that the closing span tag is right before the closing div tag.

This happens on:
Ubuntu 22.04.5 LTS
Python 3.10.12
Kernel Linux bio-alberts-03.biology.duke.edu 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Packages installed in the virtualenv are:
Mako-1.3.5 PasteDeploy-3.1.0 attrs-24.2.0 beaker-1.13.0 hupper-1.12.1 markupsafe-2.1.5 pgwui_bulk_upload-0.23.0 pgwui_common-0.23.0 pgwui_copy-0.23.0 pgwui_core-0.23.0 pgwui_logout-0.23.0 pgwui_server-0.23.0 pgwui_sql-0.23.0 pgwui_upcen-0.23.0 pgwui_upload-0.23.0 pgwui_upload_core-0.23.0 plaster-1.1.2 plaster-pastedeploy-1.0.1 plaster-yaml-1.0.2 psycopg-3.2.3 pyramid-2.0.2 pyramid-beaker-0.9 pyramid-mako-1.1.0 pyyaml-6.0.2 translationstring-1.4 typing-extensions-4.12.2 venusian-3.1.0 waitress-3.0.0 webob-1.8.8 wtforms-3.1.2 zope.deprecation-5.0 zope.interface-7.0.3

The problem still occurs when whitespace is removed from between the span open and closing tags and the span content.

I cannot reproduce this on Debian 12.7, kernel Linux slate 6.1.0-25-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26) x86_64 GNU/Linux, either with python 3.11.2 or using pyenv with python 3.10.12. The same versions of the same packages are installed in debian, although the venv on debian does contain some additional packages.

On the Debian system everything works as expected. This seems to hint that the problem is outside of mako.

I could understand how some buffering problem causes output to appear later in the output stream than expected. But what's so odd here is that, regardless of whitespace removal, the output appears just after the closing tag of an element that's supposed to contain the rendered content, and does in the template source. Not to mention not being able to reproduce the problem using as identical a software stack as (or almost as) possible.

Ideas, suggested work-arounds, etc., would be appreciated. (Even though I don't anticipate being able to devote more than a few minutes at a time to resolving this in the immediate future.)