tests : add test-jinja -py option for cross-checking

[ngxson]

Small QoL feature for test-jinja: Running test-jinja -py will output the results using the original jinja2 python. This is meant to be used to make sure that the test results are all aligned to the reference jinja2 implementation.

The way it works is by simply spawn python3 as a subprocess. Requires to have jinja2 pre-installed via pip

Some tests are currently failing on this mode, which I'm not 100% sure while. May worth investigating.

Note: I haven't tested this on windows

[CISC]

Some tests are currently failing on this mode, which I'm not 100% sure while. May worth investigating.

Will take a look.

[ngxson]

Here are some of the failed tests for example. Looks quite strange. I've upgraded to python 3.14 and latest jinja2==3.1.6

  tojson ensure_ascii=true
    Python script failed with exit code: 1
    Output: Traceback (most recent call last):
  File "<string>", line 14, in <module>
(REDACTED)
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
TypeError: do_tojson() got an unexpected keyword argument 'ensure_ascii'

  ASSERTION FAILED : python execution
  tojson ensure_ascii=true (1 of 1 assertion(s) failed)                         [FAIL]
  tojson sort_keys=true
    Python script failed with exit code: 1
    Output: Traceback (most recent call last):
  File "<string>", line 14, in <module>
(REDACTED)
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
TypeError: do_tojson() got an unexpected keyword argument 'sort_keys'

  ASSERTION FAILED : python execution
  tojson separators=(',',':') (1 of 1 assertion(s) failed)                      [FAIL]
  tojson separators=(',',': ') indent=2
    Python script failed with exit code: 1
    Output: Traceback (most recent call last):
  File "<string>", line 14, in <module>
(REDACTED)
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
TypeError: do_tojson() got an unexpected keyword argument 'separators'

[CISC]

Here are some of the failed tests for example. Looks quite strange. I've upgraded to python 3.14 and latest jinja2==3.1.6

transformers modify jinja2 a bit, see above.

[CISC]

Ah, sorry, a bit too quick on the cut'n'paste. :)

[ngxson]

no problem, I was thinking about the same possibility, just haven't dig through the correct place in transformers :)

one thing left though, seems like ImmutableSandboxedEnvironment cause in-place modifications like arr.append() to throw jinja2.exceptions.SecurityError

[ngxson]

Alright, just one problem left: jinja2.exceptions.UndefinedError: 'list object' has no attribute 'map'

[CISC]

one thing left though, seems like ImmutableSandboxedEnvironment cause in-place modifications like arr.append() to throw jinja2.exceptions.SecurityError

And rightly so, this will happen with transformers too.

[CISC]

You can verify transformers behavior using my Chat Template Editor BTW. :)

[ngxson]

I think I got it, map is a filter so it must be used as arr|map() and not arr.map()

However, it now introduce yet another set of errors:

• jinja2.exceptions.TemplateRuntimeError: No filter named 'age'
• jinja2.exceptions.TemplateRuntimeError: No filter named 0

[CISC]

I think I got it, map is a filter so it must be used as arr|map() and not arr.map()

However, it now introduce yet another set of errors:

* `jinja2.exceptions.TemplateRuntimeError: No filter named 'age'`

* `jinja2.exceptions.TemplateRuntimeError: No filter named 0`

Yeah, without attribute map will apply the argument as a filter.

[ngxson]

should be fine now, I'm done with this. @CISC feel free to push commits directly here if you need to add something else

[CISC]

I think I got it, map is a filter so it must be used as arr|map() and not arr.map()
However, it now introduce yet another set of errors:

* `jinja2.exceptions.TemplateRuntimeError: No filter named 'age'`

* `jinja2.exceptions.TemplateRuntimeError: No filter named 0`

Yeah, without attribute map will apply the argument as a filter.

Fixed that in #18883

[CISC]

Super useful.

We should probably go back to ImmutableSandboxedEnvironment and disallow pop and append though, transformers uses this and I see no value in mutable arrays.

[aldehir]

I like this approach! I was worried about adding a dependency on Python esp. since it requires Jinja2, but the flag addresses that concern.

[ngxson]

We should probably go back to ImmutableSandboxedEnvironment and disallow pop and append though, transformers uses this and I see no value in mutable arrays.

Actually it's a bit puzzling to me why the Kimi-K2-Instruct.jinja template inside models/templates use append() and pop(). It seems like a patch, as the original model doesn't have it.

[CISC]

We should probably go back to ImmutableSandboxedEnvironment and disallow pop and append though, transformers uses this and I see no value in mutable arrays.

Actually it's a bit puzzling to me why the Kimi-K2-Instruct.jinja template inside models/templates use append() and pop(). It seems like a patch, as the original model doesn't have it.

Ah, minja supports it, guess whoever added that only tested here.