Support Jinja imports with relative paths. Issue #13889 by plastikos · Pull Request #47490 · saltstack/salt

plastikos · 2018-05-04T23:20:20Z

Jinja imports will be interpreted as originating from the top of each
of the directories in the searchpath when the template name does not
begin with './' or '../'. When a template name begins with './' or
'../' then the import will be relative to the importing file.

Prior practices required the following construct:

{% from tpldir ~ '/foo' import bar %}

This patch supports a more "natural" construct:

{% from './foo' import bar %}

Comparatively when importing from a parent directory - prior practice:

{% from tpldir ~ '/../foo' import bar %}

Construct supported by this patch:

{% from '../foo' import bar %}

What does this PR do?

Supports relative Jinja imports.

What issues does this PR fix or reference?

#13889

Previous Behavior

Unfortunately {% from './foo' import bar %} does not work. This super-secret technique was required: {% from tpldir ~ '/foo' import bar %}

New Behavior

This works: {% from './foo' import bar %}

Tests written?

Yes

Commits signed with GPG?

No

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

Jinja imports will be interpreted as originating from the top of each of the directories in the searchpath when the template name does not begin with './' or '../'. When a template name begins with './' or '../' then the import will be relative to the importing file. Prior practices required the following construct: .. code-block:: yaml {% from tpldir ~ '/foo' import bar %} This patch supports a more "natural" construct: .. code-block:: yaml {% from './foo' import bar %} Comparatively when importing from a parent directory - prior practice: .. code-block:: yaml {% from tpldir ~ '/../foo' import bar %} Construct supported by this patch: .. code-block:: yaml {% from '../foo' import bar %}

plastikos

There are a few details that should be cleaned-up before this is merged.

plastikos · 2018-05-04T23:29:28Z

salt/utils/jinja.py

+        the importing file.
+
+        '''
+        # FIXME: somewhere do seprataor replacement: '\\' => '/'


Translation of \ => / might be broken. Help me, Obi Juan, you're my only hope!

@dwoz @twangboy When using jinja imports on Windows, are Windows-specific path-separators supported? If so, are they the only ones supported, or are forward slashes also supported? I would assume both are supported, since they are in Python proper, but I don't want to assume without knowing for sure.

I use Windows separators in my Jinja. I don't even have to escape them. Jinja apparently does some escaping for you.

The tpldir = os.path.dirname(template).replace('\\', '/') construct was in the original code and appears to do some path translation on Windows. I'm not sure how to test it to ensure that I haven't broken the functionality. Is there a unit test?

plastikos · 2018-05-04T23:30:05Z

salt/utils/jinja.py

+        '''
+        Salt-specific loader to find imported jinja files.
+
+        Jinja imports will be interpreted as originating from the top


This import behavior needs to be in the documentation. Where is the best location for it?

plastikos · 2018-05-08T18:06:51Z

RETEST

plastikos · 2018-05-09T16:47:14Z

Help! Suggestions welcome!

terminalmage · 2018-05-16T02:47:03Z

tests/support/parser/__init__.py

                    self.options.name.append(fpath)
                    continue
-                self.exit(status=1, msg='\'{}\' is not a valid test module'.format(fpath))
+                self.exit(status=1, msg='\'{}\' is not a valid test module\n'.format(fpath))


Can you help me understand this change? I don't know what adding a newline here fixes.

Without the newline then the error message is not on a line by itself and the command prompt is left on the same line as the error message.

terminalmage · 2018-05-16T02:53:55Z

salt/utils/jinja.py

+                    'Relative path "%s" cannot be resolved without an environment',
+                    template
+                )
+                print('Relative path "{0}" cannot be resolved without an environment'.format(template))


We should not be printing to the CLI here.

Well that's embarrassing - that should have been removed.

terminalmage · 2018-05-16T02:56:27Z

salt/utils/jinja.py

+            if _template.split('/', 1)[0] == '..':
+                log.warning(
+                    'Discarded template path "%s": attempts to'
+                    ' ascend outside of file roots', template


Since we have multiple backends, we can't assume people will be using file_roots, so this log message should reference salt:// instead.

terminalmage · 2018-05-16T02:59:30Z

salt/utils/jinja.py

+        the importing file.
+
+        '''
+        # FIXME: somewhere do seprataor replacement: '\\' => '/'


@dwoz @twangboy When using jinja imports on Windows, are Windows-specific path-separators supported? If so, are they the only ones supported, or are forward slashes also supported? I would assume both are supported, since they are in Python proper, but I don't want to assume without knowing for sure.

Change message referencing "file roots" to "salt://"

plastikos · 2018-05-19T18:14:18Z

@terminalmage - review changes, please.

rallytime · 2018-06-06T17:55:39Z

@plastikos Is this ready to go from your side?

If so, can you also add a mention of this addition to the Fluorine release notes?

plastikos · 2018-06-08T07:51:13Z

@rallytime - I'll add the mention in the release notes and it will be ready to go.

plastikos · 2018-06-08T17:23:23Z

@rallytime , thanks for the fix-ups.

Fix saltstack#61040. Jinja imports with relative paths was introduced in saltstack#47490. Unfortunately, relative imports have never been working on Windows. That's due to backslashes being unhandled by the `cache_file` function: * https://github.com/saltstack/salt/blob/7275ecf073197faed26f3a571c25fcd59bfbfd60/salt/utils/jinja.py#L114 Instead of getting: * `salt://packages/map.jinja` We're ending up with: * `salt://packages\map.jinja` Thus, the file isn't cached from the master. There's actually a `FIXME` note about it: * https://github.com/saltstack/salt/blob/7275ecf073197faed26f3a571c25fcd59bfbfd60/salt/utils/jinja.py#L136 And the replacement is actually done for setting `tpldir`: * https://github.com/saltstack/salt/blob/7275ecf073197faed26f3a571c25fcd59bfbfd60/salt/utils/jinja.py#L165 This commit ensures the replacement is also done when caching files.

Fix #61040. Jinja imports with relative paths was introduced in #47490. Unfortunately, relative imports have never been working on Windows. That's due to backslashes being unhandled by the `cache_file` function: * https://github.com/saltstack/salt/blob/7275ecf073197faed26f3a571c25fcd59bfbfd60/salt/utils/jinja.py#L114 Instead of getting: * `salt://packages/map.jinja` We're ending up with: * `salt://packages\map.jinja` Thus, the file isn't cached from the master. There's actually a `FIXME` note about it: * https://github.com/saltstack/salt/blob/7275ecf073197faed26f3a571c25fcd59bfbfd60/salt/utils/jinja.py#L136 And the replacement is actually done for setting `tpldir`: * https://github.com/saltstack/salt/blob/7275ecf073197faed26f3a571c25fcd59bfbfd60/salt/utils/jinja.py#L165 This commit ensures the replacement is also done when caching files.

plastikos commented May 4, 2018

View reviewed changes

Merge branch 'develop' into issue-13889

9409c73

Merge branch 'develop' into issue-13889

13a364f

rallytime requested a review from terminalmage May 9, 2018 17:44

Merge branch 'develop' into issue-13889

19e53fe

terminalmage suggested changes May 16, 2018

View reviewed changes

plastikos added 4 commits May 17, 2018 23:04

Remove silly print() statement from utils/jinja.py

43ed9bc

Code review fixes.

bec2517

Change message referencing "file roots" to "salt://"

Merge branch 'develop' into issue-13889

9dd9430

Merge branch 'develop' into issue-13889

7a3bd63

rallytime requested review from terminalmage and twangboy May 21, 2018 14:21

twangboy approved these changes May 21, 2018

View reviewed changes

plastikos added 3 commits May 23, 2018 13:39

Merge branch 'develop' into issue-13889

cc22068

Merge branch 'develop' into issue-13889

e6773db

Merge branch 'develop' into issue-13889

80bb6e4

terminalmage approved these changes Jun 6, 2018

View reviewed changes

Merge branch 'develop' into issue-13889

540bcde

plastikos and others added 4 commits June 8, 2018 02:02

Add Jinja include relative paths to Fluorine release notes

1be2561

RST Syntax Fix: code-blocks need a blank line

c1c8b36

Lint: Remove extra whitespace

a7e633e

Merge branch 'develop' into issue-13889

a6a5441

rallytime merged commit 160cdb2 into saltstack:develop Jun 8, 2018

joni1993 mentioned this pull request Sep 27, 2018

GitFS mountpoint option fails on Jinja imports #49783

Closed

myii mentioned this pull request Feb 11, 2019

Formula based off template can't be moved from the root salt directory saltstack-formulas/template-formula#22

Closed

myii mentioned this pull request Jun 14, 2021

refactor: Use relative state include saltstack-formulas/packages-formula#74

Merged

19 tasks

myii mentioned this pull request Sep 30, 2021

[BUG] Salt minion is not caching all the formula's files on windows machines saltstack-formulas/packages-formula#79

Open

jpsca mentioned this pull request Nov 6, 2021

Add support from relative imports pallets/jinja#1533

Closed

myii mentioned this pull request Mar 22, 2022

fix(utils/url): handle Windows relative path backslash when caching file #61829

Merged

3 tasks

lllama mentioned this pull request Apr 30, 2023

Add custom file-system loader for templates lllama/sanickit#7

Closed

Conversation

plastikos commented May 4, 2018 • edited by terminalmage Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

What issues does this PR fix or reference?

Previous Behavior

New Behavior

Tests written?

Commits signed with GPG?

Uh oh!

plastikos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plastikos commented May 8, 2018

Uh oh!

plastikos commented May 9, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plastikos commented May 19, 2018

Uh oh!

rallytime commented Jun 6, 2018

Uh oh!

plastikos commented Jun 8, 2018

Uh oh!

plastikos commented Jun 8, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

plastikos commented May 4, 2018 •

edited by terminalmage

Loading