[confcom] Adding new flag for faster hashing and patching tar file usage#7414
Merged
Conversation
adding new way to read from tar files
adding faster hashing option
|
| rule | cmd_name | rule_message | suggest_message |
|---|---|---|---|
| confcom acipolicygen | cmd confcom acipolicygen added parameter faster_hashing |
|
Hi @SethHollandsworth, |
Collaborator
|
confcom |
changing what tag name to look for in genpolicy release
Contributor
|
kairu-ms
approved these changes
Apr 1, 2024
Collaborator
|
[Release] Update index.json for extension [ confcom ] : https://dev.azure.com/azclitools/internal/_build/results?buildId=144952&view=results |
micromaomao
added a commit
to micromaomao/azure-cli-extensions
that referenced
this pull request
Apr 30, 2026
The current code extracts an item in the tar based on a path controlled by the manifest within the tar, which is prone to path traversal on Linux and Windows, and potentially other path confusion attacks on Windows if the Config field in the manifest contains special names. This extraction is unnecessary as we can read the content of the config directly from the tar file using extractfile. This commit simplifies it to do that. Also, there are two versions of the map_image_from_tar function, with the second one introduced in Azure#7414. Later code changes in Azure#8238 means that these functions now do basically the same thing (with different clean up code). After this simplification, these two functions are exactly the same, so let's also remove the _compatibility one. Before: > az confcom acipolicygen --image mcr.microsoft.com/aci/skr:2.14 --tar image-docker-malformed.tar --outraw-pretty-print ... Pulling and hashing images...: 0%| | 0/2 [00:00<?, ?percent/s] ERROR: The command failed with an unexpected error. Here is the traceback: ERROR: [Errno 13] Permission denied: '../../../../../blobs' Traceback (most recent call last): File "/home/mao/.az-cli-newpy.venv/lib/python3.13/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 677, in execute raise ex File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 820, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 789, in _run_job result = cmd_copy(params) File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 335, in __call__ return self.handler(*args, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/command_operation.py", line 120, in handler return op(**command_args) File "/home/mao/.azure/cliextensions/confcom/azext_confcom/custom.py", line 197, in acipolicygen_confcom policy.populate_policy_content_for_all_images( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ individual_image=bool(image_name), tar_mapping=tar_mapping, faster_hashing=faster_hashing ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/mao/.azure/cliextensions/confcom/azext_confcom/security_policy.py", line 485, in populate_policy_content_for_all_images image_info, tar = get_image_info(progress, message_queue, tar_mapping, image) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.azure/cliextensions/confcom/azext_confcom/template_util.py", line 111, in get_image_info image_info = os_util.map_image_from_tar_backwards_compatibility( image_name, tar_file, tar_location ) File "/home/mao/.azure/cliextensions/confcom/azext_confcom/os_util.py", line 190, in map_image_from_tar_backwards_compatibility tar.extract(info_file.name, path=tar_dir) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2416, in extract self._extract_one(tarinfo, path, set_attrs, numeric_owner) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2464, in _extract_one self._handle_fatal_error(e) ~~~~~~~~~~~~~~~~~~~~~~~~^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2458, in _extract_one self._extract_member(tarinfo, os.path.join(path, tarinfo.name), ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ set_attrs=set_attrs, ^^^^^^^^^^^^^^^^^^^^ numeric_owner=numeric_owner, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ filter_function=filter_function, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ extraction_root=path) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2539, in _extract_member os.makedirs(upperdirs, exist_ok=True) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen os>", line 218, in makedirs File "<frozen os>", line 228, in makedirs PermissionError: [Errno 13] Permission denied: '../../../../../blobs' To check existing issues, please visit: https://github.com/Azure/azure-cli/issues After: > az confcom acipolicygen --image mcr.microsoft.com/aci/skr:2.14 --tar image-docker-malformed.tar --outraw-pretty-print | grep -C10 layers ... Pulling and hashing images...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.38percent/s] WARNING: mcr.microsoft.com/aci/skr:2.14 read from local tar file "strategy": "re2" }, { "pattern": "azurecontainerinstance_restarted_by=.+", "required": false, "strategy": "re2" } ], "exec_processes": [], "id": "mcr.microsoft.com/aci/skr:2.14", "layers": [ "a189b02d4858578459fda1dfbd7c6a4557c44208b9829e02b931771a6d611c39", "300f9661fb3d46c0f299ad6f552b7ad0c41ea5141755b0b3feaca3081a108f7a", "0afffca98bacf8e7b6e6f7982459a03219f60555523163c73c4b092e0a3deef2", "eefefd5009aed4ba4478876995d1a18aa3a670661fcc61d2e4cba6e2b79da0a1", "b868a7e1bebef40e5bf4d58fe271c0a10a351e68b12179ec019af9f6c75781ae", "8b4842f06982817534a75bcf71865213b09dfa8313229c384e5201dadbd75e25" ], ... (Validated that this matches the result of using the image reference directly without a tar, and also matches with the result of using the oci tar from `oras backup`) Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com>
4 tasks
micromaomao
added a commit
to micromaomao/azure-cli-extensions
that referenced
this pull request
May 5, 2026
The current code extracts an item in the tar based on a path controlled by the manifest within the tar, which is prone to path traversal on Linux and Windows, and potentially other path confusion attacks on Windows if the Config field in the manifest contains special names. This extraction is unnecessary as we can read the content of the config directly from the tar file using extractfile. This commit simplifies it to do that. Also, there are two versions of the map_image_from_tar function, with the second one introduced in Azure#7414. Later code changes in Azure#8238 means that these functions now do basically the same thing (with different clean up code). After this simplification, these two functions are exactly the same, so let's also remove the _compatibility one. Before: > az confcom acipolicygen --image mcr.microsoft.com/aci/skr:2.14 --tar image-docker-malformed.tar --outraw-pretty-print ... Pulling and hashing images...: 0%| | 0/2 [00:00<?, ?percent/s] ERROR: The command failed with an unexpected error. Here is the traceback: ERROR: [Errno 13] Permission denied: '../../../../../blobs' Traceback (most recent call last): File "/home/mao/.az-cli-newpy.venv/lib/python3.13/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 677, in execute raise ex File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 820, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 789, in _run_job result = cmd_copy(params) File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 335, in __call__ return self.handler(*args, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/command_operation.py", line 120, in handler return op(**command_args) File "/home/mao/.azure/cliextensions/confcom/azext_confcom/custom.py", line 197, in acipolicygen_confcom policy.populate_policy_content_for_all_images( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ individual_image=bool(image_name), tar_mapping=tar_mapping, faster_hashing=faster_hashing ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/mao/.azure/cliextensions/confcom/azext_confcom/security_policy.py", line 485, in populate_policy_content_for_all_images image_info, tar = get_image_info(progress, message_queue, tar_mapping, image) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.azure/cliextensions/confcom/azext_confcom/template_util.py", line 111, in get_image_info image_info = os_util.map_image_from_tar_backwards_compatibility( image_name, tar_file, tar_location ) File "/home/mao/.azure/cliextensions/confcom/azext_confcom/os_util.py", line 190, in map_image_from_tar_backwards_compatibility tar.extract(info_file.name, path=tar_dir) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2416, in extract self._extract_one(tarinfo, path, set_attrs, numeric_owner) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2464, in _extract_one self._handle_fatal_error(e) ~~~~~~~~~~~~~~~~~~~~~~~~^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2458, in _extract_one self._extract_member(tarinfo, os.path.join(path, tarinfo.name), ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ set_attrs=set_attrs, ^^^^^^^^^^^^^^^^^^^^ numeric_owner=numeric_owner, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ filter_function=filter_function, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ extraction_root=path) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2539, in _extract_member os.makedirs(upperdirs, exist_ok=True) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen os>", line 218, in makedirs File "<frozen os>", line 228, in makedirs PermissionError: [Errno 13] Permission denied: '../../../../../blobs' To check existing issues, please visit: https://github.com/Azure/azure-cli/issues After: > az confcom acipolicygen --image mcr.microsoft.com/aci/skr:2.14 --tar image-docker-malformed.tar --outraw-pretty-print | grep -C10 layers ... Pulling and hashing images...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.38percent/s] WARNING: mcr.microsoft.com/aci/skr:2.14 read from local tar file "strategy": "re2" }, { "pattern": "azurecontainerinstance_restarted_by=.+", "required": false, "strategy": "re2" } ], "exec_processes": [], "id": "mcr.microsoft.com/aci/skr:2.14", "layers": [ "a189b02d4858578459fda1dfbd7c6a4557c44208b9829e02b931771a6d611c39", "300f9661fb3d46c0f299ad6f552b7ad0c41ea5141755b0b3feaca3081a108f7a", "0afffca98bacf8e7b6e6f7982459a03219f60555523163c73c4b092e0a3deef2", "eefefd5009aed4ba4478876995d1a18aa3a670661fcc61d2e4cba6e2b79da0a1", "b868a7e1bebef40e5bf4d58fe271c0a10a351e68b12179ec019af9f6c75781ae", "8b4842f06982817534a75bcf71865213b09dfa8313229c384e5201dadbd75e25" ], ... (Validated that this matches the result of using the image reference directly without a tar, and also matches with the result of using the oci tar from `oras backup`) Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com>
necusjz
pushed a commit
that referenced
this pull request
May 12, 2026
… integrity-vhd (#9838) * {confcom}: Simplify logic in map_image_from_tar(_compatibility)? The current code extracts an item in the tar based on a path controlled by the manifest within the tar, which is prone to path traversal on Linux and Windows, and potentially other path confusion attacks on Windows if the Config field in the manifest contains special names. This extraction is unnecessary as we can read the content of the config directly from the tar file using extractfile. This commit simplifies it to do that. Also, there are two versions of the map_image_from_tar function, with the second one introduced in #7414. Later code changes in #8238 means that these functions now do basically the same thing (with different clean up code). After this simplification, these two functions are exactly the same, so let's also remove the _compatibility one. Before: > az confcom acipolicygen --image mcr.microsoft.com/aci/skr:2.14 --tar image-docker-malformed.tar --outraw-pretty-print ... Pulling and hashing images...: 0%| | 0/2 [00:00<?, ?percent/s] ERROR: The command failed with an unexpected error. Here is the traceback: ERROR: [Errno 13] Permission denied: '../../../../../blobs' Traceback (most recent call last): File "/home/mao/.az-cli-newpy.venv/lib/python3.13/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 677, in execute raise ex File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 820, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 789, in _run_job result = cmd_copy(params) File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py", line 335, in __call__ return self.handler(*args, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/mao/src/github.com/Microsoft/azure-cli/src/azure-cli-core/azure/cli/core/commands/command_operation.py", line 120, in handler return op(**command_args) File "/home/mao/.azure/cliextensions/confcom/azext_confcom/custom.py", line 197, in acipolicygen_confcom policy.populate_policy_content_for_all_images( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ individual_image=bool(image_name), tar_mapping=tar_mapping, faster_hashing=faster_hashing ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/mao/.azure/cliextensions/confcom/azext_confcom/security_policy.py", line 485, in populate_policy_content_for_all_images image_info, tar = get_image_info(progress, message_queue, tar_mapping, image) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.azure/cliextensions/confcom/azext_confcom/template_util.py", line 111, in get_image_info image_info = os_util.map_image_from_tar_backwards_compatibility( image_name, tar_file, tar_location ) File "/home/mao/.azure/cliextensions/confcom/azext_confcom/os_util.py", line 190, in map_image_from_tar_backwards_compatibility tar.extract(info_file.name, path=tar_dir) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2416, in extract self._extract_one(tarinfo, path, set_attrs, numeric_owner) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2464, in _extract_one self._handle_fatal_error(e) ~~~~~~~~~~~~~~~~~~~~~~~~^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2458, in _extract_one self._extract_member(tarinfo, os.path.join(path, tarinfo.name), ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ set_attrs=set_attrs, ^^^^^^^^^^^^^^^^^^^^ numeric_owner=numeric_owner, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ filter_function=filter_function, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ extraction_root=path) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mao/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/tarfile.py", line 2539, in _extract_member os.makedirs(upperdirs, exist_ok=True) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen os>", line 218, in makedirs File "<frozen os>", line 228, in makedirs PermissionError: [Errno 13] Permission denied: '../../../../../blobs' To check existing issues, please visit: https://github.com/Azure/azure-cli/issues After: > az confcom acipolicygen --image mcr.microsoft.com/aci/skr:2.14 --tar image-docker-malformed.tar --outraw-pretty-print | grep -C10 layers ... Pulling and hashing images...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.38percent/s] WARNING: mcr.microsoft.com/aci/skr:2.14 read from local tar file "strategy": "re2" }, { "pattern": "azurecontainerinstance_restarted_by=.+", "required": false, "strategy": "re2" } ], "exec_processes": [], "id": "mcr.microsoft.com/aci/skr:2.14", "layers": [ "a189b02d4858578459fda1dfbd7c6a4557c44208b9829e02b931771a6d611c39", "300f9661fb3d46c0f299ad6f552b7ad0c41ea5141755b0b3feaca3081a108f7a", "0afffca98bacf8e7b6e6f7982459a03219f60555523163c73c4b092e0a3deef2", "eefefd5009aed4ba4478876995d1a18aa3a670661fcc61d2e4cba6e2b79da0a1", "b868a7e1bebef40e5bf4d58fe271c0a10a351e68b12179ec019af9f6c75781ae", "8b4842f06982817534a75bcf71865213b09dfa8313229c384e5201dadbd75e25" ], ... (Validated that this matches the result of using the image reference directly without a tar, and also matches with the result of using the oci tar from `oras backup`) Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> * {confcom} Bump version to 2.0.0b3 * {confcom}: Fix type on load_json_from_str and read_file_from_tar * {confcom}: Release 2.0.0 (non-preview) * {confcom}: Bump integrity-vhd to v2.1 Take Windows version check for C-WCOW policy generation --------- Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docker changed how tar files are formatted so this is adding compatibility for the new format. Also adding a flag to hash images faster to the detriment of more RAM usage.
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command
az confcom
General Guidelines
azdev style <YOUR_EXT>locally? (pip install azdevrequired)python scripts/ci/test_index.py -qlocally? (pip install wheel==0.30.0required)For new extensions:
About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update
src/index.jsonautomatically.You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify
src/index.json.