Clean up linear_int8_dynamic_activation_intx_weight_subclass by metascroy · Pull Request #1553 · pytorch/ao

metascroy · 2025-01-11T04:00:22Z

Summary:
Cleans up layout and quantization API:

int8_dynamic_activation_intx_weight(
    weight_dtype=torch.int4,
    granularity=PerGroup(128),
    has_weight_zeros: bool = False,
    weight_mapping_type=MappingType.ASYMMETRIC,
    act_mapping_type=MappingType.ASYMMETRIC,
    layout=PackedLinearInt8DynamicActivationIntxWeightLayout(),
)

int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower).

The fallback option is removed and instead relies on using PlainLayout().

Reviewed By: jerryzh168

Differential Revision: D67821939

pytorch-bot · 2025-01-11T04:00:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1553

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f04f3b2 with merge base de5c6e1 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-01-11T04:00:57Z

This pull request was exported from Phabricator. Differential Revision: D67821939

jerryzh168 · 2025-01-11T05:37:21Z

 quantize_(
    my_model,
    int8_dynamic_activation_intx_weight(
+        bit_width=4,


this could be updated to using real dtype now if you can use torch nightly btw, both torch.uintx and torch.intx are available in nightly (2.6 and later I think)

Changed bit_width to weight_dtype

please change README as well

Updated readme

facebook-github-bot · 2025-01-13T05:00:28Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T05:06:47Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T18:57:32Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T19:10:38Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

jerryzh168 · 2025-01-13T21:21:39Z

+            + "  Alternatively, use layout=PlainLayout() with int8_dynamic_activation_intx_weight, but note that doing so will result in much slower performance."
+        )
+
+    dtype_to_bit_width = {


why not pass this around to layout as well?

I think for layout, bit_width is more convenient a more to use because I can then do something like this to call the kernel:

getattr(torch.ops.torchao, f"_pack_8bit_act_{layout.bit_width}bit{wzp_suffix}_weight")(*args)

I can change, though, if you think it's better.

I see, depending on whether there could be cases of uintx as well I think, if not, then it's fine since it's not user facing

Let's leave it as bitwidth then. The quantizer is specifically for intx, not uintx, and the layout is not user facing.

sg, also we have a util here:

ao/torchao/quantization/quant_primitives.py

Line 179 in b3deb16

_DTYPE_TO_BIT_WIDTH.update(

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T22:09:06Z

This pull request was exported from Phabricator. Differential Revision: D67821939

jerryzh168 · 2025-01-14T20:06:04Z

+        quant_min = -(1 << (bit_width - 1))
+        quant_max = (1 << (bit_width - 1)) - 1


we also have utils here:

ao/torchao/quantization/quant_primitives.py

Line 168 in b3deb16

_SUB_BYTE_INT_BOUNDS.update(

jerryzh168

LGTM, thanks

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 11, 2025

facebook-github-bot added the fb-exported label Jan 11, 2025

metascroy requested a review from jerryzh168 January 11, 2025 04:01

metascroy mentioned this pull request Jan 11, 2025

[Feat]: Add support for kleidiai quantization schemes #1447

Merged

jerryzh168 reviewed Jan 11, 2025

View reviewed changes

metascroy force-pushed the export-D67821939 branch from 8752c4c to 23688a9 Compare January 13, 2025 05:00

metascroy added the topic: improvement Use this tag if this PR is an improvement (doesn't fit into any of the other categories) label Jan 13, 2025

metascroy force-pushed the export-D67821939 branch from 23688a9 to 8d47b97 Compare January 13, 2025 05:06

metascroy force-pushed the export-D67821939 branch from 8d47b97 to 0552dcf Compare January 13, 2025 18:57

metascroy force-pushed the export-D67821939 branch from 0552dcf to dc32105 Compare January 13, 2025 19:10

metascroy requested a review from jerryzh168 January 13, 2025 21:07

jerryzh168 reviewed Jan 13, 2025

View reviewed changes

metascroy force-pushed the export-D67821939 branch from dc32105 to f04f3b2 Compare January 13, 2025 22:09

metascroy requested a review from jerryzh168 January 14, 2025 17:36

jerryzh168 reviewed Jan 14, 2025

View reviewed changes

jerryzh168 approved these changes Jan 14, 2025

View reviewed changes

facebook-github-bot merged commit 0bc5b00 into pytorch:main Jan 14, 2025

		quant_min = -(1 << (bit_width - 1))
		quant_max = (1 << (bit_width - 1)) - 1

Conversation

metascroy commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1553

✅ No Failures

Uh oh!

facebook-github-bot commented Jan 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 13, 2025

Uh oh!

facebook-github-bot commented Jan 13, 2025

Uh oh!

facebook-github-bot commented Jan 13, 2025

Uh oh!

facebook-github-bot commented Jan 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryzh168 Jan 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryzh168 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

metascroy commented Jan 11, 2025 •

edited

Loading

pytorch-bot Bot commented Jan 11, 2025 •

edited

Loading

jerryzh168 Jan 14, 2025 •

edited

Loading