Skip to content

make gptq convert work for moe#4330

Merged
vkuzo merged 72 commits intomainfrom
gh/vkuzo/263/head
Apr 27, 2026
Merged

make gptq convert work for moe#4330
vkuzo merged 72 commits intomainfrom
gh/vkuzo/263/head

Conversation

@vkuzo
Copy link
Copy Markdown
Contributor

@vkuzo vkuzo commented Apr 24, 2026

Summary:

Makes gptq + moe + nvfp4 work e2e, results as expected on tiny model +
tiny dataset

Test Plan:

> TRITON_ALLOW_NON_CONSTEXPR_GLOBALS=1 torchao/prototype/gptq/gptq_nvfp4_olmoe_1b_7b_nonsequential_wikitext.sh

bf16
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.5895|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5047|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |8.8910|±  |   N/A|
real    0m59.219s
user    0m42.554s
sys     0m20.534s

nvfp4-rtn

| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6024|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5183|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3277|±  |   N/A|
real    0m42.528s
user    0m41.217s
sys     0m12.817s

nvfp4-nonsequential with 4096 calibration samples on c4
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6019|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5177|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3087|±  |   N/A|
real    22m28.505s
user    22m36.008s
sys     0m13.872s

vkuzo added 30 commits April 20, 2026 20:52
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
vkuzo added 4 commits April 24, 2026 14:04
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
@vkuzo
Copy link
Copy Markdown
Contributor Author

vkuzo commented Apr 24, 2026

Stack from ghstack (oldest at bottom):

@vkuzo vkuzo requested a review from jerryzh168 as a code owner April 24, 2026 17:38
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented Apr 24, 2026

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/4330

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

⏳ 11 Pending, 1 Unrelated Failure

As of commit 65c616d with merge base 1c5e71e (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 24, 2026
@vkuzo vkuzo added the module: not user facing Use this tag if you don't want this PR to show up in release notes label Apr 24, 2026
vkuzo added 7 commits April 27, 2026 12:06
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
vkuzo added a commit that referenced this pull request Apr 27, 2026
Summary:

Makes gptq + moe + nvfp4 work e2e, results as expected on tiny model +
tiny dataset

Test Plan:

```
> TRITON_ALLOW_NON_CONSTEXPR_GLOBALS=1 torchao/prototype/gptq/gptq_nvfp4_olmoe_1b_7b_nonsequential_wikitext.sh

bf16
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.5895|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5047|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |8.8910|±  |   N/A|
real    0m59.219s
user    0m42.554s
sys     0m20.534s

nvfp4-rtn

| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6024|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5183|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3277|±  |   N/A|
real    0m42.528s
user    0m41.217s
sys     0m12.817s

nvfp4-nonsequential with 4096 calibration samples on c4
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6019|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5177|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3087|±  |   N/A|
real    22m28.505s
user    22m36.008s
sys     0m13.872s

```
ghstack-source-id: 94d4d29
ghstack-comment-id: 4315147581
Pull-Request: #4330
vkuzo added 2 commits April 27, 2026 12:15
[ghstack-poisoned]
[ghstack-poisoned]
vkuzo added a commit that referenced this pull request Apr 27, 2026
Summary:

Makes gptq + moe + nvfp4 work e2e, results as expected on tiny model +
tiny dataset

Test Plan:

```
> TRITON_ALLOW_NON_CONSTEXPR_GLOBALS=1 torchao/prototype/gptq/gptq_nvfp4_olmoe_1b_7b_nonsequential_wikitext.sh

bf16
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.5895|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5047|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |8.8910|±  |   N/A|
real    0m59.219s
user    0m42.554s
sys     0m20.534s

nvfp4-rtn

| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6024|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5183|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3277|±  |   N/A|
real    0m42.528s
user    0m41.217s
sys     0m12.817s

nvfp4-nonsequential with 4096 calibration samples on c4
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6019|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5177|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3087|±  |   N/A|
real    22m28.505s
user    22m36.008s
sys     0m13.872s

```
ghstack-source-id: 94d4d29
ghstack-comment-id: 4315147581
Pull-Request: #4330
[ghstack-poisoned]
vkuzo added a commit that referenced this pull request Apr 27, 2026
Summary:

Makes gptq + moe + nvfp4 work e2e, results as expected on tiny model +
tiny dataset

Test Plan:

```
> TRITON_ALLOW_NON_CONSTEXPR_GLOBALS=1 torchao/prototype/gptq/gptq_nvfp4_olmoe_1b_7b_nonsequential_wikitext.sh

bf16
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.5895|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5047|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |8.8910|±  |   N/A|
real    0m59.219s
user    0m42.554s
sys     0m20.534s

nvfp4-rtn

| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6024|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5183|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3277|±  |   N/A|
real    0m42.528s
user    0m41.217s
sys     0m12.817s

nvfp4-nonsequential with 4096 calibration samples on c4
| Tasks  |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|-----:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  |0.6019|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  |1.5177|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |9.3087|±  |   N/A|
real    22m28.505s
user    22m36.008s
sys     0m13.872s

```
ghstack-source-id: 94d4d29
ghstack-comment-id: 4315147581
Pull-Request: #4330
@vkuzo vkuzo changed the base branch from gh/vkuzo/262/head to main April 27, 2026 12:16
@vkuzo vkuzo merged commit c2e3c87 into main Apr 27, 2026
33 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: not user facing Use this tag if you don't want this PR to show up in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants