Skip to content

Allow device_mesh have multiple dim #38949

Merged
SunMarc merged 8 commits intomainfrom
fsdp2-tp
Jul 23, 2025
Merged

Allow device_mesh have multiple dim #38949
SunMarc merged 8 commits intomainfrom
fsdp2-tp

Conversation

@S1ro1
Copy link
Copy Markdown
Contributor

@S1ro1 S1ro1 commented Jun 20, 2025

Experimental PR for me to just make stuff work on the accelerate side, will follow up with this after.

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Copy Markdown
Collaborator

@winglian winglian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like all the needed changes to me! (plus #39562)

Comment thread src/transformers/modeling_utils.py Outdated
@S1ro1 S1ro1 marked this pull request as ready for review July 23, 2025 10:17
Copy link
Copy Markdown
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM !

@SunMarc SunMarc changed the title [WIP] FSDP2 + TP + (other) Allow device_mesh have multiple dim Jul 23, 2025
@SunMarc SunMarc enabled auto-merge (squash) July 23, 2025 12:15
@SunMarc SunMarc merged commit 82603b6 into main Jul 23, 2025
26 checks passed
@SunMarc SunMarc deleted the fsdp2-tp branch July 23, 2025 12:27
@S1ro1 S1ro1 restored the fsdp2-tp branch July 26, 2025 14:01
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
* Feat: something

* Feat: initial changes

* tmp changes to unblock

* Refactor

* remove todo

* Feat: docstring

---------

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants