Share a list of weight attributes instead of a single one in TiedLayerSpec API#2035
Share a list of weight attributes instead of a single one in TiedLayerSpec API#2035thomasw21 wants to merge 8 commits intodeepspeedai:masterfrom
Conversation
thomasw21
requested review from
RezaYazdaniAminabadi,
ShadenSmith,
arashb,
awan-10,
cli99,
conglongli,
duli2012,
eltonzheng,
jeffra,
minjiaz,
mrwyattii,
samyam,
tjruwase,
xiaoxiawu-microsoft and
yaozhewei
as code owners
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
| self.fwd_map = {} | ||
| self.tied_modules = nn.ModuleDict() | ||
| self.tied_weight_attrs = {} | ||
| self.tied_weight_attrss = {} |
There was a problem hiding this comment.
Is the double s a typo or intentional?
|
@thomasw21, thanks for the PR and apologies for the delay in reviewing. Can you please provide a clarification? It seems there are two issues being addressed in this PR: (1) supporting user-specified forward function and (2) list of weight attributes. If this is correct, I think it might be better to split into two separate PRs. What do you think? Thanks! |
|
Indeed! Sorry I've sort of dropped this as well currently, as we've been focusing on other aspects of BigScience. I'll try split the PRs when I get the chance! My bad! |
|
@tjruwase Are these changes we still want? If so I can revive them with the current develop branch. |
@thomasw21, are you still interested in this PR? Are you fine with #4216 replacing this? |
|
Hey! I'm fine with #4216 overriding this PR :D |
4 participants
No description provided.