Skip to content

[fx] add vanilla activation checkpoint search with test on resnet and densenet#1433

Merged
super-dainiu merged 12 commits intohpcaitech:mainfrom
super-dainiu:feature/fx_ckpt_chen
Aug 11, 2022
Merged

[fx] add vanilla activation checkpoint search with test on resnet and densenet#1433
super-dainiu merged 12 commits intohpcaitech:mainfrom
super-dainiu:feature/fx_ckpt_chen

Conversation

@super-dainiu
Copy link
Copy Markdown
Contributor

What's new?

Now you can use the greedy algorithm from Training Deep Nets with Sublinear Memory Cost (Chen et al. 2016) with two lines of code.

B = 5 * 1024 * 1024 * 1024  # An approximate memory budget of 5GB
model = resnet18()
input_sample = torch.rand(4, 3, 224, 224)
gm = symbolic_trace(model)
MetaInfoProp(gm).run(input_sample)
gm = chen_greedy(gm, B)     # gm will run with activation checkpoint

Comment thread colossalai/fx/passes/algorithms/ckpt_solver_chen.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants