Skip to content

[fx] Add activation checkpoint solver rotor#1496

Merged
Cypher30 merged 37 commits intohpcaitech:mainfrom
Cypher30:feature/add_ckpt_solver_rotor
Aug 26, 2022
Merged

[fx] Add activation checkpoint solver rotor#1496
Cypher30 merged 37 commits intohpcaitech:mainfrom
Cypher30:feature/add_ckpt_solver_rotor

Conversation

@Cypher30
Copy link
Copy Markdown
Contributor

What's New?

In this PR, we introduce rotor algorithm to our activation checkpoint solver lists. By the great effort in #1480 , we are able to estimate the metrics we need to use rotor algorithm. Down below is a quick example to try this new feature

import torch
import torch.nn as nn
import torch.nn.functional as F
from colossalai.fx import ColoTracer, ColoGraphModule
from colossalai.fx.passes.algorithms import linearize, solver_rotor
from torch.fx import GraphModule
from torchvision.models import resnet18
from colossalai.fx.codegen import ActivationCheckpointCodeGen

model = torch_resnet18()
input = torch.rand(32, 3, 224, 224, device="meta")
tracer = ColoTracer()
graph = tracer.trace(torch_model, meta_args={"x": input})
graph.set_codegen(ActivationCheckpointCodeGen())
gm = ColoGraphModule(model, graph)
gm = solver_rotor(gm, input, mem_limit=600 * 1024 * 1024)
gm.to_folder("./fx")

I also add the unit test for it in test_ckpt_torchvion.py, currently on our CI we will skip this test, as we haven't implemented our ColoGraphModule in torch11, I will attach the test results on torch12 later.

Cypher30 and others added 30 commits July 14, 2022 16:07
…lossalAI into hotfix/fix_the_ckpt_func_def
@Cypher30
Copy link
Copy Markdown
Contributor Author

Screen Shot 2022-08-25 at 20 04 51

Above the test result in `test_fx` with torch12

Comment thread colossalai/fx/passes/algorithms/utils.py
Comment thread colossalai/fx/passes/algorithms/linearize.py
Comment thread colossalai/fx/passes/algorithms/utils.py Outdated
@Cypher30 Cypher30 merged commit de1e716 into hpcaitech:main Aug 26, 2022
@Cypher30 Cypher30 deleted the feature/add_ckpt_solver_rotor branch August 26, 2022 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants