Skip to content

[autockpt] linearize / merge shape-consistency nodes.#2271

Merged
super-dainiu merged 4 commits intohpcaitech:debug/ckpt-autoparallelfrom
super-dainiu:debug/ckpt-autoparallel
Jan 3, 2023
Merged

[autockpt] linearize / merge shape-consistency nodes.#2271
super-dainiu merged 4 commits intohpcaitech:debug/ckpt-autoparallelfrom
super-dainiu:debug/ckpt-autoparallel

Conversation

@super-dainiu
Copy link
Copy Markdown
Contributor

Merge all shape-consistency nodes to a large node in CkptSolver.
This is because:

  • shape-consistency nodes don't affect the performance of rematerialization
  • shape-consistency nodes don't have intermediate activation
  • reducing search space will speed up search time
[layer4_2_conv1, layer4_2_bn1, layer4_2_relu, layer4_2_conv2, layer4_2_bn2, layer4_2_relu_1, layer4_2_conv3, layer4_2_bn3, add_15, layer4_2_relu_2, runtime_comm_spec_apply] 0 9 0 9
[avgpool] 0 0 1 3
[flatten, fc_weight, fc_bias, runtime_apply_1, runtime_comm_spec_apply_1] 1 1 0 0
[linear, runtime_comm_spec_apply_2] 0 1 0 1

@super-dainiu super-dainiu merged commit b0d21d0 into hpcaitech:debug/ckpt-autoparallel Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant