[lazyinit] add correctness verification

[ver217]

Overview

Current implementation is not tested on many models. We have to add large scale correctness verification.

Wanna track the development progress? Take a look at

proposal: #3124
kanban: Lazy initialization

Goal

Pass the most models in torchvision, timm, transformers, torchrec, torchaudio and diffusers.

[super-dainiu]

I think it might be hard to satisfy torch.all_close condition.
checking the norm of weights might be an alternative.

[ver217]

I think it might be hard to satisfy torch.all_close condition. checking the norm of weights might be an alternative.

We will fix the random seed before each OP. All params and buffers should be equal to the non-lazy ones.