Unable to simulate large GEMMs using example experiment

[ncrago]

Hello,

I am trying to do some GEMM sweeps using the experiments/gemm.py script (via the docker container provided by the repo). When the GEMM sizes are large (e.g. MNK=16384), PyTorchSim crashes, and I am not sure how to proceed. My understanding is that the configuration (systolic_ws_128x128_c2_simple_noc_tpuv4.json) configures 32 channels x 8Gb (32GB total), and that the size should fit just fine. Can you please suggest next steps? Attached is the full log from the run.

Thank you,
Neal Crago

gemm16384.log

[YWHyuk]

Hello @ncrago

Thank you for reaching out and for your interest in PyTorchSim.

The crash you're seeing with large GEMM sizes (MNK=16384) is an issue in the current master image, specifically during the linking process. Fortunately, this has already been resolved in our latest development version.

Please try pulling the following image:

docker pull ghcr.io/psal-postech/torchsim-test-2-8:0f7f5425a489a11da35fb3970941c5b2a6561f87

Please let me know if you still encounter any issues with this version.

[ncrago]

Hi Wonhyuk,

Thanks for responding to my issue. I'm using the new docker image, and I cannot run the experiments/ - getting this error message when running "python experiments/layernorm.py" (and also for the gemm.py):

Traceback (most recent call last):
File "/workspace/PyTorchSim/experiments/layernorm.py", line 48, in
run_layernorm(size, config)
File "/workspace/PyTorchSim/experiments/layernorm.py", line 11, in run_layernorm
scheduler = Scheduler(num_request_queue=1, engine_select=Scheduler.FIFO_ENGINE, togsim_config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/PyTorchSim/Scheduler/scheduler.py", line 354, in init
self.tog_simulator.interactive_simulation()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TOGSimulator' object has no attribute 'interactive_simulation'

Any help is appreciated.

Thanks,
Neal