A question about running speed

[junranhe]

I test many time in inception-bn, I found my machine run mxnet is so fast at the beginning(184 sam/sec), but slowly down litte later(174 sam/sec), why?how can I run mxnet full speed all the time? temperatures too hot(max is 63'c)?my cpu rate is 500% (full is 1200%), 6 core, (12 thread core)
INFO:root:Start training with [gpu(0), gpu(1), gpu(2)]
INFO:root:Iter[0] Batch [10] Speed: 201.07 samples/sec
INFO:root:Iter[0] Batch [20] Speed: 188.29 samples/sec
INFO:root:Iter[0] Batch [30] Speed: 183.96 samples/sec
INFO:root:Iter[0] Batch [40] Speed: 184.11 samples/sec
INFO:root:Iter[0] Batch [50] Speed: 184.51 samples/sec
INFO:root:Iter[0] Batch [60] Speed: 185.40 samples/sec
INFO:root:Iter[0] Batch [70] Speed: 178.27 samples/sec
INFO:root:Iter[0] Batch [80] Speed: 176.74 samples/sec
INFO:root:Iter[0] Batch [90] Speed: 174.46 samples/sec
INFO:root:Iter[0] Batch [100] Speed: 175.03 samples/sec
INFO:root:Iter[0] Batch [110] Speed: 171.43 samples/sec
INFO:root:Iter[0] Batch [120] Speed: 174.72 samples/sec

[tqchen]

We also observe this slow slow down and stabilization in most cases. So for now this is normal phenomenon for mxnet

This could due to GPU freq slow down or other reasons, will look into it more.

[junranhe]

well, I see, there is a detail: I run my machine on kvstore_type = 'local' before, It run 68 sam/sec(very slow, use 'device' can run 120 sam/sec), but it first 10 batches run 78 sam/sec, so what ever the gpu speed run, it just slow down...

[tqchen]

I am going to close this for now, as this is the current expected behavior, we will be looking more into the reason in the future. Thanks!

[jsusskin]

I'm curious if there has been any recent work on the stability problem with mxnet.
I've noticed some odd behavior where training will progress at a stable rate for a while and then suddenly slow down a bunch for a few epochs before speeding up again. It seems to occur more frequently when dealing with large amounts of data.