Adding Batch L2 Normalization Layer that L2 normalizes rows of input Tensor

[karpathy]

This layer L2 normalizes an n x d Tensor. I tested the implementation on both CPU/GPU and gradient checked it with jac and also with my (other) custom code. I also compared to a slower version that uses a loop, as can be seen in this gist. (This batched version is quite a lot faster)

Also, I have only worked with Torch for ~2 weeks so exercise caution :)

[soumith]

will give in-line feedback. this has some common patterns that we dont want in nn.

[soumith]

torch.xxx(...) usually does a malloc for the result tensor.
mallocs during the training loop of the nn are usually quite bad for performance.
Another reason mallocs are bad is that they introduce a synchronize point in CUDA, so you essentially kill any multi-GPU code.

instead, consider having a buffer.

self.buffer = self.buffer or input.new()
self.norms = self.norms or input.new()
self.norms:sum(self.buffer:cmul(input, input), 2):sqrt()

[karpathy]

Thanks,

some of this was intentional on my part because I'm applying this in space-limited setting (I'm constantly running out of memory on GPU), so saving all of these cached intermediates (+ at every step of my LSTM) was costly. But if this is considered preferable I'll rewrite.

EDIT: I did not appreciate the point about multi-GPU code, interesting.

[soumith]

overall the barrier of contributing is lower to nnx (nn-experimental) which is here: https://github.com/clementfarabet/lua---nnx

I made a bunch of comments, but hope they dont stop you from contributing in the future :)

[karpathy]

Thanks for comments! Some of these issues I was aware of and some of them were just silly mistakes (e.g. reshape) - sorry about that. I did a rewrite that should now be much more time-efficient (at cost of space efficiency (due to held caches) and readability), but it works.

[Atcold]

I think it's missing the implementation when input:dim() == 1. If a user sends a non-batched input, the module should output the normalised single vector (see this implementation, which uses an approximate gradient computation but works for single samples as well).

[soumith]

squashed the commits and pushed to master via 905ea8c

[karpathy]

yay!! By the way, suggested improvements going forward:

• Incorporate a smoothing term to prevent division by zero (?) Or perhaps this should this be optional? Not clear.
• Allow 1-dimensional inputs as suggested by Atcold.

[Atcold]

@karpathy, I would suggest

• a smoothing by default of 1e-5 like shown here. If the user wants to set something else, then let her do it by an optional parameter [eps] in the module initialisation;
• if you don't want to write new code, 1-dimensional inputs could be viewed as 1 x length. So if dim(input) == 1 then newInput = input:view(1,-1);
• to add the corresponding documentation, so we avoid having ghost features.

And, thank you for the time you took to help the Torch's community 😁

[nicholas-leonard]

Some documentation would be nice.

[soumith]

I totally missed that this PR doesn't have documentation. Indeed some documentation would be nice...

[karpathy]

I'll add docs. I have a patch for eps and I tried to patch the support for 1D case but that ended up taking an hour yesterday, it's not as trivial as changing the view with 1 line of code. I'll work on it.