Add specialized StaticArrays-based methods

[jrevels]

Adds StaticArrays as dependency, and uses it to support a low-dimensional, stack-allocated differentiation API.

julia> using BenchmarkTools, ForwardDiff, StaticArrays

julia> x = rand(3, 3, 3);

julia> sx = SArray{Tuple{3,3,3}}(x);

julia> out = similar(x);

julia> cfg = ForwardDiff.GradientConfig(sum, x);

# naive old method
julia> @btime ForwardDiff.gradient(sum, $x);
  6.585 μs (3 allocations: 3.23 KiB)

# non-allocating old method
julia> @btime ForwardDiff.gradient!($out, sum, $x, $cfg);
  666.856 ns (0 allocations: 0 bytes)

# stack-allocated new method
julia> @btime ForwardDiff.gradient(sum, $sx);
  9.394 ns (0 allocations: 0 bytes)

Currently, I've only added support for gradients and Jacobians. EDIT: Added special cases for Hessians as well, see below. I still need to add back in the tests I originally used in #213. Done.

[aaowens]

Is this expected? The speedup for gradient! is minor.

julia> @btime ForwardDiff.gradient(sum, $sx);
  10.415 ns (0 allocations: 0 bytes)

julia> out2 = similar(sx);

julia> cfg2 = ForwardDiff.GradientConfig(sum, sx);

julia> @btime ForwardDiff.gradient!($out, sum, $x, $cfg);
  705.035 ns (0 allocations: 0 bytes)

julia> @btime ForwardDiff.gradient!($out2, sum, $sx, $cfg2);
  657.787 ns (0 allocations: 0 bytes)

This PR sounds really useful for tools like Optim on low-dimensional functions, both because of the potential differentiation speedup, and because returning the gradient as an SArray should enable the fast matrix solve path.

[jrevels]

Is this expected? The speedup for gradient! is minor.

Yes, this is expected. The main benefit of gradient!(out, f, ::SArray) is that an SArray is passed to the target function, which might speed up evaluation quite a bit depending on the target function. However, for simple target functions like sum, gradient!(out, f, ::SArray) will spend the bulk of its time extracting derivatives to out.

Note this extraction process is not unrolled like it is for gradient(f, ::SArray). I'm not sure how much benefit we'd see from unrolling it (since out is heap-allocated anyway), but I'll play around with it.

[jrevels]

Just added some special cases for Hessians as well:

julia> x = rand(3, 3);

julia> sx = SArray{Tuple{3, 3}}(x);

julia> out = similar(x, 9, 9);

julia> cfg = ForwardDiff.HessianConfig(prod, x);

# naive old method
julia> @btime ForwardDiff.hessian(prod, $x);
  37.159 μs (17 allocations: 17.11 KiB)

# almost non-allocating old method
julia> @btime ForwardDiff.hessian!($out, prod, $x, $cfg);
  2.115 μs (2 allocations: 832 bytes)

# stack-allocated new method
julia> @btime ForwardDiff.hessian(prod, $sx);
  1.220 μs (0 allocations: 0 bytes)

[jrevels]

Tests pass locally, and everything here is ready. However, I developed this against StaticArrays master, so this PR is now waiting on JuliaArrays/StaticArrays.jl#142.

[jrevels]

Argh, 32 bit Windows...AppVeyor was passing before this PR, and it seems like this PR doesn't touch the code that would cause that test to fail, so I'm not sure what's going on here.

[KristofferC]

Cutting the Gordian knot.