Define a very basic AD, for testing and for demonstration

[oxinabox]

Was discussing that it would be good
if ChainRules.jl defined a very dumb naive AD,
like some kind of cutdown Flux.Tracker or Nabla tape based thing.

for purposes of demonstrating how ChainRules would be used with an AD,
and for testing the rules.

This could be defined in the source (perhaps in a submodule),
or in the tests.

[ararslan]

The tests would probably be a better place for it, lest someone start relying on it for actual use...

[willtebbutt]

The tests would probably be a better place for it, lest someone start relying on it for actual use...

Given @jrevels previous experience with Capstan, I can see your apprehension here @ararslan . Maybe a gist, rather than something ChainRules' tests, would be the right place for it? Maybe it could even make an appearance in the docs... 🙂

[jrevels]

Here's an example to get people started: https://gist.github.com/jrevels/4b75ce24d8563888e54a89f6fdf2ff97

That's a toy scalar forward-mode AD using ChainRules + Cassette, based off of the toy forward-mode AD in Cassette's tests. It should be able to calculate the derivative of x w.r.t. f where f is called as f(x), x is a real-valued scalar, and f is a composition of:

1. any unary/binary function with a corresponding frule
2. nested calls to the differentiation operator
3. any other Julia function composed of 1. and 2.

I literally cooked it up in 30 minutes so it's probably ugly, chock full of edge cases, bugs and other issues (performance is probably terrible because the Julia compiler is still missing the requisite optimizations Cassette's tagging system was designed against), but hopefully it suits the purpose of illustrating basic usage for compiler-based ADs. Reverse-mode doesn't look much different except, well, you're going in reverse and using rrule 😛

For non-compiler-based ADs, ChainRules can simply replace DiffRules, existing functional rule systems etc. but there shouldn't be too much surprising stuff going on in that regard. Most of the interesting stuff will happen when/if downstream tools want to start supporting ChainRules' more advanced features (e.g. the different differential types).

More examples for both would be very helpful, and will hopefully slowly trickle in along with tests/docs.

[willtebbutt]

I did a Nabla.jl-style version of the above for reverse-mode:
https://gist.github.com/willtebbutt/39205ab845b22e6452a42705eac8d254

As with @jrevels forwards-mode example, it's just really to give a vague idea of the kind of thing you might try doing.

[oxinabox]

This is me brushing up on how DualNumbers work
A scalar forwardmode AD that uses Cassette+ChainRules+DualNumbers

https://gist.github.com/oxinabox/d6e07dc58b1e0b10b5e15b23a5b0346a

[nickrobinson251]

https://github.com/JuliaDiff/ChainRulesCore.jl/pull/182/files#diff-5de55b09272e1aca93ddbbde28456802

[oxinabox]

Closed by #182 's docs+tests.