From 4695c69a209eeba4513a09074fe8a13f37fc44f9 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 27 Jul 2021 21:12:59 +0100 Subject: [PATCH 01/17] Document sub/super-differential convention --- docs/make.jl | 1 + docs/src/nondiff_points.md | 146 +++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 docs/src/nondiff_points.md diff --git a/docs/make.jl b/docs/make.jl index 72d1cd8cd..3e1233608 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -58,6 +58,7 @@ makedocs(; "`RuleConfig`" => "rule_author/superpowers/ruleconfig.md", "Gradient accumulation" => "rule_author/superpowers/gradient_accumulation.md", ], + "Non-differentiable Points" => "nondiff_points.md", "Converting ZygoteRules.@adjoint to rrules" => "rule_author/converting_zygoterules.md", "Tips for making your package work with AD" => "rule_author/tips_for_packages.md", "Debug mode" => "rule_author/debug_mode.md", diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md new file mode 100644 index 000000000..668d7c3e4 --- /dev/null +++ b/docs/src/nondiff_points.md @@ -0,0 +1,146 @@ +# What to return for non-differentiable points + +In calculus one learns that if the derivative as computed by approaching from the left, +and the derivative one computes as approaching from the right are not equal then the derivative is not defined, +and we say the function is not differentiable at that point. +This is distinct from the notion captured by [`NoTangent`](@ref), which is that the tangent space itself is not defined: because in some sense the primal value can not be perturbed e.g. is is a discrete type. + +However, contrary to what calculus says most autodiff systems will return an answer for such functions. +For example for: `abs_left(x) = (x <= 0) ? -x : x`, AD will say the derivative at `x=0` is `-1`. +Alternatively for: `abs_right(x) = (x < 0) ? -x : x`, AD will say the derivative at `x=0` is `1`. +Those two examples are weird since they are equal at all points, but AD claims different derivatives at `x=0`. +The way to fix autodiff systems being weird is to write custom rules. +So what rule should we write for this case? + +The obvious answer, would be to write a rule that throws an error if input at a point where calculus says the derivative is not defined. +Another option is to return some error signally value like `NaN`. +Which you *can* do. +However, this is not useful. +Instead we introduce what we call the **sub/super-differential convention**: + +> It is always permissable to return any element of the sub/super-differential. +> You should return the most useful. + +Or equivalently but considering the trivial singleton sub/super-differential seperately: + +> At any point where the derivative is not defined, it is permissable to return any element of the sub/super-differential. +> You should return the most useful. + +We will justify this further below, but first let us discuss what it means. + +## What is the sub/super-differential? + +The subderivative is defined only for locally convex functions, where-as the super-deriviative is fined only for locally concave functions. +For our purpose we, basically never care which we are working with and so write sub/super-derivative. + +For a function $f$ at some point $x_0$ a sub/super-derivative is a real number $c$ such that there exists a open ball $\mathcal{B} \subset \mathrm{dom}(f)$ containing $x_0$, +and for all points $z \in \mathcal{B}$ the following holds: + +$$\mathrm{sub -derivative:}\qquad f(z) - f(x_0) \ge c\,(z-x_0)$$ + +$$\mathrm{super-derivative:}\qquad f(z) - f(x_0) \le c\,(z-x_0)$$ + +We call the the set of all values $c$ the sub/super-differential at $x_0$. + +More informally: consider a plot of the function. +The sub/super-differential at a point is the set of slopes of all lines you could draw touching that point, but with the lines either entirely above, or entirely below the curve. + +It is best illustrated with a figure: + +![plot showing subderiviatives](https://upload.wikimedia.org/wikipedia/commons/4/4e/Subderivative_illustration.png) + +In this figure a plot of a function is shown in blue. +Two subtangent lines are plotted in red. +Their slopes are sub/super-derivatives at $x_0$, and they are two elements of the subdifferential. +If you flip it upside down, it would apply for the super-differential, with the lines being above the curve. + +### Some key things to note. + +For a function where the derivative is defined on both sides of the point: + - the derivative on each side is a sub/super-derivative at the point + - as is the mean of the derivative of each side + - in-fact the mean of any subset (including the whole sub/super-differential) of sub/super-derivatives is also a sub/super-derivative. + - if the derivative one one side is negative and the other positive then zero is a sub/super-derivative. + +For a function that is only defined on one side of the point, the sub/super-differential is the full set of real numbers. +This by the subgradient convention leaves you free to chose *any* useful value. + +!!! info "Does AD always return a sub/super-derivative? No" + On consideration of this, one might be tempted to think that AD always returns a sub/super-derivative. + And that in normal cases it return the only sub/super-derivative i.e. the actual derivative; and in other case it picks one of the branches. + Thus all weirdness of AD disagreeing with calculus could be explained away in this way. + **This is not the case.** + As it is not necessarily true that all branches derivatives are correct sub/super-differentials for the function. + Consider the function from [Abadi and Plotkin, 2019](https://dl.acm.org/doi/10.1145/3371106): + `f(x::Float64) = x == 0.0 ? 0.0 : x`. + The correct derivative of this function is `1` at all points, but most ADs will say that at `x=0` the derivative is zero. + The fix for this, is to define a rule which *does* in fact return a sub/super-derivative. + So one could say a correctly functioning AD with all needed rules does always return a sub/super-differential. + + +## What is useful ? + +The sub/super-differential convention had two parts: +"It is always permissable to return any element of the sub/super-differential. +**You should return the most useful.**". +What is the most useful? +This is a value judgement you as a rule author will have to make. + + +### It is often zero + +If zero is a sub/super-derivative, then it is often the most useful one. +Especially if the point is a local minima/maxima +For a function like `abs` or `x -> x<0 ? x : -2x` setting the non-differentiable point to either side would result in it leaving that maxima. + +Further, a nice (albeit generally intractable) algorithm for finding the global optima is to take the set of all stationary points (i.e. points where the derivative is zero), combine that with the boundary points and evaluate the primal function at all of them. +The extrema of this set are the global optima. +This algorithm is only guaranteed correct for functions that are differentiable everywhere *or* that apply the sub/super-derivative convention and make all non-differentiable local optima claim to have a zero derivative. +It is also correct if you make other non-differentiable points have zero derivative, just slower. + +### It is sometimes the non-zero, especially if it boarders a flat section + +If a function has derivative zero in some large section of it's domain, like `relu` or `x->clamp(x, -1, 1)`, +a case can be made for choosing the non-zero derivative branch. +Depending exactly on the larger function being optimized, it is often the case that a zero gradient for this function means a total zero gradient (e.g. if the greater function is a chain of composed calls). +So once things move into the flat-region they stay there. +If we chose the other branch we move them into (just barely) the nonflat region. +If moving into the non-flat region was good they will move further there later. +If it is not good then they well be rapidly cast deeper into the flat region and we will not be at this boundary case. + +### It is sometimes the mean +Either the mean of the branches, or the overall mean of the sub/super-differential (which may be equal). + +A nice advantage of it is that it will agree with the result you will get from central finite differencing. +Which is what [ChainRulesTestUtils.jl](https://github.com/JuliaDiff/ChainRulesTestUtils.jl) defaults to. + +### All else being equal it is just one of the branches +Pick one. +It will be fast. +It will agree with either the forwards or reverse finite differencing results. + +### It is almost never `Inf`, `NaN` or an error +Practically speaking, people are generally using AD to perform some gradient descent like optimization procedure. +`NaN` and `Inf` do not generally take us to nice places. +Erroring can be worse, especially if it is a local minima -- the optimization fully converges to that minima and then throws an error rather than reporting the result. +If the primal function takes a non-finite value or errors on one side, then we are in the case that we are at a boundry of the domain. +Which means we are free to chose *any* value. +In particular we often want to chose value of the derivative from **the other side where the primal is defined.** + + +## Why is the sub/super-differential convention permissible? + +Math isn't real, and AD doesn't do calculus. +We are trying to accomplish some goal, and this is approach works well. +More specifically, consider our initial examples: +`abs_left(x) = (x <= 0) ? -x : x`, and `abs_right(x) = (x < 0) ? -x : x`. +These are a a primal level indistinguishable to the user. +It is impossible to tell which `Base.abs` uses without looking at the source. +Thus the rule author must be free to chose between assuming it is either. +Which is equivalent to saying they are free to chose to return the derivative or either branch. +We can then take the continuous relaxation of that choice: to chose any value between them. +Which for that case is choosing any sub-differential. +We then generalize from that chose into the sub/super-differential convention. + +## How does this generalize to n-D +Carefully, but consistently. \ No newline at end of file From 1242e742a44fd61ac8e1d16b824db587bb4da8e0 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 27 Jul 2021 21:26:17 +0100 Subject: [PATCH 02/17] I would like a markdown compatible spell-checker please --- docs/src/nondiff_points.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index 668d7c3e4..c4bfda88b 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -18,12 +18,12 @@ Which you *can* do. However, this is not useful. Instead we introduce what we call the **sub/super-differential convention**: -> It is always permissable to return any element of the sub/super-differential. +> It is always permissible to return any element of the sub/super-differential. > You should return the most useful. Or equivalently but considering the trivial singleton sub/super-differential seperately: -> At any point where the derivative is not defined, it is permissable to return any element of the sub/super-differential. +> At any point where the derivative is not defined, it is permissible to return any element of the sub/super-differential. > You should return the most useful. We will justify this further below, but first let us discuss what it means. @@ -143,4 +143,4 @@ Which for that case is choosing any sub-differential. We then generalize from that chose into the sub/super-differential convention. ## How does this generalize to n-D -Carefully, but consistently. \ No newline at end of file +Carefully, but consistently. From 3a2352724dc3456faa8e2cc3a1ad82f3365d1a8a Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 27 Jul 2021 23:30:08 +0100 Subject: [PATCH 03/17] add tldr --- docs/src/nondiff_points.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index c4bfda88b..459aba1d0 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -1,5 +1,8 @@ # What to return for non-differentiable points - +!!! info "What is the short version?" + If the function is not differentiable due to e.g. a branch, like `abs`, your rule can reasonably claim the derivative at that point is the value from either branch, *or* any value in-between (e.g. for `abs` claiming 0 is a good idea). + If it is not differentiable due to the primal not being defined on one side, you can set it to what ever you like. + Your rule should claim a derivative that is *useful*. In calculus one learns that if the derivative as computed by approaching from the left, and the derivative one computes as approaching from the right are not equal then the derivative is not defined, and we say the function is not differentiable at that point. From c190bdf75fb15e0730521d6517e09309e8c21a0b Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Wed, 28 Jul 2021 11:10:29 +0100 Subject: [PATCH 04/17] Update docs/src/nondiff_points.md Co-authored-by: Mason Protter --- docs/src/nondiff_points.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index 459aba1d0..63a2b02ee 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -134,7 +134,11 @@ In particular we often want to chose value of the derivative from **the other si ## Why is the sub/super-differential convention permissible? Math isn't real, and AD doesn't do calculus. -We are trying to accomplish some goal, and this is approach works well. +We are trying to accomplish some goal, and this approach works well. + +One way to think about this convention is to take an infinitely sharp discontinuity in a function and replace that discontinuty with a very (infinitesimally) small, smooth corner. +Due to the intermediate value theorem, we can then say that over this tiny interval, any derivative between the two extremes is realized and we are free to pick any one of them that we find useful as the 'canonical' value. + More specifically, consider our initial examples: `abs_left(x) = (x <= 0) ? -x : x`, and `abs_right(x) = (x < 0) ? -x : x`. These are a a primal level indistinguishable to the user. From 369b2737be63cb3781b7c5e1cd09f3432b6d5301 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Wed, 28 Jul 2021 11:11:52 +0100 Subject: [PATCH 05/17] Apply suggestions from code review Co-authored-by: Miha Zgubic --- docs/src/nondiff_points.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index 63a2b02ee..0bda9674c 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -33,8 +33,8 @@ We will justify this further below, but first let us discuss what it means. ## What is the sub/super-differential? -The subderivative is defined only for locally convex functions, where-as the super-deriviative is fined only for locally concave functions. -For our purpose we, basically never care which we are working with and so write sub/super-derivative. +The subderivative is defined only for locally convex functions, whereas the super-derivative is defined only for locally concave functions. +For our purpose we basically never care which we are working with and so write sub/super-derivative. For a function $f$ at some point $x_0$ a sub/super-derivative is a real number $c$ such that there exists a open ball $\mathcal{B} \subset \mathrm{dom}(f)$ containing $x_0$, and for all points $z \in \mathcal{B}$ the following holds: @@ -126,7 +126,7 @@ It will agree with either the forwards or reverse finite differencing results. Practically speaking, people are generally using AD to perform some gradient descent like optimization procedure. `NaN` and `Inf` do not generally take us to nice places. Erroring can be worse, especially if it is a local minima -- the optimization fully converges to that minima and then throws an error rather than reporting the result. -If the primal function takes a non-finite value or errors on one side, then we are in the case that we are at a boundry of the domain. +If the primal function takes a non-finite value or errors on one side, then we are in the case that we are at a boundary of the domain. Which means we are free to chose *any* value. In particular we often want to chose value of the derivative from **the other side where the primal is defined.** From cd400b6b845b5516880a0375be28b48b2a4b182a Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Wed, 28 Jul 2021 11:35:26 +0100 Subject: [PATCH 06/17] add gif --- docs/src/nondiff_points.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index 0bda9674c..f50a98998 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -139,6 +139,8 @@ We are trying to accomplish some goal, and this approach works well. One way to think about this convention is to take an infinitely sharp discontinuity in a function and replace that discontinuty with a very (infinitesimally) small, smooth corner. Due to the intermediate value theorem, we can then say that over this tiny interval, any derivative between the two extremes is realized and we are free to pick any one of them that we find useful as the 'canonical' value. +![animation showing `abs(x)` transforming into `x^2`](https://user-images.githubusercontent.com/29157027/127267988-92ca0899-ca33-4dd6-af45-e5a085283c83.gif) + More specifically, consider our initial examples: `abs_left(x) = (x <= 0) ? -x : x`, and `abs_right(x) = (x < 0) ? -x : x`. These are a a primal level indistinguishable to the user. From 78e4582fb9be978553e5b5ecba0677a8771e4a8d Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Fri, 20 Aug 2021 19:54:19 +0100 Subject: [PATCH 07/17] Add case study plots --- docs/Manifest.toml | 629 ++++++++++++++++++++++++++++++++++++- docs/Project.toml | 2 + docs/src/nondiff_points.md | 64 +++- 3 files changed, 693 insertions(+), 2 deletions(-) diff --git a/docs/Manifest.toml b/docs/Manifest.toml index 510a92fac..3a83ff319 100644 --- a/docs/Manifest.toml +++ b/docs/Manifest.toml @@ -1,5 +1,11 @@ # This file is machine-generated - editing it directly is not advised +[[Adapt]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "84918055d15b3114ede17ac6a7182f68870c16f7" +uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" +version = "3.3.1" + [[ArgTools]] uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" @@ -9,11 +15,41 @@ uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" [[Base64]] uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" +[[Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "c3598e525718abcc440f69cc6d5f60dda0a1b61e" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.6+5" + +[[Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "e2f47f6d8337369411569fd45ae5753ca10394c6" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.16.0+6" + [[ChainRulesCore]] deps = ["Compat", "LinearAlgebra", "SparseArrays"] path = ".." uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" -version = "1.11.1" +version = "1.0.1" + +[[ColorSchemes]] +deps = ["ColorTypes", "Colors", "FixedPointNumbers", "Random"] +git-tree-sha1 = "9995eb3977fbf67b86d0a0a0508e83017ded03f2" +uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4" +version = "3.14.0" + +[[ColorTypes]] +deps = ["FixedPointNumbers", "Random"] +git-tree-sha1 = "024fe24d83e4a5bf5fc80501a314ce0d1aa35597" +uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f" +version = "0.11.0" + +[[Colors]] +deps = ["ColorTypes", "FixedPointNumbers", "Reexport"] +git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40" +uuid = "5ae59095-9a9b-59fe-a467-6f913c188581" +version = "0.12.8" [[Compat]] deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"] @@ -21,6 +57,32 @@ git-tree-sha1 = "dce3e3fea680869eaa0b774b2e8343e9ff442313" uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" version = "3.40.0" +[[CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" + +[[Contour]] +deps = ["StaticArrays"] +git-tree-sha1 = "9f02045d934dc030edad45944ea80dbd1f0ebea7" +uuid = "d38c429a-6771-53c6-b99e-75d170b6e991" +version = "0.5.7" + +[[DataAPI]] +git-tree-sha1 = "ee400abb2298bd13bfc3df1c412ed228061a2385" +uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" +version = "1.7.0" + +[[DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "7d9d316f04214f7efdbb6398d545446e246eff02" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.10" + +[[DataValueInterfaces]] +git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6" +uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464" +version = "1.0.0" + [[Dates]] deps = ["Printf"] uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" @@ -55,16 +117,133 @@ version = "0.25.5" deps = ["ArgTools", "LibCURL", "NetworkOptions"] uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +[[EarCut_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "92d8f9f208637e8d2d28c664051a00569c01493d" +uuid = "5ae413db-bbd1-5e63-b57d-d24a61df00f5" +version = "2.1.5+1" + +[[Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "b3bfd02e98aedfa5cf885665493c5598c350cd2f" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.2.10+0" + +[[FFMPEG]] +deps = ["FFMPEG_jll"] +git-tree-sha1 = "b57e3acbe22f8484b4b5ff66a7499717fe1a9cc8" +uuid = "c87230d0-a227-11e9-1b43-d7ebe4e7570a" +version = "0.4.1" + +[[FFMPEG_jll]] +deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "LibVPX_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "Pkg", "Zlib_jll", "libass_jll", "libfdk_aac_jll", "libvorbis_jll", "x264_jll", "x265_jll"] +git-tree-sha1 = "3cc57ad0a213808473eafef4845a74766242e05f" +uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5" +version = "4.3.1+4" + +[[FixedPointNumbers]] +deps = ["Statistics"] +git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc" +uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93" +version = "0.8.4" + +[[Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Pkg", "Zlib_jll"] +git-tree-sha1 = "35895cf184ceaab11fd778b4590144034a167a2f" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.13.1+14" + +[[Formatting]] +deps = ["Printf"] +git-tree-sha1 = "8339d61043228fdd3eb658d86c926cb282ae72a8" +uuid = "59287772-0a20-5a39-b81b-1366585eb4c0" +version = "0.4.2" + +[[FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] +git-tree-sha1 = "cbd58c9deb1d304f5a245a0b7eb841a2560cfec6" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.10.1+5" + +[[FriBidi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "aa31987c2ba8704e23c6c8ba8a4f769d5d7e4f91" +uuid = "559328eb-81f9-559d-9380-de523a88c83c" +version = "1.0.10+0" + +[[GLFW_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libglvnd_jll", "Pkg", "Xorg_libXcursor_jll", "Xorg_libXi_jll", "Xorg_libXinerama_jll", "Xorg_libXrandr_jll"] +git-tree-sha1 = "dba1e8614e98949abfa60480b13653813d8f0157" +uuid = "0656b61e-2033-5cc2-a64a-77c0f6c09b89" +version = "3.3.5+0" + +[[GR]] +deps = ["Base64", "DelimitedFiles", "GR_jll", "HTTP", "JSON", "Libdl", "LinearAlgebra", "Pkg", "Printf", "Random", "Serialization", "Sockets", "Test", "UUIDs"] +git-tree-sha1 = "182da592436e287758ded5be6e32c406de3a2e47" +uuid = "28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71" +version = "0.58.1" + +[[GR_jll]] +deps = ["Artifacts", "Bzip2_jll", "Cairo_jll", "FFMPEG_jll", "Fontconfig_jll", "GLFW_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pixman_jll", "Pkg", "Qt5Base_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "d59e8320c2747553788e4fc42231489cc602fa50" +uuid = "d2c73de3-f751-5644-a686-071e5b155ba9" +version = "0.58.1+0" + +[[GeometryBasics]] +deps = ["EarCut_jll", "IterTools", "LinearAlgebra", "StaticArrays", "StructArrays", "Tables"] +git-tree-sha1 = "58bcdf5ebc057b085e58d95c138725628dd7453c" +uuid = "5c1252a2-5f33-56bf-86c9-59e7332b4326" +version = "0.4.1" + +[[Gettext_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046" +uuid = "78b55507-aeef-58d4-861c-77aaff3498b1" +version = "0.21.0+0" + +[[Glib_jll]] +deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE_jll", "Pkg", "Zlib_jll"] +git-tree-sha1 = "7bf67e9a481712b3dbe9cb3dac852dc4b1162e02" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.68.3+0" + +[[Grisu]] +git-tree-sha1 = "53bb909d1151e57e2484c3d1b53e19552b887fb2" +uuid = "42e2da0e-8278-4e71-bc24-59509adca0fe" +version = "1.0.2" + +[[HTTP]] +deps = ["Base64", "Dates", "IniFile", "Logging", "MbedTLS", "NetworkOptions", "Sockets", "URIs"] +git-tree-sha1 = "44e3b40da000eab4ccb1aecdc4801c040026aeb5" +uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3" +version = "0.9.13" + [[IOCapture]] deps = ["Logging"] git-tree-sha1 = "377252859f740c217b936cebcd918a44f9b53b59" uuid = "b5f81e59-6552-4d32-b1f0-c071b021bf89" version = "0.1.1" +[[IniFile]] +deps = ["Test"] +git-tree-sha1 = "098e4d2c533924c921f9f9847274f2ad89e018b8" +uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f" +version = "0.5.0" + [[InteractiveUtils]] deps = ["Markdown"] uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" +[[IterTools]] +git-tree-sha1 = "05110a2ab1fc5f932622ffea2a003221f4782c18" +uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e" +version = "1.3.0" + +[[IteratorInterfaceExtensions]] +git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856" +uuid = "82899510-4779-5014-852e-03e436cf321d" +version = "1.0.0" + [[JLLWrappers]] deps = ["Preferences"] git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e" @@ -77,6 +256,35 @@ git-tree-sha1 = "8076680b162ada2a031f707ac7b4953e30667a37" uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6" version = "0.21.2" +[[JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "d735490ac75c5cb9f1b00d8b5509c11984dc6943" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "2.1.0+0" + +[[LAME_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "f6250b16881adf048549549fba48b1161acdac8c" +uuid = "c1c5ebd0-6772-5130-a774-d5fcae4a789d" +version = "3.100.1+0" + +[[LZO_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "e5b909bcf985c5e2605737d2ce278ed791b89be6" +uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" +version = "2.10.1+0" + +[[LaTeXStrings]] +git-tree-sha1 = "c7f1c695e06c01b95a67f0cd1d34994f3e7db104" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.2.1" + +[[Latexify]] +deps = ["Formatting", "InteractiveUtils", "LaTeXStrings", "MacroTools", "Markdown", "Printf", "Requires"] +git-tree-sha1 = "a4b12a1bd2ebade87891ab7e36fdbce582301a92" +uuid = "23fbe1c1-3f47-55db-b15f-69d7ec21a316" +version = "0.15.6" + [[LibCURL]] deps = ["LibCURL_jll", "MozillaCACerts_jll"] uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" @@ -93,9 +301,63 @@ uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" deps = ["Artifacts", "Libdl", "MbedTLS_jll"] uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +[[LibVPX_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "12ee7e23fa4d18361e7c2cde8f8337d4c3101bc7" +uuid = "dd192d2f-8180-539f-9fb4-cc70b1dcf69a" +version = "1.10.0+0" + [[Libdl]] uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" +[[Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "761a393aeccd6aa92ec3515e428c26bf99575b3b" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.2.2+0" + +[[Libgcrypt_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll", "Pkg"] +git-tree-sha1 = "64613c82a59c120435c067c2b809fc61cf5166ae" +uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4" +version = "1.8.7+0" + +[[Libglvnd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll", "Xorg_libXext_jll"] +git-tree-sha1 = "7739f837d6447403596a75d19ed01fd08d6f56bf" +uuid = "7e76a0d4-f3c7-5321-8279-8d96eeed0f29" +version = "1.3.0+3" + +[[Libgpg_error_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "c333716e46366857753e273ce6a69ee0945a6db9" +uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8" +version = "1.42.0+0" + +[[Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "42b62845d70a619f063a7da093d995ec8e15e778" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.16.1+1" + +[[Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "9c30530bf0effd46e15e0fdcf2b8636e78cbbd73" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.35.0+0" + +[[Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "340e257aada13f95f98ee352d316c3bed37c8ab9" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.3.0+0" + +[[Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "7f3efec06033682db852f8b3bc3c1d2b0a0ab066" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.36.0+0" + [[LinearAlgebra]] deps = ["Libdl"] uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" @@ -103,33 +365,114 @@ uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" [[Logging]] uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" +[[MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "0fb723cd8c45858c22169b2e42269e53271a6df7" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.7" + [[Markdown]] deps = ["Base64"] uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" +[[MbedTLS]] +deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"] +git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe" +uuid = "739be429-bea8-5141-9913-cc70e7f3736d" +version = "1.0.3" + [[MbedTLS_jll]] deps = ["Artifacts", "Libdl"] uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +[[Measures]] +git-tree-sha1 = "e498ddeee6f9fdb4551ce855a46f54dbd900245f" +uuid = "442fdcdd-2543-5da2-b0f3-8c86c306513e" +version = "0.3.1" + +[[Missings]] +deps = ["DataAPI"] +git-tree-sha1 = "4ea90bd5d3985ae1f9a908bd4500ae88921c5ce7" +uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" +version = "1.0.0" + [[Mmap]] uuid = "a63ad114-7e13-5084-954f-fe012c677804" [[MozillaCACerts_jll]] uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +[[NaNMath]] +git-tree-sha1 = "bfe47e760d60b82b66b61d2d44128b62e3a369fb" +uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3" +version = "0.3.5" + [[NetworkOptions]] uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +[[Ogg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "7937eda4681660b4d6aeeecc2f7e1c81c8ee4e2f" +uuid = "e7412a2a-1a6e-54c0-be00-318e2571c051" +version = "1.3.5+0" + +[[OpenSSL_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "15003dcb7d8db3c6c857fda14891a539a8f2705a" +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "1.1.10+0" + +[[Opus_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "51a08fb14ec28da2ec7a927c4337e4332c2a4720" +uuid = "91d4177d-7536-5919-b921-800302f37372" +version = "1.3.2+0" + +[[OrderedCollections]] +git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.4.1" + +[[PCRE_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "b2a7af664e098055a7529ad1a900ded962bca488" +uuid = "2f80f16e-611a-54ab-bc61-aa92de5b98fc" +version = "8.44.0+0" + [[Parsers]] deps = ["Dates"] git-tree-sha1 = "ae4bbcadb2906ccc085cf52ac286dc1377dceccc" uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0" version = "2.1.2" +[[Pixman_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "b4f5d02549a10e20780a24fce72bea96b6329e29" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.40.1+0" + [[Pkg]] deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +[[PlotThemes]] +deps = ["PlotUtils", "Requires", "Statistics"] +git-tree-sha1 = "a3a964ce9dc7898193536002a6dd892b1b5a6f1d" +uuid = "ccf2f8ad-2431-5c83-bf29-c5338b663b6a" +version = "2.0.1" + +[[PlotUtils]] +deps = ["ColorSchemes", "Colors", "Dates", "Printf", "Random", "Reexport", "Statistics"] +git-tree-sha1 = "501c20a63a34ac1d015d5304da0e645f42d91c9f" +uuid = "995b91a9-d308-5afd-9ec6-746e21dbc043" +version = "1.0.11" + +[[Plots]] +deps = ["Base64", "Contour", "Dates", "FFMPEG", "FixedPointNumbers", "GR", "GeometryBasics", "JSON", "Latexify", "LinearAlgebra", "Measures", "NaNMath", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "RecipesPipeline", "Reexport", "Requires", "Scratch", "Showoff", "SparseArrays", "Statistics", "StatsBase", "UUIDs"] +git-tree-sha1 = "8365fa7758e2e8e4443ce866d6106d8ecbb4474e" +uuid = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" +version = "1.20.1" + [[Preferences]] deps = ["TOML"] git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a" @@ -140,6 +483,12 @@ version = "1.2.2" deps = ["Unicode"] uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" +[[Qt5Base_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "Libdl", "Libglvnd_jll", "OpenSSL_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libxcb_jll", "Xorg_xcb_util_image_jll", "Xorg_xcb_util_keysyms_jll", "Xorg_xcb_util_renderutil_jll", "Xorg_xcb_util_wm_jll", "Zlib_jll", "xkbcommon_jll"] +git-tree-sha1 = "ad368663a5e20dbb8d6dc2fddeefe4dae0781ae8" +uuid = "ea2cea3b-5b76-57ae-a6ef-0a8af62496e1" +version = "5.15.3+0" + [[REPL]] deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" @@ -148,6 +497,28 @@ uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" deps = ["Serialization"] uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" +[[RecipesBase]] +git-tree-sha1 = "b3fb709f3c97bfc6e948be68beeecb55a0b340ae" +uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" +version = "1.1.1" + +[[RecipesPipeline]] +deps = ["Dates", "NaNMath", "PlotUtils", "RecipesBase"] +git-tree-sha1 = "2a7a2469ed5d94a98dea0e85c46fa653d76be0cd" +uuid = "01d81517-befc-4cb6-b9ec-a95719d0359c" +version = "0.3.4" + +[[Reexport]] +git-tree-sha1 = "5f6c21241f0f655da3952fd60aa18477cf96c220" +uuid = "189a3867-3050-52da-a836-e630ba90ab69" +version = "1.1.0" + +[[Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.1.3" + [[SHA]] uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" @@ -157,6 +528,12 @@ git-tree-sha1 = "aa841c3738cec78b5dbccd56dda332710f35f6a5" uuid = "322a6be2-4ae8-5d68-aaf1-3e960788d1d9" version = "0.2.0" +[[Scratch]] +deps = ["Dates"] +git-tree-sha1 = "0b4b7f1393cff97c33891da2a0bf69c6ed241fda" +uuid = "6c6a2e73-6563-6170-7368-637461726353" +version = "1.1.0" + [[Serialization]] uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" @@ -164,21 +541,68 @@ uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" deps = ["Distributed", "Mmap", "Random", "Serialization"] uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" +[[Showoff]] +deps = ["Dates", "Grisu"] +git-tree-sha1 = "91eddf657aca81df9ae6ceb20b959ae5653ad1de" +uuid = "992d4aef-0814-514b-bc4d-f2e9a6c4116f" +version = "1.0.3" + [[Sockets]] uuid = "6462fe0b-24de-5631-8697-dd941f90decc" +[[SortingAlgorithms]] +deps = ["DataStructures"] +git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508" +uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" +version = "1.0.1" + [[SparseArrays]] deps = ["LinearAlgebra", "Random"] uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +[[StaticArrays]] +deps = ["LinearAlgebra", "Random", "Statistics"] +git-tree-sha1 = "3240808c6d463ac46f1c1cd7638375cd22abbccb" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.2.12" + [[Statistics]] deps = ["LinearAlgebra", "SparseArrays"] uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +[[StatsAPI]] +git-tree-sha1 = "1958272568dc176a1d881acb797beb909c785510" +uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" +version = "1.0.0" + +[[StatsBase]] +deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] +git-tree-sha1 = "fed1ec1e65749c4d96fc20dd13bea72b55457e62" +uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" +version = "0.33.9" + +[[StructArrays]] +deps = ["Adapt", "DataAPI", "StaticArrays", "Tables"] +git-tree-sha1 = "000e168f5cc9aded17b6999a560b7c11dda69095" +uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a" +version = "0.6.0" + [[TOML]] deps = ["Dates"] uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +[[TableTraits]] +deps = ["IteratorInterfaceExtensions"] +git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39" +uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c" +version = "1.0.1" + +[[Tables]] +deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"] +git-tree-sha1 = "d0c690d37c73aeb5ca063056283fde5585a41710" +uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" +version = "1.5.0" + [[Tar]] deps = ["ArgTools", "SHA"] uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" @@ -187,6 +611,11 @@ uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" +[[URIs]] +git-tree-sha1 = "97bbe755a53fe859669cd907f2d96aee8d2c1355" +uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4" +version = "1.3.0" + [[UUIDs]] deps = ["Random", "SHA"] uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" @@ -194,16 +623,196 @@ uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" [[Unicode]] uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" +[[Wayland_jll]] +deps = ["Artifacts", "Expat_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg", "XML2_jll"] +git-tree-sha1 = "3e61f0b86f90dacb0bc0e73a0c5a83f6a8636e23" +uuid = "a2964d1f-97da-50d4-b82a-358c7fce9d89" +version = "1.19.0+0" + +[[Wayland_protocols_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Wayland_jll"] +git-tree-sha1 = "2839f1c1296940218e35df0bbb220f2a79686670" +uuid = "2381bf8a-dfd0-557d-9999-79630e7b1b91" +version = "1.18.0+4" + +[[XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "Zlib_jll"] +git-tree-sha1 = "1acf5bdf07aa0907e0a37d3718bb88d4b687b74a" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.9.12+0" + +[[XSLT_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"] +git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a" +uuid = "aed1982a-8fda-507f-9586-7b0439959a61" +version = "1.1.34+0" + +[[Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "5be649d550f3f4b95308bf0183b82e2582876527" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.6.9+4" + +[[Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "4e490d5c960c314f33885790ed410ff3a94ce67e" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.9+4" + +[[Xorg_libXcursor_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXfixes_jll", "Xorg_libXrender_jll"] +git-tree-sha1 = "12e0eb3bc634fa2080c1c37fccf56f7c22989afd" +uuid = "935fb764-8cf2-53bf-bb30-45bb1f8bf724" +version = "1.2.0+4" + +[[Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "4fe47bd2247248125c428978740e18a681372dd4" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.3+4" + +[[Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"] +git-tree-sha1 = "b7c0aa8c376b31e4852b360222848637f481f8c3" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.4+4" + +[[Xorg_libXfixes_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"] +git-tree-sha1 = "0e0dc7431e7a0587559f9294aeec269471c991a4" +uuid = "d091e8ba-531a-589c-9de9-94069b037ed8" +version = "5.0.3+4" + +[[Xorg_libXi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll", "Xorg_libXfixes_jll"] +git-tree-sha1 = "89b52bc2160aadc84d707093930ef0bffa641246" +uuid = "a51aa0fd-4e3c-5386-b890-e753decda492" +version = "1.7.10+4" + +[[Xorg_libXinerama_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll"] +git-tree-sha1 = "26be8b1c342929259317d8b9f7b53bf2bb73b123" +uuid = "d1454406-59df-5ea1-beac-c340f2130bc3" +version = "1.1.4+4" + +[[Xorg_libXrandr_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll"] +git-tree-sha1 = "34cea83cb726fb58f325887bf0612c6b3fb17631" +uuid = "ec84b674-ba8e-5d96-8ba1-2a689ba10484" +version = "1.5.2+4" + +[[Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"] +git-tree-sha1 = "19560f30fd49f4d4efbe7002a1037f8c43d43b96" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.10+4" + +[[Xorg_libpthread_stubs_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "6783737e45d3c59a4a4c4091f5f88cdcf0908cbb" +uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74" +version = "0.1.0+3" + +[[Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"] +git-tree-sha1 = "daf17f441228e7a3833846cd048892861cff16d6" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.13.0+3" + +[[Xorg_libxkbfile_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"] +git-tree-sha1 = "926af861744212db0eb001d9e40b5d16292080b2" +uuid = "cc61e674-0454-545c-8b26-ed2c68acab7a" +version = "1.1.0+4" + +[[Xorg_xcb_util_image_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"] +git-tree-sha1 = "0fab0a40349ba1cba2c1da699243396ff8e94b97" +uuid = "12413925-8142-5f55-bb0e-6d7ca50bb09b" +version = "0.4.0+1" + +[[Xorg_xcb_util_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll"] +git-tree-sha1 = "e7fd7b2881fa2eaa72717420894d3938177862d1" +uuid = "2def613f-5ad1-5310-b15b-b15d46f528f5" +version = "0.4.0+1" + +[[Xorg_xcb_util_keysyms_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"] +git-tree-sha1 = "d1151e2c45a544f32441a567d1690e701ec89b00" +uuid = "975044d2-76e6-5fbe-bf08-97ce7c6574c7" +version = "0.4.0+1" + +[[Xorg_xcb_util_renderutil_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"] +git-tree-sha1 = "dfd7a8f38d4613b6a575253b3174dd991ca6183e" +uuid = "0d47668e-0667-5a69-a72c-f761630bfb7e" +version = "0.3.9+1" + +[[Xorg_xcb_util_wm_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"] +git-tree-sha1 = "e78d10aab01a4a154142c5006ed44fd9e8e31b67" +uuid = "c22f9ab0-d5fe-5066-847c-f4bb1cd4e361" +version = "0.4.1+1" + +[[Xorg_xkbcomp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxkbfile_jll"] +git-tree-sha1 = "4bcbf660f6c2e714f87e960a171b119d06ee163b" +uuid = "35661453-b289-5fab-8a00-3d9160c6a3a4" +version = "1.4.2+4" + +[[Xorg_xkeyboard_config_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xkbcomp_jll"] +git-tree-sha1 = "5c8424f8a67c3f2209646d4425f3d415fee5931d" +uuid = "33bec58e-1273-512f-9401-5d533626f822" +version = "2.27.0+4" + +[[Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "79c31e7844f6ecf779705fbc12146eb190b7d845" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.4.0+3" + [[Zlib_jll]] deps = ["Libdl"] uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +[[Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "cc4bf3fdde8b7e3e9fa0351bdeedba1cf3b7f6e6" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.0+0" + +[[libass_jll]] +deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] +git-tree-sha1 = "acc685bcf777b2202a904cdcb49ad34c2fa1880c" +uuid = "0ac62f75-1d6f-5e53-bd7c-93b484bb37c0" +version = "0.14.0+4" + +[[libfdk_aac_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "7a5780a0d9c6864184b3a2eeeb833a0c871f00ab" +uuid = "f638f0a6-7fb0-5443-88ba-1cc74229b280" +version = "0.1.6+4" + +[[libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] +git-tree-sha1 = "94d180a6d2b5e55e447e2d27a29ed04fe79eb30c" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.38+0" + [[libsass_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "663428b7ebaf60c60ee147f0f9466430e9959ad6" uuid = "47bcb7c8-5119-555a-9eeb-0afcc36cd728" version = "3.5.5+0" +[[libvorbis_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Ogg_jll", "Pkg"] +git-tree-sha1 = "c45f4e40e7aafe9d086379e5578947ec8b95a8fb" +uuid = "f27f6e37-5d2b-51aa-960f-b287f2bc3b7a" +version = "1.3.7+0" + [[nghttp2_jll]] deps = ["Artifacts", "Libdl"] uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" @@ -211,3 +820,21 @@ uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" [[p7zip_jll]] deps = ["Artifacts", "Libdl"] uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" + +[[x264_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "d713c1ce4deac133e3334ee12f4adff07f81778f" +uuid = "1270edf5-f2f9-52d2-97e9-ab00b5d0237a" +version = "2020.7.14+2" + +[[x265_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "487da2f8f2f0c8ee0e83f39d13037d6bbf0a45ab" +uuid = "dfaa095f-4041-5dcd-9319-2fabd8486b76" +version = "3.0.0+3" + +[[xkbcommon_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Wayland_jll", "Wayland_protocols_jll", "Xorg_libxcb_jll", "Xorg_xkeyboard_config_jll"] +git-tree-sha1 = "ece2350174195bb31de1a63bea3a41ae1aa593b6" +uuid = "d8fb68d0-12a3-5cfd-a85a-d49703b185fd" +version = "0.9.1+5" diff --git a/docs/Project.toml b/docs/Project.toml index a9c8bac2b..c271f32e1 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -4,7 +4,9 @@ Compat = "34da2185-b29b-5c13-b0c7-acf172513d20" DocThemeIndigo = "8bac0ac5-51bf-41f9-885e-2bf1ac2bec5f" Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a" +Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" [compat] DocThemeIndigo = "0.1.3" Documenter = "0.25.2" +Plots = "1.20" diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index f50a98998..fa0a01203 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -19,6 +19,68 @@ The obvious answer, would be to write a rule that throws an error if input at a Another option is to return some error signally value like `NaN`. Which you *can* do. However, this is not useful. + +Let us explore what is useful: +# Case Studies + +```@setup nondiff +using Plots +gr(framestyle=:origin, legend=false) +``` +### Derivative is defined in usual sense +```@example nondiff +plot(x->x^3) +``` +This is the standard case, one can returned the derivative that is defined according to school room calculus. +An interesting thing a bout `x->x^3` is that at `x=0` the derivative is defined, +but neither the sub-derivative nor super-derivative is defined. + +### Local Minima / Maxima + +```@example nondiff +plot(abs) +``` + +### Piecewise slope change +```@example nondiff +plot(x-> x < 0 ? x : 5x) +``` + +### Zero almost everywhere + +```@example nondiff +plot(round) +``` + +### Non-finite and same on both sides +```@example nondiff +plot(x->inv(x^2)) +plot!(; xlims=(-1,1), ylims=(-100,100)) #hide +``` + +### Non-finite and differing on both sides +```@example nondiff +plot(inv) +plot!(; xlims=(-1,1), ylims=(-100,100)) #hide +``` + +### Not defined on one-side +```@example nondiff +plot(x->exp(2log(x))) +``` + +We do not have to worry about what to return for the side where it is not defined. +As we will never be asked for the derivative at e.g. `x=-2.5` since the primal function errors. +But we do need to worry about at the boundary -- if that boundary point doesn't error. + +### Not defined on one side, non-finite on the other +```@example nondiff +plot(log) +``` + +### sub/super-differential convention +**TODO: Incorperate this with rest of the document. + Instead we introduce what we call the **sub/super-differential convention**: > It is always permissible to return any element of the sub/super-differential. @@ -31,7 +93,7 @@ Or equivalently but considering the trivial singleton sub/super-differential sep We will justify this further below, but first let us discuss what it means. -## What is the sub/super-differential? +### What is the sub/super-differential? The subderivative is defined only for locally convex functions, whereas the super-derivative is defined only for locally concave functions. For our purpose we basically never care which we are working with and so write sub/super-derivative. From af3b1393b71de2c6b7c2143942bdf54b2f47aff5 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 24 Aug 2021 19:20:42 +0100 Subject: [PATCH 08/17] add and fill in some more examples --- docs/src/nondiff_points.md | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index fa0a01203..e0a5fc135 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -32,8 +32,10 @@ gr(framestyle=:origin, legend=false) plot(x->x^3) ``` This is the standard case, one can returned the derivative that is defined according to school room calculus. -An interesting thing a bout `x->x^3` is that at `x=0` the derivative is defined, -but neither the sub-derivative nor super-derivative is defined. +Here we would reasonably say that at `x=0` the derivative is `3*0^2=0`. + + +An interesting thing about `x->x^3` is that at `x=0` the derivative is defined, but neither the sub-derivative nor super-derivative is defined. ### Local Minima / Maxima @@ -41,24 +43,48 @@ but neither the sub-derivative nor super-derivative is defined. plot(abs) ``` +`abs` is the classic example of a function where the derivative is not defines as the limit from above is not equal to the limit from below + +$$\mathrm{abs}'(0) = \lim_{h->0^-} \dfrac{\mathrm{abs}(0)-\mathrm{abs}(0-h)}{0-h} = -1$$ +$$\mathrm{abs}'(0) = \lim_{h->0^+} \dfrac{abs(0)-\mathrm{abs}(0-h)}{0-h} = 1$$ + +Now, as discussed in the introduction the AD system would on it's own choose either 1 or -1, depending on implementation. + +We however have a potentially much nicer answer available to use: 0. + +This has a number of advantages. +- It follows the rule that derivatives are zero at local minima (and maxima). +- If you leave a gradient decent optimizer running it will eventually actually converge absolutely to the point -- where as with it being 1 or -1 it would never outright converge it would always flee. + +Further: +- It is a perfectly nice member of the [subderivative](https://en.wikipedia.org/wiki/Subderivative). +- It is the mean of the derivative on each side; which means that it will agree with central finite differencing at the point. ### Piecewise slope change ```@example nondiff plot(x-> x < 0 ? x : 5x) ``` -### Zero almost everywhere +### Derivative zero almost everywhere + +```@example nondiff +plot(ceil) +``` + +### Primal finite, and derivative nonfinite and same on both sides ```@example nondiff -plot(round) +plot(cbrt) ``` -### Non-finite and same on both sides + +(derivative nonfinite and different on each side is not possible with a finite and defined primal.) +### Primal and derivative Non-finite and same on both sides ```@example nondiff plot(x->inv(x^2)) plot!(; xlims=(-1,1), ylims=(-100,100)) #hide ``` -### Non-finite and differing on both sides +### Primal and gradient Non-finite and differing on both sides ```@example nondiff plot(inv) plot!(; xlims=(-1,1), ylims=(-100,100)) #hide From 7807bfbe2255bea216fee3aafd5ea16964f1e5b6 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Mon, 6 Sep 2021 15:01:32 +0100 Subject: [PATCH 09/17] wip --- docs/src/nondiff_points.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index e0a5fc135..1b2938838 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -64,6 +64,16 @@ Further: plot(x-> x < 0 ? x : 5x) ``` +Here was have 3 main options, all are good. +We could say there derivative at 0 is + - 1: which agrees with backwards finite differencing + - 5: which agrees with fowards finite differencing + - 3: which is the mean of `[1, 5]`, and agrees with central finite differencing + +All of these options are perfectly nice members of the [subderivative](https://en.wikipedia.org/wiki/Subderivative). +Saying it is `3` is the arguably the nicest, but it is also the most expensive to compute; and it will + + ### Derivative zero almost everywhere ```@example nondiff @@ -75,9 +85,7 @@ plot(ceil) ```@example nondiff plot(cbrt) ``` - - -(derivative nonfinite and different on each side is not possible with a finite and defined primal.) +s(derivative nonfinite and different on each side is not possible with a finite and defined primal.) ### Primal and derivative Non-finite and same on both sides ```@example nondiff plot(x->inv(x^2)) From 4a7f0b4e12983bed2d05856086cf759ac4bfcdf6 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Fri, 8 Oct 2021 17:44:06 +0100 Subject: [PATCH 10/17] wip --- docs/src/nondiff_points.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index 1b2938838..a4f364430 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -18,7 +18,7 @@ So what rule should we write for this case? The obvious answer, would be to write a rule that throws an error if input at a point where calculus says the derivative is not defined. Another option is to return some error signally value like `NaN`. Which you *can* do. -However, this is not useful. +However, there is no where to go with an error, the user still wants a derivative; so this is not useful. Let us explore what is useful: # Case Studies @@ -65,9 +65,10 @@ plot(x-> x < 0 ? x : 5x) ``` Here was have 3 main options, all are good. -We could say there derivative at 0 is + +We could say there derivative at 0 is: - 1: which agrees with backwards finite differencing - - 5: which agrees with fowards finite differencing + - 5: which agrees with forwards finite differencing - 3: which is the mean of `[1, 5]`, and agrees with central finite differencing All of these options are perfectly nice members of the [subderivative](https://en.wikipedia.org/wiki/Subderivative). @@ -80,6 +81,13 @@ Saying it is `3` is the arguably the nicest, but it is also the most expensive t plot(ceil) ``` +Here it seems most useful to say the derivative is zero everywhere. +The limits are zero from both sides. + +The other option for `ceil` would be to say it is 1 everywhere. +But that it too weird, if the use wanted a relaxation of the problem then they would provide one. +Imposing one on `ceil` for everyone is not reasonable. + ### Primal finite, and derivative nonfinite and same on both sides ```@example nondiff From 4cfd8a173117fd2522ee0643408134016330c1af Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Fri, 8 Oct 2021 17:54:39 +0100 Subject: [PATCH 11/17] wip --- docs/src/nondiff_points.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index a4f364430..a15de4378 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -115,13 +115,20 @@ We do not have to worry about what to return for the side where it is not define As we will never be asked for the derivative at e.g. `x=-2.5` since the primal function errors. But we do need to worry about at the boundary -- if that boundary point doesn't error. +Since we will never be asked about the left-hand side (as the primal errors), we can use just the right-hand side derivative. +In this case giving 0.0. +` +Also nice in this case is that it agrees with the symbolic simplification of `x->exp(2log(x))` into `x->x^2`. + + + ### Not defined on one side, non-finite on the other ```@example nondiff plot(log) ``` ### sub/super-differential convention -**TODO: Incorperate this with rest of the document. +**TODO: Incorperate this with rest of the document. Or move to design notes** Instead we introduce what we call the **sub/super-differential convention**: From 1099b2a22550e36e3d8fbc436ad9868c023438fd Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Fri, 5 Nov 2021 14:11:23 +0000 Subject: [PATCH 12/17] wip --- docs/src/nondiff_points.md | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index a15de4378..6aca4c6cb 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -81,31 +81,40 @@ Saying it is `3` is the arguably the nicest, but it is also the most expensive t plot(ceil) ``` -Here it seems most useful to say the derivative is zero everywhere. +Here it is most useful to say the derivative is zero everywhere. The limits are zero from both sides. -The other option for `ceil` would be to say it is 1 everywhere. +The other option for `x->ceil(x)` would be relax the problem into `x->x`, and thus say it is 1 everywhere But that it too weird, if the use wanted a relaxation of the problem then they would provide one. -Imposing one on `ceil` for everyone is not reasonable. +We can not be imposing that relaxation on to `ceil` for everyone is not reasonable. ### Primal finite, and derivative nonfinite and same on both sides ```@example nondiff plot(cbrt) ``` -s(derivative nonfinite and different on each side is not possible with a finite and defined primal.) -### Primal and derivative Non-finite and same on both sides + + + +### Primal and derivative Non-finite and different on both sides ```@example nondiff plot(x->inv(x^2)) plot!(; xlims=(-1,1), ylims=(-100,100)) #hide ``` -### Primal and gradient Non-finite and differing on both sides +In this case the primal isn't finite, so the value of the derivative can be assumed to matter less. +It is not surprising to see a nonfinite gradient for nonfinite primal. +So it is fine to have a the gradient being nonfinite. + +## Primal finite and derivative nonfinite and different on each side ```@example nondiff -plot(inv) -plot!(; xlims=(-1,1), ylims=(-100,100)) #hide +plot(x-> sign(x) * cbrt(x)) ``` +In this example, the primal is defined and finite, so we would like a derivative to defined. +We are back in the case of a local minimal like we were for `abs`. +We can make most of the same arguments as we made there to justify saying the derivative is zero. + ### Not defined on one-side ```@example nondiff plot(x->exp(2log(x))) @@ -127,6 +136,8 @@ Also nice in this case is that it agrees with the symbolic simplification of `x- plot(log) ``` +Here there is no harm in taking the value on the defined, finite + ### sub/super-differential convention **TODO: Incorperate this with rest of the document. Or move to design notes** From 19f3399cf6424155edb25684444e4de471a007b1 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 9 Nov 2021 18:01:10 +0000 Subject: [PATCH 13/17] Finish examples --- docs/src/nondiff_points.md | 66 ++++++++++++++++++++------------------ 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index 6aca4c6cb..c6613499c 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -21,7 +21,7 @@ Which you *can* do. However, there is no where to go with an error, the user still wants a derivative; so this is not useful. Let us explore what is useful: -# Case Studies +## Case Studies ```@setup nondiff using Plots @@ -72,8 +72,9 @@ We could say there derivative at 0 is: - 3: which is the mean of `[1, 5]`, and agrees with central finite differencing All of these options are perfectly nice members of the [subderivative](https://en.wikipedia.org/wiki/Subderivative). -Saying it is `3` is the arguably the nicest, but it is also the most expensive to compute; and it will - +`3` is the arguably the nicest, but it is also the most expensive to compute. +In general all are acceptable. + ### Derivative zero almost everywhere @@ -88,25 +89,38 @@ The other option for `x->ceil(x)` would be relax the problem into `x->x`, and th But that it too weird, if the use wanted a relaxation of the problem then they would provide one. We can not be imposing that relaxation on to `ceil` for everyone is not reasonable. -### Primal finite, and derivative nonfinite and same on both sides - +### Not defined on one-side ```@example nondiff -plot(cbrt) +plot(x->exp(2log(x))) +plot!(; xlims=(-10,10), ylims=(-10,10)) #hide ``` +We do not have to worry about what to return for the side where it is not defined. +As we will never be asked for the derivative at e.g. `x=-2.5` since the primal function errors. +But we do need to worry about at the boundary -- if that boundary point doesn't error. + +Since we will never be asked about the left-hand side (as the primal errors), we can use just the right-hand side derivative. +In this case giving 0.0. +` +Also nice in this case is that it agrees with the symbolic simplification of `x->exp(2log(x))` into `x->x^2`. + +### Derivative nonfinite and same on both sides -### Primal and derivative Non-finite and different on both sides ```@example nondiff -plot(x->inv(x^2)) -plot!(; xlims=(-1,1), ylims=(-100,100)) #hide +plot(cbrt) ``` -In this case the primal isn't finite, so the value of the derivative can be assumed to matter less. -It is not surprising to see a nonfinite gradient for nonfinite primal. -So it is fine to have a the gradient being nonfinite. +Here we have no real choice but to say the derivative at `0` is `Inf`. +We could consider as an alternative saying some large but finite value. +However, if too large it will just overflow rapidly anyway; and if too small it will not dominate over finite terms. +It is not possible to find a given value that is always large enough. +Our alternatives woud be to consider the dederivative at `nextfloat(0.0)` or `prevfloat(0.0)`. +But this is more or less the same as choosing some large value -- in this case an extremely large value that will rapidly overflow. + + +### Derivative on-finite and different on both sides -## Primal finite and derivative nonfinite and different on each side ```@example nondiff plot(x-> sign(x) * cbrt(x)) ``` @@ -115,28 +129,18 @@ In this example, the primal is defined and finite, so we would like a derivative We are back in the case of a local minimal like we were for `abs`. We can make most of the same arguments as we made there to justify saying the derivative is zero. -### Not defined on one-side -```@example nondiff -plot(x->exp(2log(x))) -``` - -We do not have to worry about what to return for the side where it is not defined. -As we will never be asked for the derivative at e.g. `x=-2.5` since the primal function errors. -But we do need to worry about at the boundary -- if that boundary point doesn't error. - -Since we will never be asked about the left-hand side (as the primal errors), we can use just the right-hand side derivative. -In this case giving 0.0. -` -Also nice in this case is that it agrees with the symbolic simplification of `x->exp(2log(x))` into `x->x^2`. +## Conclusion +From the case studies a few general rules can be seen for how to choose a value that is _useful_. +These rough rules are: + - Say the derivative is 0 at local optima + - If the derivative from one side is defined and the other isn't, say it is the derivative taken from defined side. + - If the derivative from one side is finite and the other isn't, say it is the derivative taken from finite side. + - When derivative from each side is not equal, strongly consider reporting the average +Our goal as always, is to get a pragmatically useful result for everyone, which must by necessity also avoid a pathological result for anyone. -### Not defined on one side, non-finite on the other -```@example nondiff -plot(log) -``` -Here there is no harm in taking the value on the defined, finite ### sub/super-differential convention **TODO: Incorperate this with rest of the document. Or move to design notes** From 35f8633378c992b38bca9e183d002b46d7a0b226 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 9 Nov 2021 18:01:47 +0000 Subject: [PATCH 14/17] Delete sub/super discussion --- docs/src/nondiff_points.md | 142 +------------------------------------ 1 file changed, 1 insertion(+), 141 deletions(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index c6613499c..acb9466e3 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -138,144 +138,4 @@ These rough rules are: - If the derivative from one side is finite and the other isn't, say it is the derivative taken from finite side. - When derivative from each side is not equal, strongly consider reporting the average -Our goal as always, is to get a pragmatically useful result for everyone, which must by necessity also avoid a pathological result for anyone. - - - -### sub/super-differential convention -**TODO: Incorperate this with rest of the document. Or move to design notes** - -Instead we introduce what we call the **sub/super-differential convention**: - -> It is always permissible to return any element of the sub/super-differential. -> You should return the most useful. - -Or equivalently but considering the trivial singleton sub/super-differential seperately: - -> At any point where the derivative is not defined, it is permissible to return any element of the sub/super-differential. -> You should return the most useful. - -We will justify this further below, but first let us discuss what it means. - -### What is the sub/super-differential? - -The subderivative is defined only for locally convex functions, whereas the super-derivative is defined only for locally concave functions. -For our purpose we basically never care which we are working with and so write sub/super-derivative. - -For a function $f$ at some point $x_0$ a sub/super-derivative is a real number $c$ such that there exists a open ball $\mathcal{B} \subset \mathrm{dom}(f)$ containing $x_0$, -and for all points $z \in \mathcal{B}$ the following holds: - -$$\mathrm{sub -derivative:}\qquad f(z) - f(x_0) \ge c\,(z-x_0)$$ - -$$\mathrm{super-derivative:}\qquad f(z) - f(x_0) \le c\,(z-x_0)$$ - -We call the the set of all values $c$ the sub/super-differential at $x_0$. - -More informally: consider a plot of the function. -The sub/super-differential at a point is the set of slopes of all lines you could draw touching that point, but with the lines either entirely above, or entirely below the curve. - -It is best illustrated with a figure: - -![plot showing subderiviatives](https://upload.wikimedia.org/wikipedia/commons/4/4e/Subderivative_illustration.png) - -In this figure a plot of a function is shown in blue. -Two subtangent lines are plotted in red. -Their slopes are sub/super-derivatives at $x_0$, and they are two elements of the subdifferential. -If you flip it upside down, it would apply for the super-differential, with the lines being above the curve. - -### Some key things to note. - -For a function where the derivative is defined on both sides of the point: - - the derivative on each side is a sub/super-derivative at the point - - as is the mean of the derivative of each side - - in-fact the mean of any subset (including the whole sub/super-differential) of sub/super-derivatives is also a sub/super-derivative. - - if the derivative one one side is negative and the other positive then zero is a sub/super-derivative. - -For a function that is only defined on one side of the point, the sub/super-differential is the full set of real numbers. -This by the subgradient convention leaves you free to chose *any* useful value. - -!!! info "Does AD always return a sub/super-derivative? No" - On consideration of this, one might be tempted to think that AD always returns a sub/super-derivative. - And that in normal cases it return the only sub/super-derivative i.e. the actual derivative; and in other case it picks one of the branches. - Thus all weirdness of AD disagreeing with calculus could be explained away in this way. - **This is not the case.** - As it is not necessarily true that all branches derivatives are correct sub/super-differentials for the function. - Consider the function from [Abadi and Plotkin, 2019](https://dl.acm.org/doi/10.1145/3371106): - `f(x::Float64) = x == 0.0 ? 0.0 : x`. - The correct derivative of this function is `1` at all points, but most ADs will say that at `x=0` the derivative is zero. - The fix for this, is to define a rule which *does* in fact return a sub/super-derivative. - So one could say a correctly functioning AD with all needed rules does always return a sub/super-differential. - - -## What is useful ? - -The sub/super-differential convention had two parts: -"It is always permissable to return any element of the sub/super-differential. -**You should return the most useful.**". -What is the most useful? -This is a value judgement you as a rule author will have to make. - - -### It is often zero - -If zero is a sub/super-derivative, then it is often the most useful one. -Especially if the point is a local minima/maxima -For a function like `abs` or `x -> x<0 ? x : -2x` setting the non-differentiable point to either side would result in it leaving that maxima. - -Further, a nice (albeit generally intractable) algorithm for finding the global optima is to take the set of all stationary points (i.e. points where the derivative is zero), combine that with the boundary points and evaluate the primal function at all of them. -The extrema of this set are the global optima. -This algorithm is only guaranteed correct for functions that are differentiable everywhere *or* that apply the sub/super-derivative convention and make all non-differentiable local optima claim to have a zero derivative. -It is also correct if you make other non-differentiable points have zero derivative, just slower. - -### It is sometimes the non-zero, especially if it boarders a flat section - -If a function has derivative zero in some large section of it's domain, like `relu` or `x->clamp(x, -1, 1)`, -a case can be made for choosing the non-zero derivative branch. -Depending exactly on the larger function being optimized, it is often the case that a zero gradient for this function means a total zero gradient (e.g. if the greater function is a chain of composed calls). -So once things move into the flat-region they stay there. -If we chose the other branch we move them into (just barely) the nonflat region. -If moving into the non-flat region was good they will move further there later. -If it is not good then they well be rapidly cast deeper into the flat region and we will not be at this boundary case. - -### It is sometimes the mean -Either the mean of the branches, or the overall mean of the sub/super-differential (which may be equal). - -A nice advantage of it is that it will agree with the result you will get from central finite differencing. -Which is what [ChainRulesTestUtils.jl](https://github.com/JuliaDiff/ChainRulesTestUtils.jl) defaults to. - -### All else being equal it is just one of the branches -Pick one. -It will be fast. -It will agree with either the forwards or reverse finite differencing results. - -### It is almost never `Inf`, `NaN` or an error -Practically speaking, people are generally using AD to perform some gradient descent like optimization procedure. -`NaN` and `Inf` do not generally take us to nice places. -Erroring can be worse, especially if it is a local minima -- the optimization fully converges to that minima and then throws an error rather than reporting the result. -If the primal function takes a non-finite value or errors on one side, then we are in the case that we are at a boundary of the domain. -Which means we are free to chose *any* value. -In particular we often want to chose value of the derivative from **the other side where the primal is defined.** - - -## Why is the sub/super-differential convention permissible? - -Math isn't real, and AD doesn't do calculus. -We are trying to accomplish some goal, and this approach works well. - -One way to think about this convention is to take an infinitely sharp discontinuity in a function and replace that discontinuty with a very (infinitesimally) small, smooth corner. -Due to the intermediate value theorem, we can then say that over this tiny interval, any derivative between the two extremes is realized and we are free to pick any one of them that we find useful as the 'canonical' value. - -![animation showing `abs(x)` transforming into `x^2`](https://user-images.githubusercontent.com/29157027/127267988-92ca0899-ca33-4dd6-af45-e5a085283c83.gif) - -More specifically, consider our initial examples: -`abs_left(x) = (x <= 0) ? -x : x`, and `abs_right(x) = (x < 0) ? -x : x`. -These are a a primal level indistinguishable to the user. -It is impossible to tell which `Base.abs` uses without looking at the source. -Thus the rule author must be free to chose between assuming it is either. -Which is equivalent to saying they are free to chose to return the derivative or either branch. -We can then take the continuous relaxation of that choice: to chose any value between them. -Which for that case is choosing any sub-differential. -We then generalize from that chose into the sub/super-differential convention. - -## How does this generalize to n-D -Carefully, but consistently. +Our goal as always, is to get a pragmatically useful result for everyone, which must by necessity also avoid a pathological result for anyone. \ No newline at end of file From ca4b1ffd586bc33cedf5fe75ed0f4f1d845d5e0a Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Tue, 9 Nov 2021 19:23:43 +0000 Subject: [PATCH 15/17] Update docs/src/nondiff_points.md --- docs/src/nondiff_points.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/src/nondiff_points.md b/docs/src/nondiff_points.md index acb9466e3..656df19fc 100644 --- a/docs/src/nondiff_points.md +++ b/docs/src/nondiff_points.md @@ -35,7 +35,6 @@ This is the standard case, one can returned the derivative that is defined accor Here we would reasonably say that at `x=0` the derivative is `3*0^2=0`. -An interesting thing about `x->x^3` is that at `x=0` the derivative is defined, but neither the sub-derivative nor super-derivative is defined. ### Local Minima / Maxima From 80be21e68cf8aec5ea98a14a7b87b7e61f3efbe6 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Wed, 10 Nov 2021 13:14:35 +0000 Subject: [PATCH 16/17] move to under math docs, and fix plotting on CI --- docs/Manifest.toml | 173 +++++++++++++++---------- docs/make.jl | 4 +- docs/src/{ => maths}/nondiff_points.md | 0 3 files changed, 107 insertions(+), 70 deletions(-) rename docs/src/{ => maths}/nondiff_points.md (100%) diff --git a/docs/Manifest.toml b/docs/Manifest.toml index 3a83ff319..99b3d09a0 100644 --- a/docs/Manifest.toml +++ b/docs/Manifest.toml @@ -17,27 +17,33 @@ uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" [[Bzip2_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] -git-tree-sha1 = "c3598e525718abcc440f69cc6d5f60dda0a1b61e" +git-tree-sha1 = "19a35467a82e236ff51bc17a3a44b69ef35185a2" uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" -version = "1.0.6+5" +version = "1.0.8+0" [[Cairo_jll]] deps = ["Artifacts", "Bzip2_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] -git-tree-sha1 = "e2f47f6d8337369411569fd45ae5753ca10394c6" +git-tree-sha1 = "f2202b55d816427cd385a9a4f3ffb226bee80f99" uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" -version = "1.16.0+6" +version = "1.16.1+0" [[ChainRulesCore]] deps = ["Compat", "LinearAlgebra", "SparseArrays"] path = ".." uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" -version = "1.0.1" +version = "1.11.1" + +[[ChangesOfVariables]] +deps = ["LinearAlgebra", "Test"] +git-tree-sha1 = "9a1d594397670492219635b35a3d830b04730d62" +uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0" +version = "0.1.1" [[ColorSchemes]] deps = ["ColorTypes", "Colors", "FixedPointNumbers", "Random"] -git-tree-sha1 = "9995eb3977fbf67b86d0a0a0508e83017ded03f2" +git-tree-sha1 = "a851fec56cb73cfdf43762999ec72eff5b86882a" uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4" -version = "3.14.0" +version = "3.15.0" [[ColorTypes]] deps = ["FixedPointNumbers", "Random"] @@ -68,9 +74,9 @@ uuid = "d38c429a-6771-53c6-b99e-75d170b6e991" version = "0.5.7" [[DataAPI]] -git-tree-sha1 = "ee400abb2298bd13bfc3df1c412ed228061a2385" +git-tree-sha1 = "cc70b17275652eb47bc9e5f81635981f13cea5c8" uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" -version = "1.7.0" +version = "1.9.0" [[DataStructures]] deps = ["Compat", "InteractiveUtils", "OrderedCollections"] @@ -119,9 +125,9 @@ uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" [[EarCut_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] -git-tree-sha1 = "92d8f9f208637e8d2d28c664051a00569c01493d" +git-tree-sha1 = "3f3a2501fa7236e9b911e0f7a588c657e822bb6d" uuid = "5ae413db-bbd1-5e63-b57d-d24a61df00f5" -version = "2.1.5+1" +version = "2.2.3+0" [[Expat_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] @@ -136,10 +142,10 @@ uuid = "c87230d0-a227-11e9-1b43-d7ebe4e7570a" version = "0.4.1" [[FFMPEG_jll]] -deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "LibVPX_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "Pkg", "Zlib_jll", "libass_jll", "libfdk_aac_jll", "libvorbis_jll", "x264_jll", "x265_jll"] -git-tree-sha1 = "3cc57ad0a213808473eafef4845a74766242e05f" +deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "Pkg", "Zlib_jll", "libass_jll", "libfdk_aac_jll", "libvorbis_jll", "x264_jll", "x265_jll"] +git-tree-sha1 = "d8a578692e3077ac998b50c0217dfd67f21d1e5f" uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5" -version = "4.3.1+4" +version = "4.4.0+0" [[FixedPointNumbers]] deps = ["Statistics"] @@ -149,9 +155,9 @@ version = "0.8.4" [[Fontconfig_jll]] deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Pkg", "Zlib_jll"] -git-tree-sha1 = "35895cf184ceaab11fd778b4590144034a167a2f" +git-tree-sha1 = "21efd19106a55620a188615da6d3d06cd7f6ee03" uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" -version = "2.13.1+14" +version = "2.13.93+0" [[Formatting]] deps = ["Printf"] @@ -161,9 +167,9 @@ version = "0.4.2" [[FreeType2_jll]] deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] -git-tree-sha1 = "cbd58c9deb1d304f5a245a0b7eb841a2560cfec6" +git-tree-sha1 = "87eb71354d8ec1a96d4a7636bd57a7347dde3ef9" uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" -version = "2.10.1+5" +version = "2.10.4+0" [[FriBidi_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] @@ -173,21 +179,21 @@ version = "1.0.10+0" [[GLFW_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Libglvnd_jll", "Pkg", "Xorg_libXcursor_jll", "Xorg_libXi_jll", "Xorg_libXinerama_jll", "Xorg_libXrandr_jll"] -git-tree-sha1 = "dba1e8614e98949abfa60480b13653813d8f0157" +git-tree-sha1 = "0c603255764a1fa0b61752d2bec14cfbd18f7fe8" uuid = "0656b61e-2033-5cc2-a64a-77c0f6c09b89" -version = "3.3.5+0" +version = "3.3.5+1" [[GR]] deps = ["Base64", "DelimitedFiles", "GR_jll", "HTTP", "JSON", "Libdl", "LinearAlgebra", "Pkg", "Printf", "Random", "Serialization", "Sockets", "Test", "UUIDs"] -git-tree-sha1 = "182da592436e287758ded5be6e32c406de3a2e47" +git-tree-sha1 = "30f2b340c2fff8410d89bfcdc9c0a6dd661ac5f7" uuid = "28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71" -version = "0.58.1" +version = "0.62.1" [[GR_jll]] deps = ["Artifacts", "Bzip2_jll", "Cairo_jll", "FFMPEG_jll", "Fontconfig_jll", "GLFW_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pixman_jll", "Pkg", "Qt5Base_jll", "Zlib_jll", "libpng_jll"] -git-tree-sha1 = "d59e8320c2747553788e4fc42231489cc602fa50" +git-tree-sha1 = "fd75fa3a2080109a2c0ec9864a6e14c60cca3866" uuid = "d2c73de3-f751-5644-a686-071e5b155ba9" -version = "0.58.1+0" +version = "0.62.0+0" [[GeometryBasics]] deps = ["EarCut_jll", "IterTools", "LinearAlgebra", "StaticArrays", "StructArrays", "Tables"] @@ -207,6 +213,12 @@ git-tree-sha1 = "7bf67e9a481712b3dbe9cb3dac852dc4b1162e02" uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" version = "2.68.3+0" +[[Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.14+0" + [[Grisu]] git-tree-sha1 = "53bb909d1151e57e2484c3d1b53e19552b887fb2" uuid = "42e2da0e-8278-4e71-bc24-59509adca0fe" @@ -214,9 +226,15 @@ version = "1.0.2" [[HTTP]] deps = ["Base64", "Dates", "IniFile", "Logging", "MbedTLS", "NetworkOptions", "Sockets", "URIs"] -git-tree-sha1 = "44e3b40da000eab4ccb1aecdc4801c040026aeb5" +git-tree-sha1 = "14eece7a3308b4d8be910e265c724a6ba51a9798" uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3" -version = "0.9.13" +version = "0.9.16" + +[[HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] +git-tree-sha1 = "8a954fed8ac097d5be04921d595f741115c1b2ad" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "2.8.1+0" [[IOCapture]] deps = ["Logging"] @@ -234,6 +252,17 @@ version = "0.5.0" deps = ["Markdown"] uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" +[[InverseFunctions]] +deps = ["Test"] +git-tree-sha1 = "a7254c0acd8e62f1ac75ad24d5db43f5f19f3c65" +uuid = "3587e190-3f89-42d0-90ee-14403ec27112" +version = "0.1.2" + +[[IrrationalConstants]] +git-tree-sha1 = "7fd44fd4ff43fc60815f8e764c0f352b83c49151" +uuid = "92d709cd-6900-40b7-9082-c6be49f344b6" +version = "0.1.1" + [[IterTools]] git-tree-sha1 = "05110a2ab1fc5f932622ffea2a003221f4782c18" uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e" @@ -275,15 +304,15 @@ uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac" version = "2.10.1+0" [[LaTeXStrings]] -git-tree-sha1 = "c7f1c695e06c01b95a67f0cd1d34994f3e7db104" +git-tree-sha1 = "f2355693d6778a178ade15952b7ac47a4ff97996" uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" -version = "1.2.1" +version = "1.3.0" [[Latexify]] deps = ["Formatting", "InteractiveUtils", "LaTeXStrings", "MacroTools", "Markdown", "Printf", "Requires"] -git-tree-sha1 = "a4b12a1bd2ebade87891ab7e36fdbce582301a92" +git-tree-sha1 = "a8f4f279b6fa3c3c4f1adadd78a621b13a506bce" uuid = "23fbe1c1-3f47-55db-b15f-69d7ec21a316" -version = "0.15.6" +version = "0.15.9" [[LibCURL]] deps = ["LibCURL_jll", "MozillaCACerts_jll"] @@ -301,12 +330,6 @@ uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" deps = ["Artifacts", "Libdl", "MbedTLS_jll"] uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" -[[LibVPX_jll]] -deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] -git-tree-sha1 = "12ee7e23fa4d18361e7c2cde8f8337d4c3101bc7" -uuid = "dd192d2f-8180-539f-9fb4-cc70b1dcf69a" -version = "1.10.0+0" - [[Libdl]] uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" @@ -362,14 +385,20 @@ version = "2.36.0+0" deps = ["Libdl"] uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" +[[LogExpFunctions]] +deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"] +git-tree-sha1 = "be9eef9f9d78cecb6f262f3c10da151a6c5ab827" +uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" +version = "0.3.5" + [[Logging]] uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" [[MacroTools]] deps = ["Markdown", "Random"] -git-tree-sha1 = "0fb723cd8c45858c22169b2e42269e53271a6df7" +git-tree-sha1 = "3d3e902b31198a27340d0bf00d6ac452866021cf" uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" -version = "0.5.7" +version = "0.5.9" [[Markdown]] deps = ["Base64"] @@ -392,9 +421,9 @@ version = "0.3.1" [[Missings]] deps = ["DataAPI"] -git-tree-sha1 = "4ea90bd5d3985ae1f9a908bd4500ae88921c5ce7" +git-tree-sha1 = "bf210ce90b6c9eed32d25dbcae1ebc565df2687f" uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" -version = "1.0.0" +version = "1.0.2" [[Mmap]] uuid = "a63ad114-7e13-5084-954f-fe012c677804" @@ -463,15 +492,15 @@ version = "2.0.1" [[PlotUtils]] deps = ["ColorSchemes", "Colors", "Dates", "Printf", "Random", "Reexport", "Statistics"] -git-tree-sha1 = "501c20a63a34ac1d015d5304da0e645f42d91c9f" +git-tree-sha1 = "b084324b4af5a438cd63619fd006614b3b20b87b" uuid = "995b91a9-d308-5afd-9ec6-746e21dbc043" -version = "1.0.11" +version = "1.0.15" [[Plots]] -deps = ["Base64", "Contour", "Dates", "FFMPEG", "FixedPointNumbers", "GR", "GeometryBasics", "JSON", "Latexify", "LinearAlgebra", "Measures", "NaNMath", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "RecipesPipeline", "Reexport", "Requires", "Scratch", "Showoff", "SparseArrays", "Statistics", "StatsBase", "UUIDs"] -git-tree-sha1 = "8365fa7758e2e8e4443ce866d6106d8ecbb4474e" +deps = ["Base64", "Contour", "Dates", "Downloads", "FFMPEG", "FixedPointNumbers", "GR", "GeometryBasics", "JSON", "Latexify", "LinearAlgebra", "Measures", "NaNMath", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "RecipesPipeline", "Reexport", "Requires", "Scratch", "Showoff", "SparseArrays", "Statistics", "StatsBase", "UUIDs", "UnicodeFun"] +git-tree-sha1 = "7dc03c2b145168f5854085a16d054429d612b637" uuid = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" -version = "1.20.1" +version = "1.23.5" [[Preferences]] deps = ["TOML"] @@ -498,20 +527,20 @@ deps = ["Serialization"] uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" [[RecipesBase]] -git-tree-sha1 = "b3fb709f3c97bfc6e948be68beeecb55a0b340ae" +git-tree-sha1 = "44a75aa7a527910ee3d1751d1f0e4148698add9e" uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" -version = "1.1.1" +version = "1.1.2" [[RecipesPipeline]] deps = ["Dates", "NaNMath", "PlotUtils", "RecipesBase"] -git-tree-sha1 = "2a7a2469ed5d94a98dea0e85c46fa653d76be0cd" +git-tree-sha1 = "7ad0dfa8d03b7bcf8c597f59f5292801730c55b8" uuid = "01d81517-befc-4cb6-b9ec-a95719d0359c" -version = "0.3.4" +version = "0.4.1" [[Reexport]] -git-tree-sha1 = "5f6c21241f0f655da3952fd60aa18477cf96c220" +git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b" uuid = "189a3867-3050-52da-a836-e630ba90ab69" -version = "1.1.0" +version = "1.2.2" [[Requires]] deps = ["UUIDs"] @@ -562,9 +591,9 @@ uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" [[StaticArrays]] deps = ["LinearAlgebra", "Random", "Statistics"] -git-tree-sha1 = "3240808c6d463ac46f1c1cd7638375cd22abbccb" +git-tree-sha1 = "3c76dde64d03699e074ac02eb2e8ba8254d428da" uuid = "90137ffa-7385-5640-81b9-e52037218182" -version = "1.2.12" +version = "1.2.13" [[Statistics]] deps = ["LinearAlgebra", "SparseArrays"] @@ -576,16 +605,16 @@ uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" version = "1.0.0" [[StatsBase]] -deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] -git-tree-sha1 = "fed1ec1e65749c4d96fc20dd13bea72b55457e62" +deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] +git-tree-sha1 = "eb35dcc66558b2dda84079b9a1be17557d32091a" uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" -version = "0.33.9" +version = "0.33.12" [[StructArrays]] deps = ["Adapt", "DataAPI", "StaticArrays", "Tables"] -git-tree-sha1 = "000e168f5cc9aded17b6999a560b7c11dda69095" +git-tree-sha1 = "2ce41e0d042c60ecd131e9fb7154a3bfadbf50d3" uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a" -version = "0.6.0" +version = "0.6.3" [[TOML]] deps = ["Dates"] @@ -599,9 +628,9 @@ version = "1.0.1" [[Tables]] deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"] -git-tree-sha1 = "d0c690d37c73aeb5ca063056283fde5585a41710" +git-tree-sha1 = "fed34d0e71b91734bf0a7e10eb1bb05296ddbcd0" uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" -version = "1.5.0" +version = "1.6.0" [[Tar]] deps = ["ArgTools", "SHA"] @@ -623,6 +652,12 @@ uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" [[Unicode]] uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" +[[UnicodeFun]] +deps = ["REPL"] +git-tree-sha1 = "53915e50200959667e78a92a418594b428dffddf" +uuid = "1cfade01-22cf-5700-b092-accc4b62d6e1" +version = "0.4.1" + [[Wayland_jll]] deps = ["Artifacts", "Expat_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg", "XML2_jll"] git-tree-sha1 = "3e61f0b86f90dacb0bc0e73a0c5a83f6a8636e23" @@ -784,16 +819,16 @@ uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" version = "1.5.0+0" [[libass_jll]] -deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] -git-tree-sha1 = "acc685bcf777b2202a904cdcb49ad34c2fa1880c" +deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "HarfBuzz_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] +git-tree-sha1 = "5982a94fcba20f02f42ace44b9894ee2b140fe47" uuid = "0ac62f75-1d6f-5e53-bd7c-93b484bb37c0" -version = "0.14.0+4" +version = "0.15.1+0" [[libfdk_aac_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] -git-tree-sha1 = "7a5780a0d9c6864184b3a2eeeb833a0c871f00ab" +git-tree-sha1 = "daacc84a041563f965be61859a36e17c4e4fcd55" uuid = "f638f0a6-7fb0-5443-88ba-1cc74229b280" -version = "0.1.6+4" +version = "2.0.2+0" [[libpng_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] @@ -823,15 +858,15 @@ uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" [[x264_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] -git-tree-sha1 = "d713c1ce4deac133e3334ee12f4adff07f81778f" +git-tree-sha1 = "4fea590b89e6ec504593146bf8b988b2c00922b2" uuid = "1270edf5-f2f9-52d2-97e9-ab00b5d0237a" -version = "2020.7.14+2" +version = "2021.5.5+0" [[x265_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] -git-tree-sha1 = "487da2f8f2f0c8ee0e83f39d13037d6bbf0a45ab" +git-tree-sha1 = "ee567a171cce03570d77ad3a43e90218e38937a9" uuid = "dfaa095f-4041-5dcd-9319-2fabd8486b76" -version = "3.0.0+3" +version = "3.5.0+0" [[xkbcommon_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Wayland_jll", "Wayland_protocols_jll", "Xorg_libxcb_jll", "Xorg_xkeyboard_config_jll"] diff --git a/docs/make.jl b/docs/make.jl index 3e1233608..e817a8842 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -1,3 +1,5 @@ +ENV["GKSwstype"] = "100" # make Plots/GR work on headless machine + using ChainRulesCore using Documenter using DocThemeIndigo @@ -58,7 +60,6 @@ makedocs(; "`RuleConfig`" => "rule_author/superpowers/ruleconfig.md", "Gradient accumulation" => "rule_author/superpowers/gradient_accumulation.md", ], - "Non-differentiable Points" => "nondiff_points.md", "Converting ZygoteRules.@adjoint to rrules" => "rule_author/converting_zygoterules.md", "Tips for making your package work with AD" => "rule_author/tips_for_packages.md", "Debug mode" => "rule_author/debug_mode.md", @@ -70,6 +71,7 @@ makedocs(; ], "The maths" => [ "The propagators: pushforward and pullback" => "maths/propagators.md", + "Non-differentiable Points" => "maths/nondiff_points.md", "Complex numbers" => "maths/complex.md", "Deriving array rules" => "maths/arrays.md", ], diff --git a/docs/src/nondiff_points.md b/docs/src/maths/nondiff_points.md similarity index 100% rename from docs/src/nondiff_points.md rename to docs/src/maths/nondiff_points.md From efdb8d0b5710ab933ce894694263799f7e555b21 Mon Sep 17 00:00:00 2001 From: Lyndon White Date: Wed, 10 Nov 2021 13:36:09 +0000 Subject: [PATCH 17/17] update short version --- docs/src/maths/nondiff_points.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/src/maths/nondiff_points.md b/docs/src/maths/nondiff_points.md index 656df19fc..4200a1447 100644 --- a/docs/src/maths/nondiff_points.md +++ b/docs/src/maths/nondiff_points.md @@ -1,12 +1,10 @@ # What to return for non-differentiable points !!! info "What is the short version?" - If the function is not differentiable due to e.g. a branch, like `abs`, your rule can reasonably claim the derivative at that point is the value from either branch, *or* any value in-between (e.g. for `abs` claiming 0 is a good idea). - If it is not differentiable due to the primal not being defined on one side, you can set it to what ever you like. - Your rule should claim a derivative that is *useful*. -In calculus one learns that if the derivative as computed by approaching from the left, -and the derivative one computes as approaching from the right are not equal then the derivative is not defined, -and we say the function is not differentiable at that point. -This is distinct from the notion captured by [`NoTangent`](@ref), which is that the tangent space itself is not defined: because in some sense the primal value can not be perturbed e.g. is is a discrete type. + If the function is not-differentiable choose to return something useful rather than erroring. + For a branch a function is not differentiable due to e.g. a branch, like `abs`, your rule can reasonably claim the derivative at that point is the value from either branch, *or* any value in-between. + In particular for local optima (like in the case of `abs`) claiming the derivative is 0 is a good idea. + Similarly, if derivative is from one side is not defined, or is not finite, return the derivative from the other side. + Throwing an error, or returning `NaN` is generally the least useful option. However, contrary to what calculus says most autodiff systems will return an answer for such functions. For example for: `abs_left(x) = (x <= 0) ? -x : x`, AD will say the derivative at `x=0` is `-1`. @@ -137,4 +135,4 @@ These rough rules are: - If the derivative from one side is finite and the other isn't, say it is the derivative taken from finite side. - When derivative from each side is not equal, strongly consider reporting the average -Our goal as always, is to get a pragmatically useful result for everyone, which must by necessity also avoid a pathological result for anyone. \ No newline at end of file +Our goal as always, is to get a pragmatically useful result for everyone, which must by necessity also avoid a pathological result for anyone.