From 87f3e1f8a39e3c182a94fb9a62edcf225649bcff Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Thu, 18 Apr 2024 13:39:02 -0700 Subject: [PATCH 01/13] Add axis argument to softmax() Frameworks (TensorFlow, PyTorch, ONNX) all accept an axis parameter. Most backends also support an axis, or it can be emulated with a reshape. As @fdwr wrote: So it's achievable in each backend... but it would move the pain from the caller down to where it can be handled efficiently. Fixes #466 --- index.bs | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/index.bs b/index.bs index 3b07c538..95695a34 100644 --- a/index.bs +++ b/index.bs @@ -640,7 +640,7 @@ The {{MLGraphBuilder}} interface serves as a builder (factory) to construct a [= In WebNN, a [=computational graph=] is composed of operators which act on data, and are the nodes of the graph. {{MLOperand}}s are a representation of data that flows within the computational graph, and are the edges of the graph. {{MLOperand}}s include a [=computational graph=]'s input values for inference, constants (including trained weights) used for inference, intermediate values (often referred to as activations) computed during inference, as well as the output values of inference. An [=operator=]'s input is one or more {{MLOperand}}s. An [=operator=]'s output is one or more {{MLOperand}}s. [=Operators=] have operator-specific parameters that control their behavior, which can include zero or more activation functions, which are {{MLActivation}}s. -A key part of the {{MLGraphBuilder}} interface are methods such as {{MLGraphBuilder/gemm()}} and {{MLGraphBuilder/softmax()}} which create an [=operator=] which represents the actual operation to perform on the input data when the computation is run, and return a new {{MLOperand}} or {{MLActivation}} holding the operator. Methods that create an {{MLOperand}} connect any [=operator/inputs=] and [=operator/activations=] to the operator. Each method invocation returns a distinct new value, without changing the value of any other {{MLOperand}}. +A key part of the {{MLGraphBuilder}} interface are methods such as {{MLGraphBuilder/gemm()}} and {{MLGraphBuilder/hardSwish()}} which create an [=operator=] which represents the actual operation to perform on the input data when the computation is run, and return a new {{MLOperand}} or {{MLActivation}} holding the operator. Methods that create an {{MLOperand}} connect any [=operator/inputs=] and [=operator/activations=] to the operator. Each method invocation returns a distinct new value, without changing the value of any other {{MLOperand}}. At inference time, every {{MLOperand}} will be bound to a tensor (the actual data), which are essentially multidimensional arrays. The representation of the tensors is implementation dependent, but it typically includes the array data stored in some buffer (memory) and some metadata describing the array data (such as its shape). @@ -5274,8 +5274,8 @@ Compute the [softmax](https://en.wikipedia.org/wiki/Softmax_function) values of the 2-D input tensor along axis 1. @@ -5290,9 +5290,9 @@ partial interface MLGraphBuilder { // of the input values itself, in order to increase the numerical stability of // the result. // [1]: https://cs231n.github.io/linear-classify/#softmax - const max_x = builder.reduceMax(x, { axes: [1], keepDimensions: true }); + const max_x = builder.reduceMax(x, { axes: [axis], keepDimensions: true }); const exp_x = builder.exp(builder.sub(x, max_x)); - return builder.div(exp_x, builder.reduceSum(exp_x, { axes: [1], keepDimensions: true })); + return builder.div(exp_x, builder.reduceSum(exp_x, { axes: [axis], keepDimensions: true })); @@ -5301,6 +5301,7 @@ partial interface MLGraphBuilder {
**Arguments:** - *input*: an {{MLOperand}}. The input 2-D tensor. + - *axis*: an {{unsigned long}} scalar. The dimension the softmax will be performed on. **Returns:** - an {{MLOperand}}. The output 2-D tensor that contains the softmax results, of the same shape as *input*. @@ -5308,20 +5309,20 @@ partial interface MLGraphBuilder {
- The softmax(|input|) method steps are: + The softmax(|input|, |axis|) method steps are: 1. If [=MLGraphBuilder/validating operand=] with [=this=] and |input| returns false, then [=exception/throw=] a {{TypeError}}. 1. If |input|'s [=MLOperand/rank=] is not 2, then [=exception/throw=] a {{TypeError}}. 1. *Make graph connections:* 1. Let |output| be the result of [=copying an MLOperand=] given |input|. - 1. Let |operator| be an [=operator=] for the softmax operation. + 1. Let |operator| be an [=operator=] for the softmax operation, given |axis|. 1. Set |output|.{{MLOperand/[[operator]]}} to |operator|. 1. Set |operator|'s [=operator/input=] to |input|. 1. Set |operator|'s [=operator/output=] to |output|. 1. Return |output|.
-#### {{MLGraphBuilder/softmax()}} #### {#api-mlgraphbuilder-softmax} +#### {{MLGraphBuilder/softmax(axis)}} #### {#api-mlgraphbuilder-softmax}
**Arguments:** - None. @@ -5332,9 +5333,9 @@ partial interface MLGraphBuilder {
- The softmax() method steps are: + The softmax(|axis|) method steps are: - 1. Let |op| be the result of [=creating an MLActivation=] given [=this=] and "softmax". + 1. Let |op| be the result of [=creating an MLActivation=] given [=this=], "softmax", and «[ "axis" → |axis| ]». 1. Return |op|.
From 83349191d4adda232c13dc568b58493b1e96ceb5 Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Fri, 19 Apr 2024 09:04:55 -0700 Subject: [PATCH 02/13] revert activation example to softmax --- docs/SpecCodingConventions.md | 7 ++++--- index.bs | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/SpecCodingConventions.md b/docs/SpecCodingConventions.md index b33ab653..709a100b 100644 --- a/docs/SpecCodingConventions.md +++ b/docs/SpecCodingConventions.md @@ -78,8 +78,8 @@ Example: * The spec is encoded with UTF-8. * For non-ASCII characters, prefer to use characters directly, rather than [character references](https://html.spec.whatwg.org/multipage/syntax.html#character-references) (a.k.a. entities), except when necessary for escaping e.g. `sequence<DOMString>`. These commonly occur in names in the Acknowledgements and References sections. * Commonly used punctuation and symbol characters include: - * « » (U+00AB / U+00BB Left/Right Pointing Double Angle Quotation Marks) used for [list literals](https://infra.spec.whatwg.org/#lists) - * → (U+2192 Rightwards Arrow) used for [map iteration](https://infra.spec.whatwg.org/#map-iterate) + * « » (U+00AB / U+00BB Left/Right Pointing Double Angle Quotation Marks) used for [list literals](https://infra.spec.whatwg.org/#lists) and [map literals](https://infra.spec.whatwg.org/#maps). + * → (U+2192 Rightwards Arrow) used for [map iteration](https://infra.spec.whatwg.org/#map-iterate) and [map literals](https://infra.spec.whatwg.org/#maps). * In expressions: * Use * (U+002A Asterisk) for multiplication, / (U+002F Solidus) for division, and - (U+002D Hyphen-Minux), to reduce friction for implementers. Don't use × (U+00D7 Multiplication Sign), ∗ (U+2217 Asterisk Operator), ÷ (U+00F7 Division Sign), or − (U+2212 Minus Sign). * Use named functions like _floor(x)_ and _ceil()_ rather than syntax like ⌊_x_⌋ and ⌈_x_⌉. @@ -108,7 +108,8 @@ Example: * Use `[=list/For each=] |item| of |list|` when iterating over a list, but use more specific terms for the item (e.g. _For each dimension of dimensions:_) * Use `[=list/For each=] |index| in [=the range=] X to Y, inclusive` when iterating over a numeric range; a range is implicitly an ordered set which is a type of list. Specify _inclusive_ or _exclusive_ regarding the upper bound, for clarity. * Use "let" to introduce a variable and "set" to update a variable or assign to a property. -* Use « » notation for literal lists, which helps make it clear that they are not JavaScript arrays. +* Use « » notation for literal [lists](https://infra.spec.whatwg.org/#lists), which helps make it clear that they are not JavaScript arrays. +* Use «[ _k_ → _v_ ]» notation for literal [maps](https://infra.spec.whatwg.org/#maps). * When referring to abstract properties, use the short possessive form `|object|'s [=property=]`. Avoid the wordier `the [=property=] of |object|` form. * Use "rank" when describing the number of dimensions of a tensor (e.g. in variable names) rather than the ambiguous "size". * Only use single capital letters as variable names when referring to tensors; i.e. prefer `|shapeA|` to `|A|`, but tensor `|T|` is okay. diff --git a/index.bs b/index.bs index 0afd4eef..8f1ff6fb 100644 --- a/index.bs +++ b/index.bs @@ -640,7 +640,7 @@ The {{MLGraphBuilder}} interface serves as a builder (factory) to construct a [= In WebNN, a [=computational graph=] is composed of operators which act on data, and are the nodes of the graph. {{MLOperand}}s are a representation of data that flows within the computational graph, and are the edges of the graph. {{MLOperand}}s include a [=computational graph=]'s input values for inference, constants (including trained weights) used for inference, intermediate values (often referred to as activations) computed during inference, as well as the output values of inference. An [=operator=]'s input is one or more {{MLOperand}}s. An [=operator=]'s output is one or more {{MLOperand}}s. [=Operators=] have operator-specific parameters that control their behavior, which can include zero or more activation functions, which are {{MLActivation}}s. -A key part of the {{MLGraphBuilder}} interface are methods such as {{MLGraphBuilder/gemm()}} and {{MLGraphBuilder/hardSwish()}} which create an [=operator=] which represents the actual operation to perform on the input data when the computation is run, and return a new {{MLOperand}} or {{MLActivation}} holding the operator. Methods that create an {{MLOperand}} connect any [=operator/inputs=] and [=operator/activations=] to the operator. Each method invocation returns a distinct new value, without changing the value of any other {{MLOperand}}. +A key part of the {{MLGraphBuilder}} interface are methods such as {{MLGraphBuilder/gemm()}} and {{MLGraphBuilder/softmax(axis)|softmax()}} which create an [=operator=] which represents the actual operation to perform on the input data when the computation is run, and return a new {{MLOperand}} or {{MLActivation}} holding the operator. Methods that create an {{MLOperand}} connect any [=operator/inputs=] and [=operator/activations=] to the operator. Each method invocation returns a distinct new value, without changing the value of any other {{MLOperand}}. At inference time, every {{MLOperand}} will be bound to a tensor (the actual data), which are essentially multidimensional arrays. The representation of the tensors is implementation dependent, but it typically includes the array data stored in some buffer (memory) and some metadata describing the array data (such as its shape). From 99e87734fa8daa7f69c95d18c8c075fc6ed0b592 Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Fri, 19 Apr 2024 09:06:46 -0700 Subject: [PATCH 03/13] validate softmax axis against inputs rank --- index.bs | 1 + 1 file changed, 1 insertion(+) diff --git a/index.bs b/index.bs index 8f1ff6fb..0cac7cc4 100644 --- a/index.bs +++ b/index.bs @@ -5317,6 +5317,7 @@ partial interface MLGraphBuilder { 1. If [=MLGraphBuilder/validating operand=] with [=this=] and |input| returns false, then [=exception/throw=] a {{TypeError}}. 1. If |input|'s [=MLOperand/rank=] is not 2, then [=exception/throw=] a {{TypeError}}. + 1. If |axis| is greater than or equal to |input|'s [=MLOperand/rank=], then [=exception/throw=] a {{TypeError}}. 1. *Make graph connections:* 1. Let |output| be the result of [=copying an MLOperand=] given |input|. 1. Let |operator| be an [=operator=] for the softmax operation, given |axis|. From 66bf64a4ce6305cffb2920be626fdd8d4f22b273 Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Fri, 19 Apr 2024 09:10:33 -0700 Subject: [PATCH 04/13] update TOC headers --- index.bs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.bs b/index.bs index 0cac7cc4..149c7b7c 100644 --- a/index.bs +++ b/index.bs @@ -5301,7 +5301,7 @@ partial interface MLGraphBuilder {
-#### {{MLGraphBuilder/softmax(input)}} #### {#api-mlgraphbuilder-softmax-input} +#### {{MLGraphBuilder/softmax(input, axis)}} #### {#api-mlgraphbuilder-softmax-input-axis}
**Arguments:** - *input*: an {{MLOperand}}. The input 2-D tensor. @@ -5327,7 +5327,7 @@ partial interface MLGraphBuilder { 1. Return |output|. -#### {{MLGraphBuilder/softmax(axis)}} #### {#api-mlgraphbuilder-softmax} +#### {{MLGraphBuilder/softmax(axis)}} #### {#api-mlgraphbuilder-softmax-axis}
**Arguments:** - None. From ab22738fddec26a76a885445e10cb79f2be77be5 Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Mon, 22 Apr 2024 09:23:59 -0700 Subject: [PATCH 05/13] Update index.bs Co-authored-by: Dwayne Robinson --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index 149c7b7c..deec4f3e 100644 --- a/index.bs +++ b/index.bs @@ -5305,7 +5305,7 @@ partial interface MLGraphBuilder {
**Arguments:** - *input*: an {{MLOperand}}. The input 2-D tensor. - - *axis*: an {{unsigned long}} scalar. The dimension the softmax will be performed on. + - *axis*: an {{unsigned long}} scalar. The dimension the reduction will be performed on. **Returns:** - an {{MLOperand}}. The output 2-D tensor that contains the softmax results, of the same shape as *input*. From 7f33accb64e89ba4c9ce41fce14ee0b7b120304a Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Mon, 22 Apr 2024 10:33:20 -0700 Subject: [PATCH 06/13] camelCase not snake_case --- index.bs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/index.bs b/index.bs index deec4f3e..837fb614 100644 --- a/index.bs +++ b/index.bs @@ -5294,9 +5294,9 @@ partial interface MLGraphBuilder { // of the input values itself, in order to increase the numerical stability of // the result. // [1]: https://cs231n.github.io/linear-classify/#softmax - const max_x = builder.reduceMax(x, { axes: [axis], keepDimensions: true }); - const exp_x = builder.exp(builder.sub(x, max_x)); - return builder.div(exp_x, builder.reduceSum(exp_x, { axes: [axis], keepDimensions: true })); + const maxX = builder.reduceMax(x, { axes: [axis], keepDimensions: true }); + const expX = builder.exp(builder.sub(x, maxX)); + return builder.div(expX, builder.reduceSum(expX, { axes: [axis], keepDimensions: true }));
From 1877f2eb13c35a375c7ca86b112dc27ace7d9a4e Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Mon, 22 Apr 2024 14:51:40 -0700 Subject: [PATCH 07/13] Remove unnecessary condition --- index.bs | 1 - 1 file changed, 1 deletion(-) diff --git a/index.bs b/index.bs index 837fb614..0fd1deb4 100644 --- a/index.bs +++ b/index.bs @@ -5316,7 +5316,6 @@ partial interface MLGraphBuilder { The softmax(|input|, |axis|) method steps are: 1. If [=MLGraphBuilder/validating operand=] with [=this=] and |input| returns false, then [=exception/throw=] a {{TypeError}}. - 1. If |input|'s [=MLOperand/rank=] is not 2, then [=exception/throw=] a {{TypeError}}. 1. If |axis| is greater than or equal to |input|'s [=MLOperand/rank=], then [=exception/throw=] a {{TypeError}}. 1. *Make graph connections:* 1. Let |output| be the result of [=copying an MLOperand=] given |input|. From 3e0c6705a35653ab6bea8907a28ed0affd4da330 Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Tue, 23 Apr 2024 07:52:14 -0700 Subject: [PATCH 08/13] Update index.bs Co-authored-by: Dwayne Robinson --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index 0fd1deb4..bd868a95 100644 --- a/index.bs +++ b/index.bs @@ -5304,7 +5304,7 @@ partial interface MLGraphBuilder { #### {{MLGraphBuilder/softmax(input, axis)}} #### {#api-mlgraphbuilder-softmax-input-axis}
**Arguments:** - - *input*: an {{MLOperand}}. The input 2-D tensor. + - *input*: an {{MLOperand}}. The input N-D tensor. - *axis*: an {{unsigned long}} scalar. The dimension the reduction will be performed on. **Returns:** From eb1233b44ab5febc5bea302a1fc6ce9ffa27707c Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Tue, 23 Apr 2024 07:52:38 -0700 Subject: [PATCH 09/13] Update index.bs Co-authored-by: Dwayne Robinson --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index bd868a95..26128ba1 100644 --- a/index.bs +++ b/index.bs @@ -5308,7 +5308,7 @@ partial interface MLGraphBuilder { - *axis*: an {{unsigned long}} scalar. The dimension the reduction will be performed on. **Returns:** - - an {{MLOperand}}. The output 2-D tensor that contains the softmax results, of the same shape as *input*. + - an {{MLOperand}}. The output N-D tensor that contains the softmax results, of the same shape as *input*.
From 216c9bbbf180a338fc8aa78e42a0645997f72654 Mon Sep 17 00:00:00 2001 From: Joshua Bell Date: Tue, 23 Apr 2024 07:52:49 -0700 Subject: [PATCH 10/13] Update index.bs Co-authored-by: Dwayne Robinson --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index 26128ba1..b6805830 100644 --- a/index.bs +++ b/index.bs @@ -5275,7 +5275,7 @@ partial interface MLGraphBuilder { ### softmax ### {#api-mlgraphbuilder-softmax-method} Compute the [softmax](https://en.wikipedia.org/wiki/Softmax_function) values of -the 2-D input tensor along axis 1. +the N-D input tensor along the given axis.