-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
While updating tests from the old AP to the new API, we found that the new API does not support ConfusionMatrix in the evaluation output.
machinelearning/src/Microsoft.ML.Data/Evaluators/MultiClassClassifierEvaluator.cs
Lines 506 to 520 in 21d64aa
| public MultiClassClassifierMetrics Evaluate(IDataView data, string label, string score, string predictedLabel) | |
| { | |
| Host.CheckValue(data, nameof(data)); | |
| Host.CheckNonEmpty(label, nameof(label)); | |
| Host.CheckNonEmpty(score, nameof(score)); | |
| Host.CheckNonEmpty(predictedLabel, nameof(predictedLabel)); | |
| var roles = new RoleMappedData(data, opt: false, | |
| RoleMappedSchema.ColumnRole.Label.Bind(label), | |
| RoleMappedSchema.CreatePair(MetadataUtils.Const.ScoreValueKind.Score, score), | |
| RoleMappedSchema.CreatePair(MetadataUtils.Const.ScoreValueKind.PredictedLabel, predictedLabel)); | |
| var resultDict = ((IEvaluator)this).Evaluate(roles); | |
| Host.Assert(resultDict.ContainsKey(MetricKinds.OverallMetrics)); | |
| var overall = resultDict[MetricKinds.OverallMetrics]; |
In the Legacy API, there was support for ConfusionMatrix
machinelearning/src/Microsoft.ML.Data/Evaluators/MultiClassClassifierEvaluator.cs
Lines 1031 to 1049 in 21d64aa
| public static partial class Evaluate | |
| { | |
| [TlcModule.EntryPoint(Name = "Models.ClassificationEvaluator", Desc = "Evaluates a multi class classification scored dataset.")] | |
| public static CommonOutputs.ClassificationEvaluateOutput MultiClass(IHostEnvironment env, MultiClassMamlEvaluator.Arguments input) | |
| { | |
| Contracts.CheckValue(env, nameof(env)); | |
| var host = env.Register("EvaluateMultiClass"); | |
| host.CheckValue(input, nameof(input)); | |
| EntryPointUtils.CheckInputArgs(host, input); | |
| MatchColumns(host, input, out string label, out string weight, out string name); | |
| IMamlEvaluator evaluator = new MultiClassMamlEvaluator(host, input); | |
| var data = new RoleMappedData(input.Data, label, null, null, weight, name); | |
| var metrics = evaluator.Evaluate(data); | |
| var warnings = ExtractWarnings(host, metrics); | |
| var overallMetrics = ExtractOverallMetrics(host, metrics, evaluator); | |
| var perInstanceMetrics = evaluator.GetPerInstanceMetrics(data); | |
| var confusionMatrix = ExtractConfusionMatrix(host, metrics); |
We should add back the checks for ConfusionMatrix in the following tests :
- TrainAndPredictIrisModelTest
- TrainAndPredictIrisModelWithStringLabelTest
bartczernicki
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request