Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 76 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,56 @@ export TAILOR_TOKEN=your_access_token
patterner metrics
```

The metrics command outputs detailed JSON data about your workspace resources.
The metrics command displays metrics in a table format. Use the `--out-octocov-path` option to output metrics in octocov custom metrics format.

#### Metrics Options

- `--since, -s` (default: "30min") - Analyze execution results since the specified time period
- `--out-octocov-path` - Output metrics in octocov custom metrics format to the specified file
- `--with-lint-warnings` (default: false) - Display lint warnings along with metrics output
- `--with-coverage-full-report` (default: false) - Display detailed pipeline resolver step coverage along with metrics output

##### Option Details

**--with-coverage-full-report vs --with-lint-warnings:**
- `--with-coverage-full-report`: Displays detailed pipeline resolver step coverage information for execution quality monitoring
- Shows coverage percentage and executed steps count for each resolver
- Format: `Coverage Rate% [Executed Steps/Total Steps] Resolver Name`
- `--with-lint-warnings`: Displays detailed lint warnings list for code quality monitoring
- Shows specific lint issues and recommendations for code improvements

Both options can be used together to provide comprehensive analysis combining execution quality and code quality insights.

#### Usage Examples

```bash
# Basic metrics (last 30 minutes)
patterner metrics

# Metrics for the past hour
patterner metrics --since 1hour

# Display metrics with lint warnings
patterner metrics --with-lint-warnings

# Display metrics with coverage details
patterner metrics --with-coverage-full-report

# Display metrics with both coverage and lint warnings
patterner metrics --with-coverage-full-report --with-lint-warnings

# Output metrics to octocov custom metrics file
patterner metrics --out-octocov-path metrics.json

# Comprehensive analysis with all options
patterner metrics --since 24hours --out-octocov-path metrics.json --with-coverage-full-report --with-lint-warnings
```

**Implementation Notes:**
```
<!-- Note: Current implementation has a typo in variable name 'lint_warnigns_total'
and potential variable reference issue in metrics.go that should be addressed -->
```

### View Coverage

Expand Down Expand Up @@ -141,22 +190,35 @@ The following metrics are collected and displayed:

**Pipeline Metrics:**

- `pipelines_total` - Total number of pipelines
- `pipeline_resolvers_total` - Total number of pipeline resolvers
- `pipeline_resolver_steps_total` - Total number of pipeline resolver steps
- `pipeline_resolver_execution_paths_total` - Total number of pipeline resolver execution paths
- Calculated based on the number of steps and tests in each resolver (steps \* 2^tests)
- `pipelines_total` - Total number of pipelines (Unit: count)
- `pipeline_resolvers_total` - Total number of pipeline resolvers (Unit: count)
- `pipeline_resolver_steps_total` - Total number of pipeline resolver steps (Unit: count)
- `pipeline_resolver_execution_paths_total` - Total number of pipeline resolver execution paths (Unit: count)
- Calculation: Based on the number of steps and tests in each resolver (steps \* 2^tests)
- Includes overflow detection: Reports error if negative values are encountered
- Used to understand the total number of execution paths based on testable step combinations

**TailorDB Metrics:**

- `tailordbs_total` - Total number of TailorDBs
- `tailordb_types_total` - Total number of TailorDB types
- `tailordb_type_fields_total` - Total number of TailorDB type fields
- `tailordbs_total` - Total number of TailorDBs (Unit: count)
- `tailordb_types_total` - Total number of TailorDB types (Unit: count)
- `tailordb_type_fields_total` - Total number of TailorDB type fields (Unit: count)

**StateFlow Metrics:**

- `stateflows_total` - Total number of StateFlows
- `stateflows_total` - Total number of StateFlows (Unit: count)

**Coverage Metrics:**

- `pipeline_resolver_step_coverage_percentage` - Pipeline resolver step coverage (Unit: %)
- Calculation: (covered steps / total steps) * 100
- Provides percentage representation of pipeline resolver step execution coverage

**Lint Metrics:**

- `lint_warnings_total` - Total number of lint warnings (Unit: count)
- Calculation: Number of warnings returned from the lint function
- Helps monitor code quality and adherence to best practices

## Configuration

Expand Down Expand Up @@ -247,6 +309,10 @@ lint:
- `patterner init` - Initialize configuration file
- `patterner lint` - Lint workspace resources
- `patterner metrics` - Display workspace metrics
- `--since, -s` (default: "30min") - Analyze execution results since the specified time period
- `--out-octocov-path` - Output metrics in octocov custom metrics format to the specified file
- `--with-lint-warnings` - Display lint warnings along with metrics output
- `--with-coverage-full-report` - Display detailed pipeline resolver step coverage along with metrics output
- `patterner coverage` - Display pipeline resolver step coverage

---
Expand Down
11 changes: 9 additions & 2 deletions cmd/coverage.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,10 @@ var coverageCmd = &cobra.Command{
var total, covered int
for _, rc := range coverage {
if fullReport {
cover := float64(float64(rc.CoveredSteps)/float64(rc.TotalSteps)) * 100
var cover float64
if rc.TotalSteps > 0 {
cover = float64(float64(rc.CoveredSteps)/float64(rc.TotalSteps)) * 100
}
fmt.Printf("%5s%% [%d/%d] %s\n", fmt.Sprintf("%.1f", cover), rc.CoveredSteps, rc.TotalSteps, rc.Name)
}
total += rc.TotalSteps
Expand All @@ -86,7 +89,11 @@ var coverageCmd = &cobra.Command{
if fullReport {
fmt.Println()
}
fmt.Printf("%s %.1f%% [%d/%d]\n", "Pipeline Resolver Step Coverage", float64(float64(covered)/float64(total))*100, covered, total)
var coverTotal float64
if total > 0 {
coverTotal = float64(float64(covered)/float64(total)) * 100
}
fmt.Printf("%s %.1f%% [%d/%d]\n", "Pipeline resolver step coverage", coverTotal, covered, total)

return nil
},
Expand Down
169 changes: 165 additions & 4 deletions cmd/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,25 @@ package cmd
import (
"encoding/json"
"fmt"
"math"
"os"
"time"

"github.com/k1LoW/duration"
"github.com/olekukonko/tablewriter"
"github.com/olekukonko/tablewriter/renderer"
"github.com/olekukonko/tablewriter/tw"
Comment thread
k1LoW marked this conversation as resolved.
"github.com/spf13/cobra"
"github.com/tailor-platform/patterner/config"
"github.com/tailor-platform/patterner/tailor"
)

var (
outOctocovPath string
withLintWarnings bool
withCoverageFullReport bool
)

var metricsCmd = &cobra.Command{
Use: "metrics",
Short: "retrieve and display metrics about the resources in the workspace",
Expand All @@ -49,24 +62,172 @@ var metricsCmd = &cobra.Command{
if err != nil {
return err
}
resources, err := c.Resources(cmd.Context())
d, err := duration.Parse(since)
if err != nil {
return err
}
s := time.Now().Add(-d)
resources, err := c.Resources(cmd.Context(), tailor.WithExecutionResults(&s))
if err != nil {
return err
}
spi.Disable()

if withLintWarnings {
fmt.Println("Lint warnings")
fmt.Println("============================================================")
warns, err := c.Lint(resources)
if err != nil {
return err
}
for _, w := range warns {
fmt.Printf("[%s] %s: %s\n", w.Type, w.Name, w.Message)
}
fmt.Println()
}

if withCoverageFullReport {
fmt.Println("Coverage")
fmt.Println("============================================================")
coverages, err := c.Coverage(resources)
if err != nil {
return err
}
for _, rc := range coverages {
var cover float64
if rc.TotalSteps > 0 {
cover = float64(float64(rc.CoveredSteps)/float64(rc.TotalSteps)) * 100
}
fmt.Printf("%5s%% [%d/%d] %s\n", fmt.Sprintf("%.1f", cover), rc.CoveredSteps, rc.TotalSteps, rc.Name)
}
fmt.Println()
}

if withCoverageFullReport || withLintWarnings {
fmt.Println("Metrics")
fmt.Println("============================================================")
}

metrics, err := c.Metrics(resources)
if err != nil {
return err
}
b, err := json.MarshalIndent(metrics, "", " ")
if err != nil {
table := tablewriter.NewTable(os.Stdout,
tablewriter.WithTrimSpace(tw.Off),
tablewriter.WithRenderer(renderer.NewBlueprint(tw.Rendition{
Borders: tw.BorderNone,
Symbols: tw.NewSymbols(tw.StyleNone),
Settings: tw.Settings{
Lines: tw.Lines{
ShowTop: tw.Off,
ShowBottom: tw.Off,
ShowHeaderLine: tw.Off,
ShowFooterLine: tw.Off,
},
Separators: tw.Separators{
ShowHeader: tw.Off,
ShowFooter: tw.Off,
BetweenRows: tw.Off,
BetweenColumns: tw.Off,
},
},
})),
tablewriter.WithHeaderConfig(tw.CellConfig{
Formatting: tw.CellFormatting{
AutoFormat: tw.Off,
Alignment: tw.AlignLeft,
},
Padding: tw.CellPadding{
Global: tw.Padding{Left: tw.Space, Right: tw.Space, Top: tw.Empty, Bottom: tw.Empty},
},
}),
tablewriter.WithRowConfig(tw.CellConfig{
Formatting: tw.CellFormatting{
AutoFormat: tw.Off,
},
ColumnAligns: []tw.Align{tw.AlignLeft, tw.AlignRight},
Padding: tw.CellPadding{
Global: tw.Padding{Left: tw.Space, Right: tw.Space, Top: tw.Empty, Bottom: tw.Empty},
},
}),
)
data := make([][]string, 0, len(metrics))
for _, m := range metrics {
if m.Error != nil {
data = append(data, []string{m.Name, fmt.Sprintf("Error: %v", m.Error)})
continue
}
if m.Value == (math.Round(m.Value*10) / 10) {
data = append(data, []string{m.Name, fmt.Sprintf("%.0f%s", m.Value, m.Unit)})
} else {
data = append(data, []string{m.Name, fmt.Sprintf("%.1f%s", m.Value, m.Unit)})
}
}
if err := table.Bulk(data); err != nil {
return err
}
if err := table.Render(); err != nil {
return err
}
fmt.Println(string(b))
if outOctocovPath != "" {
metricSet := &CustomMetricSet{
Key: "workspace_metrics",
Name: "Workspace metrics using [Patterner](https://github.com/tailor-platform/patterner)",
Metadata: []*MetadataKV{
{
Key: "workspace_id",
Value: cfg.WorkspaceID,
},
},
}
for _, m := range metrics {
metricSet.Metrics = append(metricSet.Metrics, &CustomMetric{
Key: m.Key,
Name: m.Name,
Value: m.Value,
Unit: m.Unit,
})
}
csets := []*CustomMetricSet{metricSet}
b, err := json.MarshalIndent(csets, "", " ")
if err != nil {
return err
}
if err := os.WriteFile(outOctocovPath, b, 0644); err != nil { //nolint:gosec
return err
}
}

return nil
},
}

func init() {
rootCmd.AddCommand(metricsCmd)
metricsCmd.Flags().StringVarP(&since, "since", "s", "30min", "only consider executions since the given duration (e.g., 24hours, 30min, 15sec)")
metricsCmd.Flags().StringVarP(&outOctocovPath, "out-octocov-path", "", "", "output the metrics in octocov custom metrics format to the specified file (e.g., ./metrics.json)")
metricsCmd.Flags().BoolVarP(&withLintWarnings, "with-lint-warnings", "", false, "display the lint warnings along with the metrics")
metricsCmd.Flags().BoolVarP(&withCoverageFullReport, "with-coverage-full-report", "", false, "display the coverage full report along with the metrics")
}

// copy from github.com/k1LoW/octocov/report
// because octocov use tablewriter v0.
type MetadataKV struct {
Key string `json:"key"`
Name string `json:"name,omitempty"`
Value string `json:"value"`
}

type CustomMetricSet struct {
Key string `json:"key"`
Name string `json:"name,omitempty"`
Metadata []*MetadataKV `json:"metadata,omitempty"`
Metrics []*CustomMetric `json:"metrics"`
}

type CustomMetric struct {
Key string `json:"key"`
Name string `json:"name,omitempty"`
Value float64 `json:"value"`
Unit string `json:"unit,omitempty"`
}
21 changes: 13 additions & 8 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,27 @@ require (
github.com/creasty/defaults v1.8.0
github.com/goccy/go-yaml v1.18.0
github.com/k1LoW/duration v1.2.0
github.com/olekukonko/tablewriter v1.0.9
github.com/spf13/cobra v1.10.1
github.com/vektah/gqlparser/v2 v2.5.30
google.golang.org/protobuf v1.36.9
)

require (
buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.36.9-20250717185734-6c6e0d3c608e.1 // indirect
github.com/fatih/color v1.7.0 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/mattn/go-colorable v0.1.2 // indirect
github.com/mattn/go-isatty v0.0.8 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/olekukonko/errors v1.1.0 // indirect
github.com/olekukonko/ll v0.0.9 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/spf13/pflag v1.0.10 // indirect
golang.org/x/net v0.42.0 // indirect
golang.org/x/sys v0.34.0 // indirect
golang.org/x/term v0.33.0 // indirect
golang.org/x/text v0.28.0 // indirect
golang.org/x/net v0.43.0 // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/term v0.34.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250826171959-ef028d996bc1 // indirect
)
Loading