[DATAFLINT-4425] - duration instrumentation on all nodes

[minskya]

Summary

Adds server-side duration attribution and topology-based stage grouping to the DataFlint Spark plugin. Duration computation and stage assignment move from the frontend (TypeScript) to the backend (Scala), operating on SparkPlanGraph + metric values from the status store — works for both live execution and history server.

• GraphDurationAttribution computes stage groups from graph topology (Exchange boundaries) and per-node exclusive durations via pipelined subtraction
• /dataflint/sqlplan and /dataflint/sqlmetrics endpoints enriched with stageGroups, nodeDurations, and exchangeDurations
• Frontend simplified from ~650 lines to ~130 lines — reads pre-computed data from backend, zero guessing
• Shared API case classes consolidated in DurationApi.scala (no more duplication between Spark 3/4 modules)
• Tests: GraphStageGroupSpec (topology) + GraphDurationAttributionSpec (durations, exchanges, normalization)

Changed files

File	Change
GraphDurationAttribution.scala	NEW — stage grouping, duration attribution, normalization, exchange durations
DurationApi.scala	NEW — shared case classes (ResolvedStageGroup, NodeDurationData, ExchangeDurationData, etc.)
PlanDurationAttribution.scala	DELETED — replaced by GraphDurationAttribution
DataflintSQLPlanPage.scala (v3/v4)	Enriched with stageGroups + nodeDurations for all SQLs
DataflintSQLMetricsPage.scala (v3/v4)	Enriched with stageGroups + nodeDurations for running SQLs
api.scala (v3/v4)	SqlEnrichedData includes stage groups + durations; duplicates moved to shared module
TimedExec.scala	Codegen sleep config for test timing
DataFlintRDDUtils.scala	Duration metric utilities
SQLNodeStageReducer.ts	Rewritten — uses backend ResolvedStageGroup for stages, backend durations for display
PlanDurationAttribution.ts	DELETED — replaced by backend
SqlReducer.ts	Stores backend data, null-safe metric handling
AppStore.ts	New types: ResolvedStageGroup, NodeDurationData, ExchangeBoundary
SqlReducerUtils.ts	Added "duration" to input/output/join/sort metric allowlists
SparkApi.tsx	Fetches SQLMetricsWithDuration, guards against empty responses
GraphStageGroupSpec.scala	NEW — 4 tests: no-shuffle, single exchange, chained exchanges, unique node IDs
GraphDurationAttributionSpec.scala	NEW — 4 tests: instrumented durations, exchange write/read, resolved groups, normalization

---

Attribution Model

Node with native exclusive metric (sortTime, aggTime):
  exclusive = metric value

Pipelined node (parent inclusive > max child inclusive):
  exclusive = inclusive - max_child_inclusive

Blocking node (Sort, HashAggregate, Python UDFs):
  exclusive = inclusive (independent timer)

Non-instrumented node inside codegen:
  exclusive = codegen pipelineTime (fallback)

Normalization:
  per-stage durations scaled to match executorRunTime

Stage Grouping

Topology-based (deterministic, no metrics needed):

1. Walk graph from root, collect nodes between Exchange boundaries
2. Each Exchange creates a new stage group
3. Chained exchanges (Exchange → Exchange) handled via recursive drill-through
4. Codegen cluster inner nodes included in their parent group
5. Resolve plan-stage IDs to Spark stage IDs via metric extraction + elimination

Exchange Duration Split

Each exchange node provides:

• writeDurationMs = shuffle write time (belongs to write stage)
• readDurationMs = fetch wait time + remote durations (belongs to read stage)

[notion-workspace]

OSS instrumentation refactoring