[CALCITE-7465] Make MATCH_RECOGNIZE tolerant to FINAL and RUNNING non function MEASURES

[snuyanzin]

Jira Link

CALCITE-7465

Current issue is that some of MEASURES

1. might be without FINAL or RUNNING keyword
2. might be with a keyword however the computed one is different.
   In both cases unparsed sql is wrong.

Since calculation is happenning during validation it impacts unparse logic,
The proposal is movement calculation to SqlToRelConverter then unparse will work as expected

query for the first case

SELECT *
FROM emp
MATCH_RECOGNIZE (
  MEASURES
     FINAL COUNT(A.deptno) AS deptno,
     A.ename AS ename
  PATTERN (A B)
  DEFINE
    A AS A.empno = 123
) AS T

if not all rows then the default operation will be FINAL and it will be unparsed (non parsable anymore) to

SELECT *
FROM `EMP` MATCH_RECOGNIZE(
MEASURES FINAL COUNT(`A`.`DEPTNO`) AS `DEPTNO`, FINAL `A`.`ENAME` AS `ENAME`
PATTERN (`A` `B`)
DEFINE `A` AS (PREV(`A`.`EMPNO`, 0) = 123)) AS `T`

for the second case

SELECT *
  from "product" match_recognize
  (
   measures STRT."net_weight" as start_nw,
   FINAL COUNT("net_weight") as down_cnt,
   RUNNING COUNT("net_weight") as running_cnt
    pattern (strt down+ up+)
    define
      down as down."net_weight" < PREV(down."net_weight"),
      up as up."net_weight" > prev(up."net_weight")
  ) mr

here for one of the measures it is RUNNING however it is not ALL ROWS then will be replaced with FINAL
the problem is that it is unparsed as

SELECT *
FROM (SELECT *
FROM "foodmart"."product") 
MATCH_RECOGNIZE(
MEASURES 
FINAL "STRT"."net_weight" AS "START_NW",
FINAL COUNT("product"."net_weight") AS "DOWN_CNT", 
FINAL (RUNNING COUNT("product"."net_weight")) AS "RUNNING_CNT"
ONE ROW PER MATCH
AFTER MATCH SKIP TO NEXT ROW
PATTERN ("STRT" "DOWN" + "UP" +)
DEFINE 
"DOWN" AS PREV("DOWN"."net_weight", 0) < 
PREV("DOWN"."net_weight", 1), 
"UP" AS PREV("UP"."net_weight", 0) > 
PREV("UP"."net_weight", 1))

i.e. there is FINAL and RUNNING together for the same measure which makes the query unparsable.

[snuyanzin]

old test producing unparsable query

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[snuyanzin]

The PR name is the subject to be changed once we align on the approach in jira

[mihaibudiu]

Is there a spec of the grammar for this construct somewhere?
Ideally we would follow that.
I don't know enough about this construct to make a choice myself.

[snuyanzin]

Didn't get if your question is about Expression3 then https://github.com/snuyanzin/calcite/blob/dd71d5b3aa0bd004f4d5369986e50301b628b0ad/core/src/main/codegen/templates/Parser.jj#L4045-L4049

if your question is about MATCH_RECOGNIZE then I have only links to other vendors
like Snowflake https://docs.snowflake.com/en/sql-reference/constructs/match_recognize#syntax

    [ MEASURES <expr> [AS] <alias> [, ... ] ]

and below they have explanation for FINAL, RUNNING (Snowlake doesn't allow FINAL, RUNNING for non function, however Calcite always sets either FINAL or RUNNING and unparses it like that)

expr ::= ... [ { RUNNING | FINAL } ] windowFunction ...

BigQuery seems doesn't have FINAL, RUNNING at all https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#match_recognize_clause

Oracle doesn't use these keyword and determines FINAL or RUNNING implicitly https://docs.oracle.com/cd/E29542_01/apirefs.1111/e12048/pattern_recog.htm#BEJBFEGJ

[mihaibudiu]

Ok, so there isn't a universally agreed standard for MATCH RECOGNIZE.
It's fine for the parser to be more permissive and the validator to reject some things later (e.g., if you want to emulate restrictions of a specific dialect).

[mihaibudiu]

But is Expression3 too permissive?
Can you make up some expressions that would be accepted which are not legal?

[mihaibudiu]

Otherwise this approach is nice, because it is minimally invasive.

[snuyanzin]

I spotted another case where this PR will help #4907
I tried other case where expression under FIRST, LAST, NEXT , PREV might be expanded however parser after that not able to parse it.
After changing to expression parser started to parse such unparsed queries, example in linked PR and related jira

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud