Keywords out of sync with parser

[dveeden]

Error Report

File: keywords.md

Keywords like STATS_EXTENDED are missing while they are defined in https://github.com/pingcap/tidb/blob/master/parser/parser.y

#!/bin/python3
import re
import sys
from pathlib import Path

parser=Path('../tidb/parser/parser.y')
if not parser.exists():
    sys.exit(f"{parser} doesn't exist")

kwdocs=Path('keywords.md')
if not parser.exists():
    sys.exit(f"{kwdocs} doesn't exist")

keywords = kwdocs.read_text()

section = "Unknown"
for line in parser.read_text().split('\n'):
    if line.find('The following tokens belong to ReservedKeyword') >= 0:
        section='ReservedKeyword'

    if line.find('The following tokens belong to UnReservedKeyword') >= 0:
        section='UnReservedKeyword'

    if line.find('The following tokens belong to NotKeywordToken') >= 0:
        section='NotKeywordToken'

    if section == 'ReservedKeyword':
        if m := re.match(r'^\t\w+\s+"(\w+)"$', line):
            kw = m.groups()[0]
            if not (kwm := re.search(f'^- {kw} \(R\)$', keywords, re.MULTILINE)):
                print(f'Missing docs for reserved keyword: {kw}')

    if section == 'UnReservedKeyword':
        if m := re.match(r'^\t\w+\s+"(\w+)"$', line):
            kw = m.groups()[0]
            if not (kwm := re.search(f'^- {kw}$', keywords, re.MULTILINE)):
                print(f'Missing docs for non-reserved keyword: {kw}')

$ ./scripts/check-keywords.sh | sort
Missing docs for non-reserved keyword: ATTRIBUTE
Missing docs for non-reserved keyword: ATTRIBUTES
Missing docs for non-reserved keyword: BERNOULLI
Missing docs for non-reserved keyword: BINDING_CACHE
Missing docs for non-reserved keyword: CALIBRATE
Missing docs for non-reserved keyword: CAUSAL
Missing docs for non-reserved keyword: CLIENT_ERRORS_SUMMARY
Missing docs for non-reserved keyword: CLUSTER
Missing docs for non-reserved keyword: CLUSTERED
Missing docs for non-reserved keyword: CONSISTENCY
Missing docs for non-reserved keyword: DIGEST
Missing docs for non-reserved keyword: DISABLED
Missing docs for non-reserved keyword: ENABLED
Missing docs for non-reserved keyword: FAILED_LOGIN_ATTEMPTS
Missing docs for non-reserved keyword: HELP
Missing docs for non-reserved keyword: HISTOGRAM
Missing docs for non-reserved keyword: LOCKED
Missing docs for non-reserved keyword: MEMBER
Missing docs for non-reserved keyword: NONCLUSTERED
Missing docs for non-reserved keyword: OFF
Missing docs for non-reserved keyword: OLTP_READ_ONLY
Missing docs for non-reserved keyword: OLTP_READ_WRITE
Missing docs for non-reserved keyword: OLTP_WRITE_ONLY
Missing docs for non-reserved keyword: OPTIONAL
Missing docs for non-reserved keyword: PASSWORD_LOCK_TIME
Missing docs for non-reserved keyword: PAUSE
Missing docs for non-reserved keyword: PERCENT
Missing docs for non-reserved keyword: POINT
Missing docs for non-reserved keyword: POLICY
Missing docs for non-reserved keyword: PRESERVE
Missing docs for non-reserved keyword: PROXY
Missing docs for non-reserved keyword: PURGE
Missing docs for non-reserved keyword: REPLICAS
Missing docs for non-reserved keyword: REQUIRED
Missing docs for non-reserved keyword: RESOURCE
Missing docs for non-reserved keyword: RESTART
Missing docs for non-reserved keyword: RESUME
Missing docs for non-reserved keyword: REUSE
Missing docs for non-reserved keyword: SAN
Missing docs for non-reserved keyword: SAVEPOINT
Missing docs for non-reserved keyword: SKIP
Missing docs for non-reserved keyword: STATS_COL_CHOICE
Missing docs for non-reserved keyword: STATS_COL_LIST
Missing docs for non-reserved keyword: STATS_OPTIONS
Missing docs for non-reserved keyword: STATS_SAMPLE_RATE
Missing docs for non-reserved keyword: SYSTEM
Missing docs for non-reserved keyword: TPCC
Missing docs for non-reserved keyword: TTL
Missing docs for non-reserved keyword: TTL_ENABLE
Missing docs for non-reserved keyword: TTL_JOB_INTERVAL
Missing docs for non-reserved keyword: WAIT
Missing docs for non-reserved keyword: WORKLOAD
Missing docs for reserved keyword: ARRAY
Missing docs for reserved keyword: CALL
Missing docs for reserved keyword: CUME_DIST
Missing docs for reserved keyword: DENSE_RANK
Missing docs for reserved keyword: FETCH
Missing docs for reserved keyword: FIRST_VALUE
Missing docs for reserved keyword: GROUPS
Missing docs for reserved keyword: ILIKE
Missing docs for reserved keyword: INTERSECT
Missing docs for reserved keyword: LAG
Missing docs for reserved keyword: LAST_VALUE
Missing docs for reserved keyword: LEAD
Missing docs for reserved keyword: NTH_VALUE
Missing docs for reserved keyword: NTILE
Missing docs for reserved keyword: OF
Missing docs for reserved keyword: OVER
Missing docs for reserved keyword: PERCENT_RANK
Missing docs for reserved keyword: RANK
Missing docs for reserved keyword: RECURSIVE
Missing docs for reserved keyword: ROW_NUMBER
Missing docs for reserved keyword: ROWS
Missing docs for reserved keyword: STATS_EXTENDED
Missing docs for reserved keyword: TABLESAMPLE
Missing docs for reserved keyword: TiDB_CURRENT_TSO
Missing docs for reserved keyword: WINDOW

[qiancai]

Hi @dveeden Thanks for the amazing work👍 The script will definitely save us a lot of effort in locating these missing keywords. Perhaps, would you please create a PR to add the script to https://github.com/pingcap/docs/tree/master/scripts so we can actively maintain and use it?

[dveeden]

Added the PR with updates for the docs and the script. My initial script incorrectly flagged some keywords that were marked with '(R-window)' as missing, so the list is a bit shorter than listed in the description.