planner: fix wrong TopN's ByItem with expression.ScalarFunction when to PushDownTopN

[hawkingrei]

What problem does this PR solve?

Issue Number: close #60655

Problem Summary:

In the #40593,

There is a column in topN.ByItems is generated by proj, when topN is pushed down below proj, topN cannot obtain the column from datasource, to fix this, we can check whether topN.ByItems contains a column(with ID=0) generated by proj, if so, proj will prevent the optimizer from pushing topN down.

1. A column with ID=0 indicates that the column cannot be resolved by data source.
2. To determine whether a column is generated by projection, we can check whether the child of proj contains the column.

What changed and how does it work?

If you find that the TopN you can push down cannot actually be pushed down, you should use the original TopN to assemble into a structure of TopN -> Projection, rather than using the processed TopN to assemble.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

planner: fix wrong TopN's ByItem with expression.ScalarFunction when to PushDownTopN

修复在 TopN 下推过程中错误生成带表达式的TopN排序项 

[hawkingrei]

/check-issue-triage-complete

[hawkingrei]

/test fast_tiprow_test

[tiprow]

@ti-chi-bot[bot]: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test fast_test_tiprow
• /test tidb_parser_test

Use /test all to run the following jobs that were automatically triggered:

• tidb_parser_test

Details

In response to this:

@hawkingrei: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-integration-ddl-test

/test pull-integration-e2e-test

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-unit-test-ddlv1

/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test

/test pull-common-test

/test pull-e2e-test

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-next-gen-real-tikv-test

/test pull-sqllogic-test

/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_integration_ddl_test

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_mysql_client_test

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.5929%. Comparing base (a0293bf) to head (ecb5e20).
Report is 11 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #60822        +/-   ##
================================================
+ Coverage   73.1429%   73.5929%   +0.4499%     
================================================
  Files          1722       1722                
  Lines        476948     481220      +4272     
================================================
+ Hits         348854     354144      +5290     
+ Misses       106703     105535      -1168     
- Partials      21391      21541       +150     

Flag	Coverage Δ
integration	42.6384% <63.6363%> (?)
unit	72.3697% <100.0000%> (+0.0125%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.7804% <ø> (ø)
parser	∅ <ø> (∅)
br	46.4900% <ø> (-0.9561%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hawkingrei]

/retest

[Copilot]

Pull Request Overview

This PR addresses a bug that causes the wrong schema to be used when pushing down TopN into a projection. Key changes include:

• Removing an extra blank line in optimizer.go.
• Adding a new unit test to verify the TopN push down behavior in projection.
• Updating the push down logic in logical_projection.go with an additional check using a new helper method (isSetTopNChild).

Reviewed Changes

Copilot reviewed 3 out of 6 changed files in this pull request and generated no comments.

File	Description
pkg/planner/core/optimizer.go	Removed an unnecessary blank line to improve code clarity.
pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go	Added unit tests that verify the correct behavior for projection push down TopN.
pkg/planner/core/operator/logicalop/logical_projection.go	Updated TopN push down logic and added the helper method isSetTopNChild to better determine if the TopN child is set appropriately.

Files not reviewed (3)

• pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel: Language not supported
• tests/integrationtest/r/planner/core/plan.result: Language not supported
• tests/integrationtest/t/planner/core/plan.test: Language not supported

Comments suppressed due to low confidence (1)

pkg/planner/core/operator/logicalop/logical_projection.go:228

• [nitpick] The helper function name 'isSetTopNChild' is ambiguous. Consider renaming it to something like 'areTopNByItemsInChildSchema' so that its purpose is clearer.

if !p.Children()[0].Schema().Contains(col) {

[Copilot]

Pull Request Overview

This PR fixes an issue with the TopN push down logic in the planner when dealing with projections that generate columns with ID=0. The changes update the push-down behavior in LogicalProjection and add new unit tests to verify the correct operator tree under different configuration scenarios.

• Update LogicalProjection.PushDownTopN to create a TopN operator without by-items when a projection-generated column is detected.
• Add unit tests in logical_operator_test to ensure the correct handling of TopN push down.

Reviewed Changes

Copilot reviewed 2 out of 5 changed files in this pull request and generated no comments.

File	Description
pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go	Adds tests validating the TopN push down logic with projections.
pkg/planner/core/operator/logicalop/logical_projection.go	Adjusts the push-down logic to generate a new TopN operator when the child schema is missing a projection-generated column.

Files not reviewed (3)

• pkg/planner/core/operator/logicalop/logicalop_test/BUILD.bazel: Language not supported
• tests/integrationtest/r/planner/core/plan.result: Language not supported
• tests/integrationtest/t/planner/core/plan.test: Language not supported

Comments suppressed due to low confidence (2)

pkg/planner/core/operator/logicalop/logical_projection.go:228

• The condition checking for a projection-generated column (col.ID == 0) may inadvertently trigger the creation and push down of a TopN operator without by-items even for cases that are not truly projection-related. Consider refining this condition with additional checks to ensure that only columns generated by projection result in this behavior.

if !p.Children()[0].Schema().Contains(col) {

pkg/planner/core/operator/logicalop/logicalop_test/logical_operator_test.go:128

• [nitpick] Consider adding more granular assertions on the plan structure rather than relying on exact string matching, to reduce brittleness if plan formatting changes.

AS s 

[hawkingrei]

/retest

[hawkingrei]

/retest

[hawkingrei]

/retest

[hawkingrei]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [AilinKid,winoros]
• pkg/planner/OWNERS [AilinKid,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-05-07 07:21:07.397709417 +0000 UTC m=+426114.580612676: ☑️ agreed by AilinKid.
• 2025-05-07 16:40:32.145241422 +0000 UTC m=+459679.328144678: ☑️ agreed by winoros.

[hawkingrei]

/cherrypick release-8.5
/cherrypick release-8.1
/cherrypick release-7.5

[ti-chi-bot]

@hawkingrei: new pull request created to branch release-7.5: #61004.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherrypick release-8.5
/cherrypick release-8.1
/cherrypick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@hawkingrei: new pull request created to branch release-8.1: #61005.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherrypick release-8.5
/cherrypick release-8.1
/cherrypick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@hawkingrei: new pull request created to branch release-8.5: #61006.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherrypick release-8.5
/cherrypick release-8.1
/cherrypick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.