Use _doc + _shard_doc as sort tiebreaker to get better performance

[LantaoJin]

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Swiddis]

Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

[LantaoJin]

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

[anasalkouz]

Can you share the performance benchmark for the 3 approaches?

[LantaoJin]

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.