Regex auto-atomicity too aggressive about \w and \b

[mathy-plutoflume]

Description

System.Text.RegularExpressions.Regex.Matches not returning expected matches. It returns no matches at all.

Reproduction Steps

var stringToMatch = "sydney bogota berlin tokyo nairobi denver rio";
Console.WriteLine($"Input: {stringToMatch}");

GetMatches(@"(\b(?!bogo|nai)\w*\b)\w+", stringToMatch);
GetMatches(@"\w*\b\w+", stringToMatch);

void GetMatches(string expression, string stringToMatch)
{
    var reg = new System.Text.RegularExpressions.Regex(expression);

    Console.WriteLine($"\nOutput:");
    foreach (var match in reg.Matches(stringToMatch))
    {
        Console.WriteLine(match);
    }
}

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net6.0</TargetFrameworks>
    <ImplicitUsings>enable</ImplicitUsings>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>
</Project>

Expected behavior

Input: sydney bogota berlin tokyo nairobi denver rio

Output:
sydney
berlin
tokyo
denver
rio

Output:
sydney
bogota
berlin
tokyo
nairobi
denver
rio

Example 1: https://regex101.com/r/Xn7X3w/2
Example 2: https://regex101.com/r/Q10441/1

Actual behavior

Input: sydney bogota berlin tokyo nairobi denver rio

Output:

Output:

Regression?

When running the same code pointing to netstandard2.0 and net472 it's returning correct matches.

Whereas .net5 and .net6.0 returns no match at all.

Known Workarounds

None

Configuration

• .NET 6.0
• Windows 11
• x64

Other information

Other community members in Stackoverflow were also able to reproduce this issue: https://stackoverflow.com/questions/73498919/regex-match-differs-between-net-versions

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

System.Text.RegularExpressions.Regex.Matches not returning expected matches. It returns no matches at all.

Reproduction Steps

var stringToMatch = "sydney bogota berlin tokyo nairobi denver rio";
Console.WriteLine($"Input: {stringToMatch}");

GetMatches(@"(\b(?!bogo|nai)\w*\b)\w+", stringToMatch);
GetMatches(@"\w*\b\w+", stringToMatch);

void GetMatches(string expression, string stringToMatch)
{
    var reg = new System.Text.RegularExpressions.Regex(expression);

    Console.WriteLine($"Output:");
    foreach (var match in reg.Matches(stringToMatch))
    {
        Console.WriteLine(match);
    }
}

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net6.0</TargetFrameworks>
    <ImplicitUsings>enable</ImplicitUsings>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>
</Project>

Expected behavior

Input: sydney bogota berlin tokyo nairobi denver rio

Output:
sydney
berlin
tokyo
denver
rio

Output:
sydney
bogota
berlin
tokyo
nairobi
denver
rio

Example 1: https://regex101.com/r/Xn7X3w/2
Example 2: https://regex101.com/r/Q10441/1

Actual behavior

Input: sydney bogota berlin tokyo nairobi denver rio

Output:

Output:

Regression?

When running the same code pointing to netstandard2.0 and net472 it's returning correct matches.

Whereas .net5 and .net6.0 returns to match at all.

Known Workarounds

No response

Configuration

• .NET 6.0
• Windows 11
• x64

Other information

Other community members in Stackoverflow were also able to reproduce this issue: https://stackoverflow.com/questions/73498919/regex-match-differs-between-net-versions

Author:	mathy-plutoflume
Assignees:	-
Labels:	area-System.Text.RegularExpressions
Milestone:	-

[stephentoub]

Looks like the issue is that we're making the starting loop atomic, and we shouldn't be, as doing so visibly affects the behavior. The \b can be used to make the first loop atomic, but it's zero-width, so we also need to evaluate against what comes after it; we might be able to get away with only doing so when the first loop has a non-0 minimum iteration count, but it's likely also diminishing returns at that point.