Skip to content

Fix whitespace after fragment specifiers in macro pretty printing#154087

Merged
rust-bors[bot] merged 1 commit intorust-lang:mainfrom
aytey:fix-fragment-specifier-whitespace
Mar 20, 2026
Merged

Fix whitespace after fragment specifiers in macro pretty printing#154087
rust-bors[bot] merged 1 commit intorust-lang:mainfrom
aytey:fix-fragment-specifier-whitespace

Conversation

@aytey
Copy link
Contributor

@aytey aytey commented Mar 19, 2026

When a macro-generating-macro captures fragment specifier tokens (like $x:ident) as tt metavariables and replays them before a keyword (like where), the pretty printer concatenates them into an invalid fragment specifier (e.g. $x:identwhere instead of $x:ident where).

This happens because tt captures preserve the original token spacing. When the fragment specifier name (e.g. ident) was originally the last token before a closing delimiter, it retains JointHidden spacing. The print_tts function only checks space_between for Spacing::Alone tokens, so JointHidden tokens skip the space check entirely, causing adjacent identifier-like tokens to merge.

The fix adds a check in print_tts to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens.

This is similar to the existing space_between mechanism that prevents token merging for Spacing::Alone tokens, extended to also handle Joint/JointHidden cases where two identifier-like tokens would merge.

Example

before (rustc 1.96.0-nightly (3b1b0ef4d 2026-03-11)):

#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }

fn main() {}

after (this branch):

#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }

fn main() {}

Notice the $x:identwhere in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as $x:ident where.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 19, 2026
@rustbot
Copy link
Collaborator

rustbot commented Mar 19, 2026

r? @mati865

rustbot has assigned @mati865.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: compiler
  • compiler expanded to 69 candidates
  • Random selection from 15 candidates

When a macro-generating-macro captures fragment specifier tokens (like
`$x:ident`) as `tt` metavariables and replays them before a keyword
(like `where`), the pretty printer concatenates them into an invalid
fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`).

This happens because `tt` captures preserve the original token spacing.
When the fragment specifier name (e.g. `ident`) was originally the last
token before a closing delimiter, it retains `JointHidden` spacing. The
`print_tts` function only checks `space_between` for `Spacing::Alone`
tokens, so `JointHidden` tokens skip the space check entirely, causing
adjacent identifier-like tokens to merge.

The fix adds a check in `print_tts` to insert a space between adjacent
identifier-like tokens (identifiers and keywords) regardless of the
original spacing, preventing them from being concatenated into invalid
tokens.

This is similar to the existing `space_between` mechanism that prevents
token merging for `Spacing::Alone` tokens, extended to also handle
`Joint`/`JointHidden` cases where two identifier-like tokens would merge.

Signed-off-by: Andrew V. Teylu <andrew.teylu@vector.com>
@aytey aytey force-pushed the fix-fragment-specifier-whitespace branch from b9f8016 to c777685 Compare March 19, 2026 10:35
@Kivooeo
Copy link
Member

Kivooeo commented Mar 19, 2026

can you show before and after please :)

@aytey
Copy link
Contributor Author

aytey commented Mar 19, 2026

can you show before and after please :)

before (rustc 1.95.0-nightly (873d4682c 2026-01-25)):

#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }

fn main() {}

after (this branch):

#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }

fn main() {}

Notice the $x:identwhere in the before.

@Kivooeo
Copy link
Member

Kivooeo commented Mar 19, 2026

yeah, new behaviour looks correct! thanks

r? me @bors r+ rollup

I also noticed that you have opened a few PRs on pretty printer today, if it's not too much trouble, could you update the description of each one to show the differences before and after? this would make the review process easier, as it would be possible to see the visual differences. I'll probably take a look at the others at a later time

@rust-bors
Copy link
Contributor

rust-bors bot commented Mar 19, 2026

📌 Commit c777685 has been approved by Kivooeo

It is now in the queue for this repository.

@rust-bors rust-bors bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 19, 2026
@aytey
Copy link
Contributor Author

aytey commented Mar 19, 2026

I also noticed that you have opened a few PRs on pretty printer today, if it's not too much trouble, could you update the description of each one to show the differences before and after?

@Kivooeo: all done!

I'll probably take a look at the others at a later time

Thanks 🫶

JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 19, 2026
…space, r=Kivooeo

Fix whitespace after fragment specifiers in macro pretty printing

When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`).

This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge.

The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens.

This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge.

## Example

**before** (`rustc 1.96.0-nightly (3b1b0ef 2026-03-11)`):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }

fn main() {}
```

**after** (this branch):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }

fn main() {}
```

Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
rust-bors bot pushed a commit that referenced this pull request Mar 19, 2026
…uwer

Rollup of 9 pull requests

Successful merges:

 - #153556 (`impl` restriction lowering)
 - #153992 (bootstrap: Optionally print a backtrace if a command fails)
 - #154019 (two smaller feature cleanups)
 - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`)
 - #154075 (Rewrite `query_ensure_result`.)
 - #154082 (Updates derive_where and removes workaround)
 - #154084 (Preserve braces around `self` in use tree pretty printing)
 - #154086 (Insert space after float literal ending with `.` in pretty printer)
 - #154087 (Fix whitespace after fragment specifiers in macro pretty printing)
rust-bors bot pushed a commit that referenced this pull request Mar 19, 2026
…uwer

Rollup of 9 pull requests

Successful merges:

 - #153556 (`impl` restriction lowering)
 - #153992 (bootstrap: Optionally print a backtrace if a command fails)
 - #154019 (two smaller feature cleanups)
 - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`)
 - #154075 (Rewrite `query_ensure_result`.)
 - #154082 (Updates derive_where and removes workaround)
 - #154084 (Preserve braces around `self` in use tree pretty printing)
 - #154086 (Insert space after float literal ending with `.` in pretty printer)
 - #154087 (Fix whitespace after fragment specifiers in macro pretty printing)
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 19, 2026
…space, r=Kivooeo

Fix whitespace after fragment specifiers in macro pretty printing

When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`).

This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge.

The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens.

This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge.

## Example

**before** (`rustc 1.96.0-nightly (3b1b0ef 2026-03-11)`):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }

fn main() {}
```

**after** (this branch):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }

fn main() {}
```

Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 19, 2026
…space, r=Kivooeo

Fix whitespace after fragment specifiers in macro pretty printing

When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`).

This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge.

The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens.

This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge.

## Example

**before** (`rustc 1.96.0-nightly (3b1b0ef 2026-03-11)`):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }

fn main() {}
```

**after** (this branch):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }

fn main() {}
```

Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
rust-bors bot pushed a commit that referenced this pull request Mar 20, 2026
…uwer

Rollup of 12 pull requests

Successful merges:

 - #152909 (sess: `-Zbranch-protection` is a target modifier)
 - #153556 (`impl` restriction lowering)
 - #154048 (Don't emit rustdoc `missing_doc_code_examples` lint on impl items)
 - #153992 (bootstrap: Optionally print a backtrace if a command fails)
 - #154019 (two smaller feature cleanups)
 - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`)
 - #154075 (Rewrite `query_ensure_result`.)
 - #154082 (Updates derive_where and removes workaround)
 - #154084 (Preserve braces around `self` in use tree pretty printing)
 - #154086 (Insert space after float literal ending with `.` in pretty printer)
 - #154087 (Fix whitespace after fragment specifiers in macro pretty printing)
 - #154109 (tests: Add regression test for async closures involving HRTBs)
rust-bors bot pushed a commit that referenced this pull request Mar 20, 2026
Rollup of 15 pull requests

Successful merges:

 - #152909 (sess: `-Zbranch-protection` is a target modifier)
 - #153556 (`impl` restriction lowering)
 - #154048 (Don't emit rustdoc `missing_doc_code_examples` lint on impl items)
 - #150935 (Introduce #[diagnostic::on_move(message)])
 - #152973 (remove -Csoft-float)
 - #153862 (Rename `cycle_check` to `find_cycle`)
 - #153992 (bootstrap: Optionally print a backtrace if a command fails)
 - #154019 (two smaller feature cleanups)
 - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`)
 - #154075 (Rewrite `query_ensure_result`.)
 - #154082 (Updates derive_where and removes workaround)
 - #154084 (Preserve braces around `self` in use tree pretty printing)
 - #154086 (Insert space after float literal ending with `.` in pretty printer)
 - #154087 (Fix whitespace after fragment specifiers in macro pretty printing)
 - #154109 (tests: Add regression test for async closures involving HRTBs)
@rust-bors rust-bors bot merged commit a00818d into rust-lang:main Mar 20, 2026
11 checks passed
@rustbot rustbot added this to the 1.96.0 milestone Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants