Fix whitespace after fragment specifiers in macro pretty printing#154087
Fix whitespace after fragment specifiers in macro pretty printing#154087rust-bors[bot] merged 1 commit intorust-lang:mainfrom
Conversation
|
r? @mati865 rustbot has assigned @mati865. Use Why was this reviewer chosen?The reviewer was selected based on:
|
When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`). This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge. The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens. This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge. Signed-off-by: Andrew V. Teylu <andrew.teylu@vector.com>
b9f8016 to
c777685
Compare
|
can you show before and after please :) |
before ( #![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
macro_rules! outer {
($d:tt $($params:tt)*) =>
{
#[macro_export] macro_rules! inner
{ ($($params)* where $d($rest:tt)*) => {}; }
};
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }
fn main() {}after (this branch): #![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
macro_rules! outer {
($d:tt $($params:tt)*) =>
{
#[macro_export] macro_rules! inner
{ ($($params)* where $d($rest:tt)*) => {}; }
};
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }
fn main() {}Notice the |
|
yeah, new behaviour looks correct! thanks r? me @bors r+ rollup I also noticed that you have opened a few PRs on pretty printer today, if it's not too much trouble, could you update the description of each one to show the differences before and after? this would make the review process easier, as it would be possible to see the visual differences. I'll probably take a look at the others at a later time |
@Kivooeo: all done!
Thanks 🫶 |
…space, r=Kivooeo Fix whitespace after fragment specifiers in macro pretty printing When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`). This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge. The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens. This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge. ## Example **before** (`rustc 1.96.0-nightly (3b1b0ef 2026-03-11)`): ```rust #![feature(prelude_import)] #![no_std] extern crate std; #[prelude_import] use ::std::prelude::rust_2015::*; //@ pretty-mode:expanded //@ pp-exact:macro-fragment-specifier-whitespace.pp // Test that fragment specifier names in macro definitions are properly // separated from the following keyword/identifier token when pretty-printed. // This is a regression test for a bug where `$x:ident` followed by `where` // was pretty-printed as `$x:identwhere` (an invalid fragment specifier). macro_rules! outer { ($d:tt $($params:tt)*) => { #[macro_export] macro_rules! inner { ($($params)* where $d($rest:tt)*) => {}; } }; } #[macro_export] macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; } fn main() {} ``` **after** (this branch): ```rust #![feature(prelude_import)] #![no_std] extern crate std; #[prelude_import] use ::std::prelude::rust_2015::*; //@ pretty-mode:expanded //@ pp-exact:macro-fragment-specifier-whitespace.pp // Test that fragment specifier names in macro definitions are properly // separated from the following keyword/identifier token when pretty-printed. // This is a regression test for a bug where `$x:ident` followed by `where` // was pretty-printed as `$x:identwhere` (an invalid fragment specifier). macro_rules! outer { ($d:tt $($params:tt)*) => { #[macro_export] macro_rules! inner { ($($params)* where $d($rest:tt)*) => {}; } }; } #[macro_export] macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; } fn main() {} ``` Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
…uwer Rollup of 9 pull requests Successful merges: - #153556 (`impl` restriction lowering) - #153992 (bootstrap: Optionally print a backtrace if a command fails) - #154019 (two smaller feature cleanups) - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`) - #154075 (Rewrite `query_ensure_result`.) - #154082 (Updates derive_where and removes workaround) - #154084 (Preserve braces around `self` in use tree pretty printing) - #154086 (Insert space after float literal ending with `.` in pretty printer) - #154087 (Fix whitespace after fragment specifiers in macro pretty printing)
…uwer Rollup of 9 pull requests Successful merges: - #153556 (`impl` restriction lowering) - #153992 (bootstrap: Optionally print a backtrace if a command fails) - #154019 (two smaller feature cleanups) - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`) - #154075 (Rewrite `query_ensure_result`.) - #154082 (Updates derive_where and removes workaround) - #154084 (Preserve braces around `self` in use tree pretty printing) - #154086 (Insert space after float literal ending with `.` in pretty printer) - #154087 (Fix whitespace after fragment specifiers in macro pretty printing)
…space, r=Kivooeo Fix whitespace after fragment specifiers in macro pretty printing When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`). This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge. The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens. This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge. ## Example **before** (`rustc 1.96.0-nightly (3b1b0ef 2026-03-11)`): ```rust #![feature(prelude_import)] #![no_std] extern crate std; #[prelude_import] use ::std::prelude::rust_2015::*; //@ pretty-mode:expanded //@ pp-exact:macro-fragment-specifier-whitespace.pp // Test that fragment specifier names in macro definitions are properly // separated from the following keyword/identifier token when pretty-printed. // This is a regression test for a bug where `$x:ident` followed by `where` // was pretty-printed as `$x:identwhere` (an invalid fragment specifier). macro_rules! outer { ($d:tt $($params:tt)*) => { #[macro_export] macro_rules! inner { ($($params)* where $d($rest:tt)*) => {}; } }; } #[macro_export] macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; } fn main() {} ``` **after** (this branch): ```rust #![feature(prelude_import)] #![no_std] extern crate std; #[prelude_import] use ::std::prelude::rust_2015::*; //@ pretty-mode:expanded //@ pp-exact:macro-fragment-specifier-whitespace.pp // Test that fragment specifier names in macro definitions are properly // separated from the following keyword/identifier token when pretty-printed. // This is a regression test for a bug where `$x:ident` followed by `where` // was pretty-printed as `$x:identwhere` (an invalid fragment specifier). macro_rules! outer { ($d:tt $($params:tt)*) => { #[macro_export] macro_rules! inner { ($($params)* where $d($rest:tt)*) => {}; } }; } #[macro_export] macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; } fn main() {} ``` Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
…space, r=Kivooeo Fix whitespace after fragment specifiers in macro pretty printing When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`). This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge. The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens. This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge. ## Example **before** (`rustc 1.96.0-nightly (3b1b0ef 2026-03-11)`): ```rust #![feature(prelude_import)] #![no_std] extern crate std; #[prelude_import] use ::std::prelude::rust_2015::*; //@ pretty-mode:expanded //@ pp-exact:macro-fragment-specifier-whitespace.pp // Test that fragment specifier names in macro definitions are properly // separated from the following keyword/identifier token when pretty-printed. // This is a regression test for a bug where `$x:ident` followed by `where` // was pretty-printed as `$x:identwhere` (an invalid fragment specifier). macro_rules! outer { ($d:tt $($params:tt)*) => { #[macro_export] macro_rules! inner { ($($params)* where $d($rest:tt)*) => {}; } }; } #[macro_export] macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; } fn main() {} ``` **after** (this branch): ```rust #![feature(prelude_import)] #![no_std] extern crate std; #[prelude_import] use ::std::prelude::rust_2015::*; //@ pretty-mode:expanded //@ pp-exact:macro-fragment-specifier-whitespace.pp // Test that fragment specifier names in macro definitions are properly // separated from the following keyword/identifier token when pretty-printed. // This is a regression test for a bug where `$x:ident` followed by `where` // was pretty-printed as `$x:identwhere` (an invalid fragment specifier). macro_rules! outer { ($d:tt $($params:tt)*) => { #[macro_export] macro_rules! inner { ($($params)* where $d($rest:tt)*) => {}; } }; } #[macro_export] macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; } fn main() {} ``` Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
…uwer Rollup of 12 pull requests Successful merges: - #152909 (sess: `-Zbranch-protection` is a target modifier) - #153556 (`impl` restriction lowering) - #154048 (Don't emit rustdoc `missing_doc_code_examples` lint on impl items) - #153992 (bootstrap: Optionally print a backtrace if a command fails) - #154019 (two smaller feature cleanups) - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`) - #154075 (Rewrite `query_ensure_result`.) - #154082 (Updates derive_where and removes workaround) - #154084 (Preserve braces around `self` in use tree pretty printing) - #154086 (Insert space after float literal ending with `.` in pretty printer) - #154087 (Fix whitespace after fragment specifiers in macro pretty printing) - #154109 (tests: Add regression test for async closures involving HRTBs)
Rollup of 15 pull requests Successful merges: - #152909 (sess: `-Zbranch-protection` is a target modifier) - #153556 (`impl` restriction lowering) - #154048 (Don't emit rustdoc `missing_doc_code_examples` lint on impl items) - #150935 (Introduce #[diagnostic::on_move(message)]) - #152973 (remove -Csoft-float) - #153862 (Rename `cycle_check` to `find_cycle`) - #153992 (bootstrap: Optionally print a backtrace if a command fails) - #154019 (two smaller feature cleanups) - #154059 (tests: Activate `must_not_suspend` test for `MutexGuard` dropped before `await`) - #154075 (Rewrite `query_ensure_result`.) - #154082 (Updates derive_where and removes workaround) - #154084 (Preserve braces around `self` in use tree pretty printing) - #154086 (Insert space after float literal ending with `.` in pretty printer) - #154087 (Fix whitespace after fragment specifiers in macro pretty printing) - #154109 (tests: Add regression test for async closures involving HRTBs)
When a macro-generating-macro captures fragment specifier tokens (like
$x:ident) asttmetavariables and replays them before a keyword (likewhere), the pretty printer concatenates them into an invalid fragment specifier (e.g.$x:identwhereinstead of$x:ident where).This happens because
ttcaptures preserve the original token spacing. When the fragment specifier name (e.g.ident) was originally the last token before a closing delimiter, it retainsJointHiddenspacing. Theprint_ttsfunction only checksspace_betweenforSpacing::Alonetokens, soJointHiddentokens skip the space check entirely, causing adjacent identifier-like tokens to merge.The fix adds a check in
print_ttsto insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens.This is similar to the existing
space_betweenmechanism that prevents token merging forSpacing::Alonetokens, extended to also handleJoint/JointHiddencases where two identifier-like tokens would merge.Example
before (
rustc 1.96.0-nightly (3b1b0ef4d 2026-03-11)):after (this branch):
Notice the
$x:identwherein the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as$x:ident where.