Dart code breaks editor

[roblourens]

• Install Dart extension
• Using OSS with current main
• Open a .dart file
• Paste this

int calculate() {
  void main() {
  var myMultiline = '''
First
Second
Third
''';
  print(myMultiline);
}
  
  return 6 * 7;
}

It's invalid but I was in the middle of typing

• Try to move the cursor with the arrow keys or do other things in the editor
• The cursor doesn't move, I can't interact with the editor
• Lots of these errors

2025-03-21 09:28:06.164 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at NonPeekableTextBufferTokenizer.read (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:195:63)
    at TextBufferTokenizer.peek (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:79:39)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:59:46)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
2025-03-21 09:28:06.203 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at TokenizationTextModelPart.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:205:33)
    at TextModel.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/textModel.js:1534:48)
    at InlayHintsFragments._getRangeAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:116:28)
    at new InlayHintsFragments (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:96:55)
    at InlayHintsFragments.create (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:82:16)
    at async RunOnceScheduler.runner (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHintsController.js:189:36)
2025-03-21 09:28:06.205 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at NonPeekableTextBufferTokenizer.read (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:195:63)
    at TextBufferTokenizer.peek (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:79:39)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:59:46)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
2025-03-21 09:28:06.244 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at TokenizationTextModelPart.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:205:33)
    at TextModel.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/textModel.js:1534:48)
    at InlayHintsFragments._getRangeAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:116:28)
    at new InlayHintsFragments (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:96:55)
    at InlayHintsFragments.create (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:82:16)
    at async RunOnceScheduler.runner (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHintsController.js:189:36)
2025-03-21 09:28:06.465 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at NonPeekableTextBufferTokenizer.read (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:195:63)
    at TextBufferTokenizer.peek (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:79:39)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:59:46)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
2025-03-21 09:28:06.505 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at TokenizationTextModelPart.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:205:33)
    at TextModel.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/textModel.js:1534:48)
    at InlayHintsFragments._getRangeAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:116:28)
    at new InlayHintsFragments (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:96:55)
    at InlayHintsFragments.create (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:82:16)
    at async RunOnceScheduler.runner (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHintsController.js:189:36)
2025-03-21 09:28:06.514 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at NonPeekableTextBufferTokenizer.read (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:195:63)
    at TextBufferTokenizer.peek (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:79:39)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:59:46)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
2025-03-21 09:28:06.555 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at TokenizationTextModelPart.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:205:33)
    at TextModel.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/textModel.js:1534:48)
    at InlayHintsFragments._getRangeAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:116:28)
    at new InlayHintsFragments (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:96:55)
    at InlayHintsFragments.create (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:82:16)
    at async RunOnceScheduler.runner (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHintsController.js:189:36)
2025-03-21 09:28:07.148 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at NonPeekableTextBufferTokenizer.read (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:195:63)
    at TextBufferTokenizer.peek (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/tokenizer.js:79:39)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:59:46)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
    at Parser.parseChild (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:114:36)
    at Parser.parseList (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/parser.js:65:30)
2025-03-21 09:28:07.187 [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!
    at new LineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/lineTokens.js:64:19)
    at SparseTokensStore.addSparseTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/tokens/sparseTokensStore.js:167:16)
    at TokenizationTextModelPart.getLineTokens (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:124:37)
    at TokenizationTextModelPart.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/tokenizationTextModelPart.js:205:33)
    at TextModel.getWordAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/common/model/textModel.js:1534:48)
    at InlayHintsFragments._getRangeAtPosition (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:116:28)
    at new InlayHintsFragments (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:96:55)
    at InlayHintsFragments.create (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHints.js:82:16)
    at async RunOnceScheduler.runner (vscode-file://vscode-app/Users/roblou/code/vscode/out/vs/editor/contrib/inlayHints/browser/inlayHintsController.js:189:36)

[roblourens]

Untitled.mov

When it pauses, I'm using the arrow keys, to no effect

[roblourens]

Reproes in Insiders, not stable

[ChrisKader]

Same as #243923

[peku33]

I got the same for python & rust in insiders dc289883be5d37d5d2b2f7d30926aa42a3123437, I don't have anything dart-related installed.

I came to this message because interface freezes and drops pressed keys from time to time. I got the same set of messages in the UI: [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!

[alexr00]

I'm not able to repro the error, but I'm removing the throw here.

[hediet]

I'm not able to repro the error, but I'm removing the throw here.

This should revert any changes we did this iteration. Moving this issue to next iteration, as we know something is off, but it is not a recent regression.