Skip to content

Conversation

@firewave
Copy link
Collaborator

@firewave firewave commented Dec 4, 2023

Each Token (should) be connected to a TokenList. Tokenizer just encapsulates that so we have no need to check the Tokenizer but can simply ask the Token.

Also if we have function calls we pass in a flag to tell it if it is C/C++ we can get rid of that flag and simply ask the Token.

@firewave
Copy link
Collaborator Author

firewave commented Dec 4, 2023

I have the same changes for isCpp() and that should help with things as we still have code which assumes that it is C++ since we do not have access to a Tokenizer object.

This should also allow us to get rid of some references being passed around - maybe even getting rid of one of the various isCpp() methods. It's really awkward we can access the same object via different accessors - it makes things just more complex.

IMO we should also be passing the TokenList around instead of the Tokenizer if possible.

@firewave firewave marked this pull request as draft December 4, 2023 23:27
@firewave firewave force-pushed the tokenizer-c branch 3 times, most recently from 05b773c to 2c9b09a Compare January 9, 2024 11:59
@firewave
Copy link
Collaborator Author

Requires #5853 to be merged first.

@firewave firewave marked this pull request as ready for review February 27, 2024 17:59
@firewave
Copy link
Collaborator Author

firewave commented Mar 5, 2024

Almost a whole week (actually three months including the draft period) without any feedback - merging.

@firewave firewave merged commit 14833a4 into danmar:main Mar 5, 2024
@firewave firewave deleted the tokenizer-c branch March 5, 2024 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant