gh-74902: Add Unicode Grapheme Cluster Break algorithm #143076
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex python#29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>
AA-Turner
reviewed
Dec 22, 2025
Comment on lines
+790
to
+794
| <https://www.unicode.org/reports/tr29/>`_. | ||
| Add :func:`~unicodedata.grapheme_cluster_break`, | ||
| :func:`~unicodedata.indic_conjunct_break` and | ||
| :func:`~unicodedata.extended_pictographic` functions to get the properties | ||
| of the character which are related to the above algorithm. |
Member
There was a problem hiding this comment.
Suggested change
| <https://www.unicode.org/reports/tr29/>`_. | |
| Add :func:`~unicodedata.grapheme_cluster_break`, | |
| :func:`~unicodedata.indic_conjunct_break` and | |
| :func:`~unicodedata.extended_pictographic` functions to get the properties | |
| of the character which are related to the above algorithm. | |
| <https://www.unicode.org/reports/tr29/>`_. | |
| Add :func:`~unicodedata.grapheme_cluster_break`, | |
| :func:`~unicodedata.indic_conjunct_break` and | |
| :func:`~unicodedata.extended_pictographic` functions to get the properties | |
| of the character which are related to the above algorithm. |
|
|
||
| Converting an emitted item to string returns a substring corresponding to | ||
| the grapheme cluster. | ||
| It's ``start`` and ``end`` attributes denote the start and the end of |
Member
There was a problem hiding this comment.
Suggested change
| It's ``start`` and ``end`` attributes denote the start and the end of | |
| Its ``start`` and ``end`` attributes denote the start and the end of |
merwok
reviewed
Dec 23, 2025
| ``False`` otherwise. | ||
|
|
||
| .. versionadded:: next | ||
|
|
Member
There was a problem hiding this comment.
The order of functions in this file doesn’t seem to be alphabetical or topical.
I think another ticket should be created to add a quick links table at the top.
merwok
reviewed
Dec 23, 2025
| .. data:: ucd_3_2_0 | ||
|
|
||
| This is an object that has the same methods as the entire module, but uses the | ||
| This is an object that has most of the methods of the entire module, but uses the |
Member
There was a problem hiding this comment.
This sentence is not fully right, but I can’t find the right suggestion with both «most of» and «same as».
Member
|
These functions help compute width? |
Member
Author
|
At least two implementations (in Perl's Unicode::GCString and builtin in C++) use graphemes. Naive implementation in C's |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex
#29.Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm.