[fix] fix token healing tests and usage errors#33931
[fix] fix token healing tests and usage errors#33931ArthurZucker merged 16 commits intohuggingface:mainfrom
Conversation
|
cc @gante! 🤗 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
gante
left a comment
gante
left a comment
There was a problem hiding this comment.
@alpertunga-bile thank you for pinning the issue, finding the cause and problems with the test setup, and opening a PR with the fix -- contributors like you make transformers shine 💛
LGTM 👍
Note for our future selves, in case we need more context about PR: check this comment
|
The red CI should get fixed after #33950 is merged. At that point, I'd like to kindly ask you to rebase and force-push, to re-run CI 🤗 |
|
@e can see token healing tests being run in the CI 👍 from https://app.circleci.com/pipelines/github/huggingface/transformers/106787/workflows/081efefd-ecd1-4eed-8496-b4d03d6e492e/jobs/1417470/parallel-runs/7?filterBy=ALL : 
|
alpertunga-bile
force-pushed
the
fix_token_healing
branch
from
e4dac42 to
47a04f7
Compare
|
cc @ArthurZucker as we just discussed it in person |
alpertunga-bile
force-pushed
the
fix_token_healing
branch
2 times, most recently
from
81ced1c to
641b200
Compare
|
@alpertunga-bile sorry about that :( #34067 -- this would be the PR |
alpertunga-bile
force-pushed
the
fix_token_healing
branch
from
ea06e24 to
4a476ba
Compare
Thanks, it is passed through the tests now. |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
Nice, let's merge 🚀
* auto-gptq requirement is removed & model is changed & tokenizer pad token is assigned * values func is changed with extensions & sequence key value bug is fixed * map key value check is added in ExtensionsTree * empty trimmed_ids bug is fixed * tail_id IndexError is fixed * empty trimmed_ids bug fix is updated for failed test * too much specific case for specific tokenizer is removed * input_ids check is updated * require auto-gptq import is removed * key error check is changed with empty list check * empty input_ids check is added * empty trimmed_ids fix is checked with numel function * usage change comments are added * test changes are commented * comment style and quality bugs are fixed * test comment style and quality bug is fixed
5 participants
What does this PR do?
Hello! As I described in the #33827; token healing functionality is failing in usage and tests.
This PR is fixing the token healing tests and usage bugs.
Fixes #33827
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@gante