Skip to content

Fix various small problems#367

Open
janEbert wants to merge 3 commits intobigscience-workshop:mainfrom
janEbert:misc-fixes
Open

Fix various small problems#367
janEbert wants to merge 3 commits intobigscience-workshop:mainfrom
janEbert:misc-fixes

Conversation

@janEbert
Copy link
Collaborator

Problems:

  • BERT/T5 dataset handles already corrupted indices incorrectly.
  • GPT tokenizer vocab size does not include special tokens.
  • In Eval Harness evaluation, the last token of the target is unconditionally truncated. This results in wrong evaluation metrics and is actually not such a small problem.

This PR fixes these issues. Since the changes are so small, I didn't bother creating 3 PRs.

Did not include additional special tokens.
This corrupts the targets. There is no good reason for this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant