Tokenisation

[tanzheen]

Hi, thank you for your work!

I am currently looking through but I am struggling to understand where does the tokenisation of the ground truth sentence actually take place?

Thank you!

[EtaEnding]

Hi! We followed SLT’s pipeline so it should be in files like dataset and training.py! Thank you!