Fix tokenization issue of CJK languages for evaluation

[owaski]

For CJK languages, we need to tokenize them with CJSegmenter before sending them to mweralign.align_texts.

This PR makes the following modifications:

1. Apply CJSegmenter before calling mweralign.align_texts. This is done for both latency scorer and quality scorer.
2. Add latency_unit argument to the quality scorer and use this argument to trigger CJSegmenter in the quality scorer.

[mgaido91]

thank you @owaski for you contribution. I know we are not doing great on UTs at the moment, but if you could add one it would be great. If I manage to, I will try and provide a suggestion for it, but since you know better what is the expected output for character-level segmentation.

[mgaido91]

nit: shall we move it in the init and have it as self.segmenter? This way, if we have multiple score calls, we have only one instance created.

[owaski]

That's a good idea. I'll do it.

[mgaido91]

The CI has been fixed in #22 . Please pull from the main branch next time you push so that the CI gets fixed here as well, thanks.

[mgaido91]

Regading the UTs, I think it would be great to add a file like this in the UTs:

# Copyright 2026 FBK

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License

import unittest
from argparse import Namespace

from simulstream.metrics.readers import OutputWithDelays, ReferenceSentenceDefinition
from simulstream.metrics.scorers.latency import LatencyScoringSample
from simulstream.metrics.scorers.latency.stream_laal import StreamLaal


class StreamLaalTestCase(unittest.TestCase):
    def test_basic(self):
        reference = [
            ReferenceSentenceDefinition(
                "A New York, sono a capo di un'associazione no profit, chiamata Robin Hood.",
                12.61,
                4.07,
            ),
            ReferenceSentenceDefinition(
                "Quando non combatto la povertà, combatto gli incendi come assistente capitano di "
                "una brigata di pompieri volontari.",
                16.9,
                5.14,
            )
        ]
        hypothesis = OutputWithDelays(
            "Tornando a New York, sono il capo dello sviluppo per un non-profit chiamato Robin "
            "Hood. Quando non sto combattendo la povertà, sto combattendo i fuochi.",
            [14.0, 14.0, 14.0, 14.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 18.0,
             18.0, 18.0, 18.0, 18.0, 18.0, 18.0, 18.0, 20.0, 20.0, 20.0, 20.0],
            [18.22, 18.22, 18.22, 18.22, 19.93, 19.93, 19.93, 19.93, 19.93, 19.93, 19.93, 19.93,
             19.93, 23.01, 23.01, 23.01, 23.01, 23.01, 23.01, 23.01, 23.01, 27.30, 27.30, 27.30,
             27.30,]
        )
        scorer = StreamLaal(Namespace(latency_unit="word"))
        score = scorer.score([LatencyScoringSample("a", hypothesis, reference)])
        self.assertAlmostEqual(score.ideal_latency, 0.868587, 4)
        self.assertAlmostEqual(score.computational_aware_latency, 5.86, 4)
    
    def test_with_characters(self):
        ... # TODO


if __name__ == '__main__':
    unittest.main()

Where we put in the TODO a simple example taken e.g. from past experiments on which we know the expected output latency. And ideally we should add a test case to uts/metrics/log_reader.py to enforce that it works as expected as well. I can also help with these things if you can give a meaningful example, but probably it is faster if you just add it since I have no CJK example at hand. Thanks.

[owaski]

Sure I can add a unit test for Chinese.

[mgaido91]

Sure I can add a unit test for Chinese.

That would be awesome, thanks! Once we have those UTs, this LGTM, thanks!

[owaski]

Sure I can add a unit test for Chinese.

That would be awesome, thanks! Once we have those UTs, this LGTM, thanks!

unit test added.

[owaski]

wait, I found an issue with quality scorer. the self._tokenize function will do in-place modification which changes the sample.reference. let me work on a quick fix

[mgaido91]

Can we add a UT to test the problem you mentioned with the quality segmenter? So that we are sure that future changes will not (re-)introduce it. something similar to the latency scorer UT but with the sacrebleu scorer. It would be gold to check that before your last fix the UT fails and after fixing the issue it passes. Then we can merge this, thanks!

[owaski]

Can we add a UT to test the problem you mentioned with the quality segmenter? So that we are sure that future changes will not (re-)introduce it. something similar to the latency scorer UT but with the sacrebleu scorer. It would be gold to check that before your last fix the UT fails and after fixing the issue it passes. Then we can merge this, thanks!

Just added the unit test for both quality and latency tokenize function

[mgaido91]

just few minor style comments, LGTM otherwise, thanks. Once these three small things are fixed I will merge it. Thanks.

[owaski]

just few minor style comments, LGTM otherwise, thanks. Once these three small things are fixed I will merge it. Thanks.

style fixed!