Optimize corpus tests: mock downloads, separate data validation, suppress CLI output

[Copilot]

What do these changes do

Separates corpus testing concerns: fast mocked unit tests vs comprehensive data validation. Eliminates network dependencies and output pollution from regular test runs.

What was wrong

Unit tests: Downloaded large corpus files (96MB OSCAR, 186MB TNC) on every run, taking 21s and requiring network access. Caused slow development cycles and flaky tests.

CLI tests: Printed verbose catalog listings and corpus metadata to logs, obscuring test failures. Made network requests during testing.

Missing coverage: No tests for corpus catalog operations (get_corpus_db, corpus_db_url, catalog structure validation).

How this fixes it

1. Mock corpus downloads in unit tests (tests/core/test_corpus.py)

# Before: Downloads 96MB, parses 8.6s
def test_oscar(self):
    self.assertIsNotNone(oscar.word_freqs())

# After: Mock data, validates parsing, <0.01s
with patch('pythainlp.corpus.oscar.get_corpus_path', return_value="/mock"):
    with patch('builtins.open', mock_open(read_data=mock_csv)):
        result = oscar.word_freqs()
        # Validates CSV parsing, empty string handling, quote filtering

Result: 21s → 2s (90% faster), no network dependency

2. Separate corpus data workflow (.github/workflows/corpus.yml)

• Tests actual file loading without mocks
• Triggers only on pythainlp/corpus/** or tests/corpus/** changes
• Three categories: catalog, built-in, downloadable

3. Corpus catalog tests (tests/corpus/test_catalog.py)

• Catalog download and JSON structure validation
• Known entries verification
• Version metadata checks
• Local DB query testing

4. Suppress CLI output (tests/core/test_cli.py, tests/extra/testx_cli.py)

with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
    with patch('pythainlp.corpus.download', return_value=True):
        DataApp(["thainlp", "data", "get", "test"])

Result: Clean logs, 200x faster data CLI tests

5. Simplified naming

• Directory: tests/corpus/
• Package: tests.corpus
• Workflow: "Corpus Test"
• Clear, concise, easy to type

Test coverage maintained: all parsing logic validated via representative mock data.

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

Original prompt

Test cases of pythainlp.corpus downloads slowing down the development-test cycles. Some of the test cases are not reliable too, as sometimes they timed out and failed and make the whole test workflow failed.

Initial thought is that it involves network request (test corpus existence / download / query corpus catalog).

Try to improve the speed and stability of these corpus test cases, while maintain relatively high test coverage of the submodule.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[coveralls]

coverage: 65.333% (-0.02%) from 65.355%
when pulling 2a817fd on copilot/improve-corpus-test-speed
into d55e615 on dev.

[bact]

@copilot apply suggestions about documentation and tests in
#1291 (review)
#1291 (review)
#1291 (review)
#1291 (review)

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud