Refactor Document Chunker to always use docling#430

Merged

mergify[bot] merged 11 commits intoinstructlab:mainfrom

khaledsulayman:ks-chunking-refactor

Jan 15, 2025

Commits on Jan 13, 2025

Replace DocumentChunker factory with docling-based chunker by default
khaledsulayman
committed
Modify tests to use new DocumentChunker interface
khaledsulayman
committed
Check for html in markdown files and error out
khaledsulayman
committed
replace test_valid_knowledge_skill.yaml with example with no html
khaledsulayman
committed
Update for ruff and pylint issues

aakankshaduggal
authored and
khaledsulayman
committed
Update chunkers.py with minor cleanup

aakankshaduggal
authored and
khaledsulayman
committed
Update test_chunkers to be more modular

aakankshaduggal
authored and
khaledsulayman
committed
Update HTML error to warning to avoid exiting

aakankshaduggal
authored and
khaledsulayman
committed
delete unused arg taxonomy_path in leaf_node_to_samples
khaledsulayman
committed

Commits on Jan 14, 2025

remove commented code in chunkers.py
khaledsulayman
committed

Commits on Jan 15, 2025

optionally accept pathlib.Path for tokenizer_model_name
khaledsulayman
committed