Adding Semantic Text Splitting & Token Text Splitting

[pasupathimuniyappan]

Description

Add Chonky Semantic Text Splitting feature in the existing flow and route that by passing specific parameter for text splitting.

Changes Made

Added split_text_semantically() function in text_splitting.py file and integrated with split_text()function to route it properly. It requires to modify the codes in the below files.

• pyproject.toml -> added the version of chonky library that used.
• text_splitting.py -> Added the required function and integration.
• vector_db_test.py -> Modified bit in the test_split_text() testcase to test the split_text_semantically function.

How to Test

Add the chonky library to your Python environment.

The smss can be configured with a default chunking method like this: DEFAULT_CHUNKING_METHOD semantic
Or it can be passed in the params of the call.

REACTOR USAGE

CreateEmbeddingsFromDocuments (engine = "1222b449-1bc6-4358-9398-1ed828e4f26a", filePaths = ["fileName1.pdf"], paramValues = [{"chunkingMethod": "semantic"}]);

PYTHON USAGE

from gaas_gpt_vector import VectorEngine
vectorEngine = VectorEngine(engine_id = "1222b449-1bc6-4358-9398-1ed828e4f26a", insight_id = '${i}')
vectorEngine.addDocument(file_paths = ['fileName1.pdf'], param_dict={"chunkingMethod": "semantic"})

SDK USAGE

from ai_server import VectorEngine, ServerClient

server_connection = ServerClient(
    base="http://localhost:9090/Monolith/api",
    access_key="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    secret_key="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
)

is_connected = server_connection.connected
print(f"Am I connected to the server? {is_connected}")

vector_engine = VectorEngine(
    engine_id="1222b449-1bc6-4358-9398-1ed828e4f26a",
    insight_id=server_connection.cur_insight,
)

file_path = "C:/Users/rweiler/Desktop/REPOS/python-sdk/python/ai-server/src/ai_server/tests/test_files/constitution.pdf"

vector_engine.addDocument(
    file_paths=[file_path], param_dict={"chunkingMethod": "semantic"}
)

1. Using sample code snippet:

• Here is the sample code snippet to test the above changes.

from genai_client import get_tokenizer
import vector_database

embed_tokenizer = get_tokenizer(
        tokenizer_name="BAAI/bge-large-en-v1.5",
        max_tokens=None,
        tokenizer_type="EMBEDDED",
    )

vector_database.split_text(
        csv_file_location=sample_csv,
        cfg_tokenizer=embed_tokenizer,
        chunk_unit="tokens",
        chunk_size=10,
        chunk_overlap=0,
        chunking_strategy="PAGE_BY_PAGE",
        split_method="semantic",  # by passing this parameter to test the split_text_semantically function
    )

2. Using Pytest testcase: (Recommended)

• Instead, we can also use the pytest testcase, by uncommenting this line. # split_method="semantic", # enable it if you want to test the split_text_semantically function
• File Path - Semoss\py\testing\vector_database\vector_db_test.py
• Function Name - test_split_text()
  

Notes

• Here is the link to know about the usages of chonky library

[github-actions]

@CodiumAI-Agent /describe

[QodoAI-Agent]

Title

[DRAFT PR - DO NOT MERGE] - 696 Python Add Chonky Semantic Text Splitting

---

User description

Description

Add Chonky Semantic Text Splitting feature in the existing flow and route that by passing specific parameter for text splitting.

Changes Made

Added split_text_semantically() function in text_splitting.py file and integrated with split_text()function to route it properly. It requires to modify the codes in the below files.

• pyproject.toml -> added the version of chonky library that used.
• text_splitting.py -> Added the required function and integration.
• vector_db_test.py -> Modified bit in the test_split_text() testcase to test the split_text_semantically function.

How to Test

1. Using sample code snippet:

• Here is the sample code snippet to test the above changes.

from genai_client import get_tokenizer
import vector_database

embed_tokenizer = get_tokenizer(
        tokenizer_name="BAAI/bge-large-en-v1.5",
        max_tokens=None,
        tokenizer_type="EMBEDDED",
    )

vector_database.split_text(
        csv_file_location=sample_csv,
        cfg_tokenizer=embed_tokenizer,
        chunk_unit="tokens",
        chunk_size=10,
        chunk_overlap=0,
        chunking_strategy="PAGE_BY_PAGE",
        split_method="semantic",  # by passing this parameter to test the split_text_semantically function
    )

2. Using Pytest testcase: (Recommended)

• Instead, we can also use the pytest testcase, by uncommenting this line. # split_method="semantic", # enable it if you want to test the split_text_semantically function
• File Path - Semoss\py\testing\vector_database\vector_db_test.py
• Function Name - test_split_text()
  

Notes

• Here is the link to know about the usages of chonky library

---

PR Type

Enhancement, Tests

---

Description

• Add semantic text splitting via Chonky

• Route split_text by split_method parameter

• Implement split_text_semantically function

• Include Chonky dependency in pyproject.toml

• Update test fixture for semantic splitting

---

Changes walkthrough 📝

	Relevant files
Tests	vector_db_test.py Expand test fixture and semantic test option                          py/testing/vector_database/vector_db_test.py Expand sample_csv fixture with longer content Add commented split_method="semantic" option +5/-3
Enhancement	text_splitting.py Integrate semantic splitting via Chonky                                    py/vector_database/utils/text_splitting.py Add split_method check in split_text Introduce split_text_semantically function Use chonky.TextSplitter for semantic chunking +62/-9
Dependencies	pyproject.toml Add Chonky dependency                                                                        py/install_config/pyproject.toml Add "chonky==0.1.4" dependency +1/-0

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[github-actions]

@CodiumAI-Agent /review

[github-actions]

@CodiumAI-Agent /improve

[QodoAI-Agent]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Divider Type Inconsistency The semantic splitter sets the Divider column to the string "semantic", which may conflict with numerical expectations downstream and cause type or sorting errors. "text", "semantic", # No specific page number since it's semantic text splitting i, cfg_tokenizer.count_tokens(chunk), chunk, ] Potential NameError clean_up_string is used inside split_text_semantically without being imported or defined in this scope, leading to a potential NameError at runtime. # Initialize Chonky's semantic text splitter (uses transformer models under the hood) Missing Semantic Test The split_method="semantic" parameter remains commented out, so the semantic splitting branch isn’t actually exercised by the tests; add or enable a test for that path. csv_file_location=sample_csv, cfg_tokenizer=embed_tokenizer, chunk_unit="tokens", chunk_size=10, chunk_overlap=0, chunking_strategy="PAGE_BY_PAGE", # split_method="semantic", # enable it if you want to test the split_text_semantically function

[QodoAI-Agent]

PR Code Suggestions ✨

Category	Suggestion	Impact
Possible issue	Declare split_method parameter Ensure split_method is declared in the split_text signature so it is defined when used. py/vector_database/utils/text_splitting.py [141-146] -if split_method == "semantic": - text_results_df = split_text_semantically( - text_results_df=text_results_df, - document_name=document_name, - cfg_tokenizer=cfg_tokenizer, - ) +def split_text( + main_df: pd.DataFrame, + chunking_strategy: str, + cfg_tokenizer, + chunk_unit: str, + chunk_size: int, + chunk_overlap: int, + split_method: str = "recursive", +) -> pd.DataFrame: + document_name = main_df["Source"][0] + if split_method == "semantic": + text_results_df = split_text_semantically( + text_results_df=text_results_df, + document_name=document_name, + cfg_tokenizer=cfg_tokenizer, + ) + else: + text_results_df = split_text_recursively( + text_results_df=text_results_df, + chunking_strategy=chunking_strategy, + document_name=document_name, + cfg_tokenizer=cfg_tokenizer, + chunk_unit=chunk_unit, + chunk_size=chunk_size, + chunk_overlap=chunk_overlap, + ) Suggestion importance[1-10]: 9 __ Why: Without split_method in the signature, the semantic branch will raise a NameError; adding the parameter is crucial for functionality.	High
	Initialize missing DataFrame Initialize other_modalities_df before concatenation to avoid a NameError in the semantic branch. py/vector_database/utils/text_splitting.py [158-159] # Combine text chunks with other modalities and save to CSV +other_modalities_df = pd.DataFrame() result = pd.concat([text_results_df, other_modalities_df], ignore_index=True) Suggestion importance[1-10]: 8 __ Why: other_modalities_df is undefined when split_method is semantic, so initializing it prevents a NameError.	Medium
General	Consistent Divider column type Use a consistent type (e.g. None or a numeric code) for the Divider column instead of a string to keep column types uniform. py/vector_database/utils/text_splitting.py [307-318] text_results_df = pd.DataFrame( [ [ document_name, "text", - "semantic", # No specific page number since it's semantic text splitting + None, # No specific divider for semantic chunks i, cfg_tokenizer.count_tokens(chunk), chunk, ] for i, chunk in enumerate(chunks) ], columns=["Source", "Modality", "Divider", "Part", "Tokens", "Content"], ) Suggestion importance[1-10]: 5 __ Why: Changing the Divider value to None ensures a uniform column type, improving data consistency with minor impact.	Low

[github-actions]

@CodiumAI-Agent /update_changelog

[QodoAI-Agent]

Changelog updates: 🔄

2025-05-05 *

Added

• Configurable chunkingMethod with support for semantic (Chonky) and token‐based text splitting
• Integrated semantic and token splitters into Java and Python flows

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'