Skip to content

Added AttachSourceToVectorDbReactor for uploading pdf file to an exis…#736

Merged
themaherkhalil merged 8 commits intodevfrom
AttachSourceToVectorDbChanges
May 1, 2025
Merged

Added AttachSourceToVectorDbReactor for uploading pdf file to an exis…#736
themaherkhalil merged 8 commits intodevfrom
AttachSourceToVectorDbChanges

Conversation

@ManjariYadav2310
Copy link
Copy Markdown
Contributor

…ting csv file and modified VectorFileDownloadReactor

Description

Changes Made

How to Test

  1. Steps to reproduce/test the behavior
  2. Expected outcomes

Notes

…ting csv file and modified VectorFileDownloadReactor
@github-actions
Copy link
Copy Markdown

@CodiumAI-Agent /describe

@QodoAI-Agent
Copy link
Copy Markdown

Title

Added AttachSourceToVectorDbReactor for uploading pdf file to an exis…


User description

…ting csv file and modified VectorFileDownloadReactor

Description

Changes Made

How to Test

  1. Steps to reproduce/test the behavior
  2. Expected outcomes

Notes


PR Type

Enhancement, Bug fix


Description

  • Add reactor for attaching source files to vector DB

  • Move and copy uploaded files into DB directory

  • Enhance download reactor with zip and single-file logic

  • Improve error handling and logging messages


Changes walkthrough 📝

Relevant files
Enhancement
AttachSourceToVectorDbReactor.java
Add reactor for attaching sources                                               

src/prerna/reactor/vector/AttachSourceToVectorDbReactor.java

  • Introduce new AttachSourceToVectorDbReactor class
  • Validate engine access via SecurityEngineUtils
  • Create insight folder and normalize paths
  • Copy source file into vector DB directory
  • +80/-0   
    Bug fix
    VectorFileDownloadReactor.java
    Enhance download reactor with zip support                               

    src/prerna/reactor/vector/VectorFileDownloadReactor.java

  • Streamline single-file download with buffer improvements
  • Add multi-file zipping via ZipOutputStream
  • Validate file existence and count missing files
  • Refine exception messages and logging
  • +37/-31 

    Need help?
  • Type /help how to ... in the comments thread for any questions about PR-Agent usage.
  • Check out the documentation for more information.
  • @github-actions
    Copy link
    Copy Markdown

    @CodiumAI-Agent /review

    @QodoAI-Agent
    Copy link
    Copy Markdown

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
    🧪 No relevant tests
    🔒 Security concerns

    Path traversal:
    File names from fileNameList are used directly to build file paths without sanitization, allowing potential directory traversal if user-supplied names include ../ sequences.

    ⚡ Recommended focus areas for review

    NullPointer Risk

    fileLocationNew is initialized to null and used in a comparison and Paths.get without null checks, which can lead to NullPointerException if the input path key is missing.

    String fileLocationNew = null;
    
    // if the file location is not defined generate a random path and set
    // location so that the front end will download
    
    String insightFolder = null;
    if (fileLocation != null) {
    	insightFolder = this.insight.getInsightFolder();
    	{
    		File f = new File(Utility.normalizePath(insightFolder));
    		if (!f.exists()) {
    			f.mkdirs();
    		}
    	}
    	fileLocation = fileLocation.replace("\\", "");
    	fileName = fileName.replace("\\", "");
    	fileLocation = insightFolder + DIR_SEPARATOR + fileName;
    	fileLocationNew = insightFolder + DIR_SEPARATOR + this.keyValue.get(ReactorKeysEnum.FILE_PATH.getKey()).replace("\\", "");
    
    }
    if (!fileLocation.equals(fileLocationNew)) {
    Exception Handling

    The catch block for file move prints stack trace directly (e.printStackTrace()) instead of using the logger and does not propagate the error, potentially hiding failures.

    try {
    	Files.move(oldSourcePath, oldSourcePath.resolveSibling(this.keyValue.get(ReactorKeysEnum.FILE_NAME.getKey()).replace("\\", "")));
    } catch (IOException e) {
    	// TODO Auto-generated catch block
    	e.printStackTrace();
    }
    Resource Leak

    File streams (FileInputStream, FileOutputStream, ZipOutputStream) opened during download are not closed in all code paths, leading to potential resource leaks.

    if (fileNameList != null && fileNameList.size() > 0) {
    	int fileExistsCount = 0;
    	if (fileNameList.size() == 1) {
    		String filepath = thisEngineDir + DIR_SEPARATOR + fileNameList.get(0);
    		File fileToCheck = new File(filepath);
    		if (fileToCheck.exists()) {
    			fileExistsCount++;
    			FileInputStream fis = new FileInputStream(fileToCheck);
    			outFilePath = outputDir + DIR_SEPARATOR + fileNameList.get(0);
    			fileoutStream = new FileOutputStream(outFilePath);
    			byte[] buffer = new byte[2048];
    			int bytesRead;
    			while ((bytesRead = fis.read(buffer)) >= 0) {
    				fileoutStream.write(buffer, 0, bytesRead);
    			}
    		}

    @github-actions
    Copy link
    Copy Markdown

    @CodiumAI-Agent /improve

    @QodoAI-Agent
    Copy link
    Copy Markdown

    PR Code Suggestions ✨

    CategorySuggestion                                                                                                                                    Impact
    Possible issue
    Add null safety to file comparison

    Add null checks for fileLocation and fileLocationNew before comparison and handle
    missing values to prevent NullPointerException.

    src/prerna/reactor/vector/AttachSourceToVectorDbReactor.java [61-68]

    -if (!fileLocation.equals(fileLocationNew)) {
    +if (fileLocation != null && fileLocationNew != null && !fileLocation.equals(fileLocationNew)) {
         Path oldSourcePath = Paths.get(fileLocationNew);
         try {
             Files.move(oldSourcePath, oldSourcePath.resolveSibling(this.keyValue.get(ReactorKeysEnum.FILE_NAME.getKey()).replace("\\", "")));
         } catch (IOException e) {
    -        // TODO Auto-generated catch block
    -        e.printStackTrace();
    +        classLogger.error("Error moving file from {} to {}: {}", fileLocationNew, fileName, e);
    +        throw new IllegalArgumentException("Failed to move source file", e);
         }
     }
    Suggestion importance[1-10]: 8

    __

    Why: This prevents a potential NullPointerException when comparing paths and adds logging with a clear exception for better failure handling.

    Medium
    Use try-with-resources for streams

    Use try-with-resources for FileInputStream, FileOutputStream, and ZipOutputStream to
    ensure all streams are closed properly.

    src/prerna/reactor/vector/VectorFileDownloadReactor.java [86-93]

    -FileInputStream fis = new FileInputStream(fileToCheck);
    -outFilePath = outputDir + DIR_SEPARATOR + fileNameList.get(0);
    -fileoutStream = new FileOutputStream(outFilePath);
    -byte[] buffer = new byte[2048];
    -int bytesRead;
    -while ((bytesRead = fis.read(buffer)) >= 0) {
    -    fileoutStream.write(buffer, 0, bytesRead);
    +try (FileInputStream fis = new FileInputStream(fileToCheck);
    +     FileOutputStream fileoutStream = new FileOutputStream(outFilePath)) {
    +    byte[] buffer = new byte[2048];
    +    int bytesRead;
    +    while ((bytesRead = fis.read(buffer)) >= 0) {
    +        fileoutStream.write(buffer, 0, bytesRead);
    +    }
     }
    Suggestion importance[1-10]: 6

    __

    Why: Applying try-with-resources guarantees streams are closed automatically, preventing resource leaks and simplifying the cleanup logic.

    Low
    General
    Improve exception handling

    Replace e.printStackTrace() with proper logging and throw a descriptive exception to
    ensure upstream handlers can react.

    src/prerna/reactor/vector/AttachSourceToVectorDbReactor.java [65-68]

     catch (IOException e) {
    -    // TODO Auto-generated catch block
    -    e.printStackTrace();
    +    classLogger.error("Failed to move file from {}: {}", fileLocationNew, e);
    +    throw new SemossPixelException("Unable to move source file: " + e.getMessage(), e);
     }
    Suggestion importance[1-10]: 7

    __

    Why: Replacing printStackTrace with structured logging and a custom exception ensures errors are propagated properly and are easier to diagnose.

    Medium

    @themaherkhalil themaherkhalil merged commit 77273d4 into dev May 1, 2025
    3 checks passed
    @themaherkhalil themaherkhalil deleted the AttachSourceToVectorDbChanges branch May 1, 2025 05:24
    @github-actions
    Copy link
    Copy Markdown

    github-actions bot commented May 1, 2025

    @CodiumAI-Agent /update_changelog

    @QodoAI-Agent
    Copy link
    Copy Markdown

    Changelog updates: 🔄

    2025-05-01 [*][https://github.com//pull/736]

    Added

    • VectorAttachFileToSourceReactor to attach source files to vector DB entries

    Changed

    • VectorFileDownloadReactor: renamed download key, added multi-file support, warnings and improved error handling

    to commit the new content to the CHANGELOG.md file, please type:
    '/update_changelog --pr_update_changelog.push_changelog_changes=true'

    manamittal added a commit that referenced this pull request May 20, 2025
    * fix(python): handle eval when it is a single line execution but there is string input with space (#756)
    
    * Update Dockerfile.tomcat (#757)
    
    * fix: tomcat builder setting env var
    
    * fix: updating tomcat to 9.0.104
    
    * Update Dockerfile.ubuntu22.04
    
    * Update Dockerfile.ubuntu22.04
    
    * Update Dockerfile.ubuntu22.04
    
    * feat: creating KubernetesModelScaler class (#763)
    
    * Update Dockerfile.ubuntu22.04
    
    * feat: adding ability to attach a file to a vector db source (#736)
    
    * Added AttachSourceToVectorDbReactor for uploading pdf file to an existing csv file and modified VectorFileDownloadReactor
    
    * fix: proper return for the download and matching the reactor name
    
    * fix: error for downloading single file vs multiple; error for copyToDirectory instead of copyFile
    
    * chore: renaming so reactor matches VectorFileDownload
    
    ---------
    
    Co-authored-by: Maher Khalil <themaherkhalil@gmail.com>
    
    * Update Dockerfile.ubuntu22.04
    
    * Update ubuntu2204.yml
    
    * Update ubuntu2204.yml
    
    * Update ubuntu2204_cuda.yml
    
    * Update Dockerfile.nvidia.cuda.12.5.1.ubuntu22.04
    
    * Update ubuntu2204_cuda.yml
    
    * Update ubuntu2204.yml
    
    * feat: exposing tools calling through models (#764)
    
    * 587 unit test for prernadsutil (#654)
    
    * test(unit): unit tests for the prerna.util.ds package
    
    * test(unit): unit tests for the prerna.util.ds.flatfile package
    
    * test(unit): removed reflections, added paraquet tests
    
    * test(unit): unit tests for the prerna.util.ds package
    
    * test(unit): unit tests for the prerna.util.ds.flatfile package
    
    * test(unit): removed reflections, added paraquet tests
    
    * Update ubuntu2204.yml
    
    * Update ubuntu2204.yml
    
    * Update ubuntu2204.yml
    
    * fix: update pipeline docker buildx version
    
    * fix: ignore buildx
    
    * fix: adjusting pipeline for cuda
    
    * feat: switching dynamic sas to default false (#766)
    
    * fix: changes to account for version 2.0.0 of pyjarowinkler (#769)
    
    * chore: using 'Py' instead of 'py' to be consistent (#770)
    
    * feat: full ast parsing of code to return evaluation of the last expression (#771)
    
    * Python Deterministic Token Trimming for Message Truncation (#765)
    
    * feat: deterministic-token-trimming
    
    * feat: modifying logic such that system prompt is second to last message for truncation
    
    ---------
    
    Co-authored-by: Maher Khalil <themaherkhalil@gmail.com>
    
    * fix: added date added column to enginepermission table (#768)
    
    * fix: add docker-in-docker container to run on sef-hosted runner (#773)
    
    Co-authored-by: Raul Esquivel <resmas.work@gmail.com>
    
    * fix: properly passing in the parameters from kwargs/smss into model limits calculation (#774)
    
    * fix: removing legacy param from arguments (#777)
    
    * fix: Fix docker cache build issue (#778)
    
    * adding no cache
    
    * adding no cache
    
    * feat: Adding Semantic Text Splitting & Token Text Splitting (#720)
    
    * [696] - build - Add chonky semantic text splitting - Added the function for chonky semantic text splitting and integrated with existing flow.
    
    * [696] - build - Add chonky semantic text splitting - Updated the code
    
    * [696] - build - Add chonky semantic text splitting - Updated the code comments
    
    * feat: adding reactor support through java
    
    * feat: updating pyproject.toml with chonky package
    
    * feat: check for default chunking method in smss
    
    * [696] - feat - Add chonku semantic text splitting - Resolved the conflicts
    
    * [696] - feat - Add chonky semantic text splitting - Organized the code.
    
    * feat: adding chunking by tokens and setting as default
    
    * updating comments on chunking strategies
    
    ---------
    
    Co-authored-by: Weiler, Ryan <ryanweiler92@gmail.com>
    Co-authored-by: kunal0137 <kunal0137@gmail.com>
    
    * feat: allowing for tools message in full prompt (#780)
    
    * UPDATE ::: Add docker in docker Dockerfiler (#784)
    
    * add docker in docker Dockerfile
    
    * Update Dockerfile.dind
    
    Remove python and tomcat arguments from Dockerfile
    
    * fix: remove-paddle-ocr (#786)
    
    * [#595] test(unit): adds unit test for prerna.engine.impl.model.kserve
    
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * feat: Tag semoss image (#789)
    
    * adding changes for non-release docker build
    
    * adding non-release build logic to cuda-semoss builder
    
    * updating push branches
    
    * fix: branch names on docker builds
    
    * fix: branch names on docker builds cuda
    
    * fix: adding push condition - change to pyproject toml file; adding event input vars to env vars (#790)
    
    * fix: python builder toml file change (#792)
    
    * fix: Catch errors when calling pixels from Python (#787)
    
    Co-authored-by: Weiler, Ryan <ryanweiler92@gmail.com>
    
    * Creating db links between engines and default apps (#693)
    
    * create db links between engine and default app
    
    * Rename column APPID to TOOL_APP
    
    * feat: add database_tool_app to getUserEngineList
    
    ---------
    
    Co-authored-by: Weiler, Ryan <ryanweiler92@gmail.com>
    
    * Adding sort options to the myengines reactor (#479)
    
    * added sort feature to MyEnginesReactor and genericized reactor imports
    
    * formatting
    
    * overloading method
    
    * validate sortList
    
    ---------
    
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * feat: cleaning up unused imports in MyEngine reactor (#793)
    
    * feat: Create Enum projectTemplate and update CreateAppFromTemplateReactor to accept existing appID for cloning applications (#621)
    
    Co-authored-by: kunal0137 <kunal0137@gmail.com>
    
    * Update GetEngineUsageReactor.java (#417)
    
    Co-authored-by: Maher Khalil <themaherkhalil@gmail.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * Issue 596: Adds Unit Tests for prerna/engine/impl/model/responses and workers (#727)
    
    * [#596] test(unit): adds unit tests
    
    * fix: implements ai-agents suggestions
    
    ---------
    
    Co-authored-by: Jeff Vitunac <jvitunac@gmail.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * 609 implement native blob storage for azure gcp and aws (#674)
    
    * Initial commit : implementation for azure blob storage
    
    * added dependency for azure in pom.xml
    
    * update logic to fetch the metadata from list details
    
    * changed functionality from listing containers to listing files within a selected container
    
    * initial commit for google cloud storage implementation
    
    * added field contant in enum class and removed unused method
    
    * add methods to parse comma-separated local and cloud paths
    
    * add methods to parse comma-separated local and cloud paths
    
    * implementation for aws s3 bucket
    
    * normalize container prefix path
    
    * merged all: implementation for azure, aws and gcp
    
    * refactor(storage): replace manual path normalization with normalizePath from Utility class
    
    ---------
    
    Co-authored-by: pvijayaraghavareddy <pvijayaraghavareddy@WORKSPA-6QV71G7.us.deloitte.com>
    Co-authored-by: Parth <parthpatel3@deloitte.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * Get Node Pool Information for Remote Models (#806)
    
    * 590 unit test for prernaengineimpl (#808)
    
    * test(unit): update to filesystems hijacking for testing files
    
    * test: start of unit tests for abstract database engine
    
    * test(unit): added unit test for prerna.engine.impl
    
    * test(unit): finsihed tests for prerna.engine.impl
    
    * test(unit): adding back unused assignment
    
    ---------
    
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * Creating WordCountTokenizer Class (#802)
    
    * feat: creating word count tokenizer class && falling back to word count tokenizer if tiktok fails
    
    * feat: updating comment
    
    * feat: setting default chunking method as recursive (#810)
    
    * Unit tests fixes and Unit test Class file location updates (#812)
    
    * test(unit): moved tests to correct packages
    
    * test(unit): fixed a couple of unit tests
    
    * VectorDatabaseQueryReactor: output divider value for word doc chunks always 1 (#804)
    
    * Code implementation for #733
    
    * feat: Added code to resolve Divider page issue
    
    * Console output replaced by LOGGERs as per review comments
    
    * feat: replaced Console with Loggers
    
    ---------
    
    Co-authored-by: Varaham <katchabi50@gmail.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * GetCurrentUserReactor (#818)
    
    Adding GetCurrentUserReactor to return user info including if user is an admin.
    
    * Python User Class (#819)
    
    * fix: trimming properties read from smss; fix: logging commands before executing (#821)
    
    * Updating getNodePoolsInfo() to parse and return zk info and models active actual (#822)
    
    * feat: update get node pool information for zk info and models active actual
    
    * feat: get remote model configs
    
    * Add unit tests for package prerna\engine\impl\vector (#728)
    
    * Create ChromaVectorDatabaseEngineUnitTests.java
    
    * completed tests for ChromaVectorDatabaseEngine class
    
    * [#604] test(unit): Created ChromaVectorDatabaseEngine unit tests
    
    * [604] tests(unit) : Completed test cases for ChromaVectorDatabaseEngine; update File operations to nio operations in ChromaVectorDatabaseEngine.java
    
    * [#604] tests(unit): added unit tests for all vector database engines and util classes in the prerna\engine\impl\vector package
    
    * [604] test(unit): replaced creating file paths with string literals with java.nio Paths.resolve/Paths.get methods
    
    ---------
    
    Co-authored-by: Maher Khalil <themaherkhalil@gmail.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * feat: adding to the return of getenginemetadata (#813)
    
    * feat: adding to the return of getenginemetadata
    
    * fix: removing throws
    
    ---------
    
    Co-authored-by: Arash Afghahi <48933336+AAfghahi@users.noreply.github.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    
    * 718 create a single reactor to search both engines and apps (#794)
    
    * feat(engineProject): Initial commit
    
    * chore: 718 create a single reactor to search both engines and apps
    
    * chore: 718 create a single reactor to search both engines and apps
    
    ---------
    
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    Co-authored-by: Vijayaraghavareddy <pvijayaraghavareddy@deloitte.com>
    
    * feat: update openai wrapper to handle multiple images (#832)
    
    * feat: adding user room map (#840)
    
    * feat: hiding side menu bar for non admins (#833)
    
    * Side menu changes
    
    * Review Comments fixed
    
    * Flag is renamed in  Constants.java
    
    * Review Comment fixed in Utility.java
    
    * fix: cleaning up defaults and comments
    
    ---------
    
    Co-authored-by: kunal0137 <kunal0137@gmail.com>
    
    ---------
    
    Co-authored-by: Maher Khalil <themaherkhalil@gmail.com>
    Co-authored-by: kunal0137 <kunal0137@gmail.com>
    Co-authored-by: Ryan Weiler <ryanweiler92@gmail.com>
    Co-authored-by: ManjariYadav2310 <manjayadav@deloitte.com>
    Co-authored-by: dpartika <dpartika@deloitte.com>
    Co-authored-by: Raul Esquivel <resmas.work@gmail.com>
    Co-authored-by: Pasupathi Muniyappan <pasupathi.muniyappan@kanini.com>
    Co-authored-by: resmas-tx <131498457+resmas-tx@users.noreply.github.com>
    Co-authored-by: AndrewRodddd <62724891+AndrewRodddd@users.noreply.github.com>
    Co-authored-by: radkalyan <107957324+radkalyan@users.noreply.github.com>
    Co-authored-by: samarthKharote <samarth.kharote@kanini.com>
    Co-authored-by: Shubham Mahure <shubham.mahure@kanini.com>
    Co-authored-by: rithvik-doshi <81876806+rithvik-doshi@users.noreply.github.com>
    Co-authored-by: Mogillapalli Manoj kumar <86736340+Khumar23@users.noreply.github.com>
    Co-authored-by: Jeff Vitunac <jvitunac@gmail.com>
    Co-authored-by: pvijayaraghavareddy <pvijayaraghavareddy@WORKSPA-6QV71G7.us.deloitte.com>
    Co-authored-by: Parth <parthpatel3@deloitte.com>
    Co-authored-by: KT Space <119169984+Varaham@users.noreply.github.com>
    Co-authored-by: Varaham <katchabi50@gmail.com>
    Co-authored-by: ericgonzal8 <ericgonzalez8@deloitte.com>
    Co-authored-by: Arash Afghahi <48933336+AAfghahi@users.noreply.github.com>
    Co-authored-by: Vijayaraghavareddy <pvijayaraghavareddy@deloitte.com>
    Co-authored-by: ammb-123 <ammb@deloitte.com>
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Labels

    None yet

    Projects

    None yet

    Development

    Successfully merging this pull request may close these issues.

    3 participants