feat: retry validation feature (MAPCO-8327)#42
Merged
Conversation
razbroc
changed the title
…ValidationTaskParameters interface
…ers in IngestionManager
…ameters in job request functions
CL-SHLOMIKONCHA
requested changes
Nov 18, 2025
| operationId: retryLayer | ||
| tags: | ||
| - ingestion | ||
| summary: retry a failed ingestion job caused by validation errors |
Contributor
There was a problem hiding this comment.
this retry will handle also other task failures but not only validation errors
CL-SHLOMIKONCHA
requested changes
Nov 20, 2025
|
|
||
| return validationTask.parameters.isValid | ||
| ? this.handleRetryWithoutErrors(jobId, validationTask.id, logCtx) | ||
| : this.handleRetryWithErrors(jobId, retryJob, validationTask, logCtx); |
Contributor
Author
There was a problem hiding this comment.
why my friend? i return a promise therefor returning an awaited promise is not allowed in this context
…rity and accuracy
… to debug for checksum calculations
modifying isValid approach to optional
… and rename reset method for clarity
…id field and adjust related logic in IngestionManager
…ager and interfaces
CL-SHLOMIKONCHA
requested changes
Dec 3, 2025
… handling in IngestionManager
… IngestionManager
CL-SHLOMIKONCHA
requested changes
Dec 4, 2025
Contributor
CL-SHLOMIKONCHA
left a comment
CL-SHLOMIKONCHA
left a comment
There was a problem hiding this comment.
just add what we discussed about:
expect for update task parameters body
CL-SHLOMIKONCHA
approved these changes
Dec 4, 2025
* test: add retry ingestion corresponding tests * style: adjust formatting in IngestionRequestSender constructor and update bad request test cases for ingestion * style: improve formatting in IngestionRequestSender constructor and update test case descriptions for clarity * test: add polygonPartsManagerURL to default test configuration * style:lint * fix: handle NotFoundError and improve input file path handling in ingestion process + fix tests * test: update job reset descriptions for clarity in ingestion tests * test: refactor bad request test cases for ingestion API to improve clarity and structure * style: lint * chore: revet coverage to original * test: update coverage threshold for statements in jest configuration and add checksum unit test * test: update jest configuration for unused schema and adjust coverage statements * style: lint * feat: changed all shapefiles path inside parameters to be relative * chore: upgrade raster shared (#49) * chore: update to new raster-shared * refactor: remove unused validationTaskParametersSchema and update validation logic to use ingestionValidationTaskParamsSchema * refactor: remove unused checksumSchema import from interfaces * feat: add generateFullChecksum function and update usages in IngestionManager tests * Refactor ingestion tests to improve validation cases and adjust mock data generation - Updated bad request test cases in ingestion.spec.ts to ensure comprehensive coverage of input validation scenarios. - Adjusted the count of regions in metadata generation to a maximum of 5 instead of 100 for better test performance. - Modified input files generation to limit gpkgFilesPath to a single file for more controlled testing. - Ensured consistency in file path generation across various mock functions. * style: lint * feat: remove unused generateChecksum import in ingestionManager tests * refactor: rename getFilesChecksum to getChecksum and update usages * refactor: update validation task parameters to use IngestionValidationTaskParams and remove unused imports * refactor: remove trailing whitespace in interfaces.ts * test: fix pr comments - update error handling and checksum generation in tests * test: remove excessive timeout from nonexistent file response test * test: update error handling to throw NotFoundError instead of FileNotFoundError * fix: update coverage threshold for statements in jest config --------- Co-authored-by: shlomiko <shlomiko@rafael.co.il>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add new “retry” api for an active failed job caused by “failed” validations task (in case of errors in validations):
First design the new api via swagger.
check if job is “retry-able”:
ingestion job status is “Failed” & validations task status is “Completed”/”Failed” - (Recoveable):
if validation task has been completed with errors → can be try.
if validation task has been completed without errors → cannot be retry
if validation task has been failed:
delete by the polygon-parts-manager api - the validation table → can be retry
use the shp hash for job parameter to compare and detect for a shp file changes (via task parameters):
if equal → throw an Bad Request error with relevant error message
add new checksum (hash) to the task parameters checksums array.
“reset” task validation result within task parameters
set the job and task statuses back to “Pending” for re-consume.
** if the requested retry job is contain more than just “validation” task (means validation completed successfully) - trigger the job-manager api to retry job.
Related issues: #MAPCO-8327