Aux data inserters-->aux data managers#67
Merged
webbpinner merged 11 commits intoOceanDataTools:dev_2.5from Mar 6, 2026
Merged
Aux data inserters-->aux data managers#67webbpinner merged 11 commits intoOceanDataTools:dev_2.5from
webbpinner merged 11 commits intoOceanDataTools:dev_2.5from
Conversation
…x and Coriolix implementations Note: initial refactor was done by CoPilot and then edited by me
insert_aux_data_from_ws now runs one connection session and raises exceptions cleanly on disconnect The outer loop in run_aux_data_inserter is now the sole responsibility for retry logic. inner loop = "listen for events", outer loop = "reconnect on failure"
…t all inserters need a query
Instead of directly interacting with the eventAuxDataTable, the events API endpoints now publish deleteEvents messages, which the aux_data_managers are listening for. When an aux_data_manager receives a deleteEvents messages, it sends the contents to its aux_data_file_cleaner(s) which do any external clean up required and then send a delete request to the API event_aux_data endpoint # Conflicts: # routes/api/v1/events.js
…en you delete framegrab aux data, it will also delete the images from the sealog files Note: we do not use framegrab, so I didn't have a great way to test this, but it is very similar to our stillcap_ffmpeg aux data mgr, which I have tested extensively
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change simplifies and expands aux data processes by:
aux_data_record_builders into an abstract base class and standardizing external connectionsaux_data_inserters into a runner scriptaux_data_file_cleaners to handle custom requirements when deleting aux dataFramegrabHTTPAuxDataRecordBuilderto copy an image onto the sealog server before creating an aux data record, you would probably want to use aDeleteFilesAuxDataFileCleanerto delete that image from the sealog server before deleting the corresponding aux data recordaux_data_inserters intoaux_data_managers and updating event aux data deletion logic to match event aux data creation logicdeleteEventswebsocket message instead of deleting aux data straight from the db. Theaux_data_managers are subscribed to these messages and will trigger theAuxDataFileCleaners before sending a request to the event_aux_data API to delete the dataOET is using this code in our dev environment (with some other managers/builders/cleaners that I did not include since they are pretty OET-specific) and plan on using it in production once the season starts. I thought I would create a pull request in case it's helpful for anyone else.