fix: added graceful shutdown and entrypoint wrapper#1319
Merged
Conversation
github-actions
Bot
added
backend
lucaseduoli
changed the title
mpawlow
requested changes
Apr 1, 2026
| - opensearch-data:/usr/share/opensearch/data | ||
| stop_grace_period: 2m | ||
| healthcheck: | ||
| test: ["CMD-SHELL", "curl -ks https://localhost:9200 >/dev/null 2>&1 || exit 1"] |
Collaborator
There was a problem hiding this comment.
(b) [Normal] Healthcheck Does Not Verify OpenSearch Authentication
Problem
- The Docker Compose healthcheck uses an unauthenticated
curlrequest:test: ["CMD-SHELL", "curl -ks https://localhost:9200 >/dev/null 2>&1 || exit 1"]
curlexits0for any HTTP response, including401 Unauthorized- After the security plugin is configured, the root endpoint returns
401- The healthcheck still reports the container as healthy in this case
- Services using
depends_onwithcondition: service_healthywould start even if security is not yet correctly configured- This can cause authentication failures at runtime
Solutions
- (Recommended) Include credentials and assert a successful response:
test: ["CMD-SHELL", "curl -ku admin:$$OPENSEARCH_PASSWORD https://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s -s | grep -q '\"status\"'"]
- Use
-w "%{http_code}"to assert the HTTP status code explicitly rather than relying on curl's exit code. - Fall back to TCP-only checking (
nc -z localhost 9200) but document the limitation that it does not validate auth.
Issue - #1170 Summary - Improved OpenSearch graceful shutdown reliability by flushing pending writes, adding a force-kill fallback in the entrypoint wrapper, and preventing double-close of the client connection. - Hardened the Docker Compose healthcheck to verify authenticated cluster health status rather than bare connectivity. OpenSearch Shutdown Improvements - Replaced `cluster.health()` call with `indices.flush(index="_all", wait_if_ongoing=True)` in `graceful_opensearch_shutdown` to ensure pending write operations are persisted before the client closes. - Set `clients.opensearch = None` after graceful shutdown in `src/main.py` to prevent a redundant double-close during `clients.cleanup()`. - Added a 90-second wait loop with `kill -0` polling in `opensearch-entrypoint-wrapper.sh` before issuing a force `SIGKILL`, ensuring the process has time to stop cleanly before being forcibly terminated. - Removed stale "Made with Bob" comment from `opensearch-entrypoint-wrapper.sh`. Docker Compose Healthcheck - Updated the OpenSearch healthcheck command to authenticate with `admin:$OPENSEARCH_PASSWORD` and query `/_cluster/health`, asserting the cluster status is `green` or `yellow` rather than only checking for a successful TCP connection.
github-actions
Bot
added
bug
mpawlow
reviewed
Apr 2, 2026
mpawlow
reviewed
Apr 2, 2026
github-actions
Bot
added
bug
2 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces improvements to the OpenSearch container orchestration and application shutdown process, focusing on more reliable startup, graceful shutdown, and better operational health management. The most significant changes include the addition of a custom OpenSearch entrypoint wrapper for graceful shutdown, enhancements to the Docker Compose setup for health checks and persistent data, and the implementation of a graceful OpenSearch client shutdown in the application.
OpenSearch Container Entrypoint and Lifecycle Management:
opensearch-entrypoint-wrapper.shscript that starts OpenSearch, applies security setup after a delay, and handles graceful shutdown signals to ensure OpenSearch stops cleanly. (opensearch-entrypoint-wrapper.sh,Dockerfile) [1] [2] [3]opensearch-data), and improved stop behavior with a grace period. (docker-compose.yml) [1] [2]Application Shutdown Improvements:
graceful_opensearch_shutdowninutils/opensearch_utils.pyto ensure all operations complete and connections close properly on app shutdown. (src/utils/opensearch_utils.py)src/main.py)Code Quality and Minor Fixes:
src/api/settings.py) [1] [2]openrag-backendin favor of building from source. (docker-compose.yml)Closes #1170