Improve daemon resilience: handle task failures for listener and worker

[sourcery-ai]

Currently, both the listener and worker are spawned as tasks. If either of these tasks fails, the process may exit, which can reduce the resilience of the daemon. To improve robustness, we should implement error handling or task restarts so that the daemon can recover from individual task failures without exiting entirely.

Suggested actions:

• Add error handling for both listener and worker tasks.
• Consider implementing automatic task restarts if a task fails unexpectedly.
• Ensure that the daemon remains operational even if one of the tasks encounters an error.

This will help improve the overall reliability and uptime of the daemon process.

Issue: #7

---

I created this issue for @leynos from #4 (comment).

Tips and commands

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[leynos]

@coderabbitai looking at the codebase, is this issue still relevant? If the issue has been fixed in the code, does the documentation reflect this?

If it is still relevant, please provide an assessment of the impact of the required changes.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

I'll analyze the current codebase to determine if the daemon resilience issue is still relevant and check the documentation status.

---

🏁 Script executed:

#!/bin/bash
# First, let's understand the project structure and find daemon-related files
echo "=== Project structure ==="
fd -t f | head -20

echo -e "\n=== Looking for main daemon/server files ==="
fd -e rs -e py -e js -e ts | grep -E "(main|daemon|server|worker|listener)" | head -10

echo -e "\n=== Searching for task spawning and management code ==="
rg -n -A5 -B5 "spawn|task|tokio::spawn|async" --type rust

echo -e "\n=== Looking for error handling around tasks ==="
rg -n -A3 -B3 "error|Error|Result|unwrap|expect" --type rust

Length of output: 145749

---

Let me examine the current daemon implementation to assess if the resilience issue is still relevant.

---

🏁 Script executed:

#!/bin/bash
# Get the full daemon.rs file to understand current task management
echo "=== Current daemon implementation ==="
cat crates/comenqd/src/daemon.rs

echo -e "\n=== Check for any restart/retry mechanisms ==="
rg -n -A3 -B3 "restart|retry|recover|resilien" --type rust

echo -e "\n=== Look for documentation about resilience ==="
find . -name "*.md" -exec grep -l -i "resilien\|restart\|error.handling\|task.fail" {} \;

Length of output: 30812

---

Now let me check the documentation to see if resilience has been addressed.

---

🏁 Script executed:

#!/bin/bash
# Check specific documentation files for resilience mentions
echo "=== Checking README for daemon info ==="
cat README.md | grep -i -A3 -B3 "daemon\|resilien\|error\|fail"

echo -e "\n=== Check roadmap for resilience mentions ==="
cat docs/roadmap.md | grep -i -A5 -B5 "resilien\|restart\|error\|task"

echo -e "\n=== Check design document ==="
cat docs/comenq-design.md | grep -i -A5 -B5 "resilien\|restart\|error\|task\|fail"

echo -e "\n=== Check if issue #7 is still open ==="
gh issue list --state=all --search="resilience OR restart OR task OR daemon" --json number,title,state,url || echo "GitHub CLI not available or no access"

Length of output: 24009

---

@leynos Based on my analysis of the codebase, Issue #7 is still relevant and the daemon resilience issue has not been addressed.

Current State Analysis

Issue Still Exists

Looking at the current daemon implementation in crates/comenqd/src/daemon.rs, the resilience problem described in issue #7 is still present:

Lines 144-154 show the problematic behavior:

// Await either task; on completion, trigger a coordinated shutdown.
tokio::select! {
    res = &mut listener => match res {
        Ok(inner) => inner?,
        Err(e) => return Err(e.into()),
    },
    res = &mut worker => match res {
        Ok(inner) => inner?,
        Err(e) => return Err(e.into()),
    },
}

The daemon still has the exact problem described in issue #7:

• When either the listener or worker task completes (including on failure), the daemon immediately triggers shutdown
• There are no automatic restart mechanisms for failed tasks
• Task failures cause the entire daemon process to exit

Impact Assessment

The required changes would have moderate to high impact:

Code Changes Required:

1. Modify the main run function - Replace the current tokio::select! pattern that exits on first task completion
2. Implement task restart logic - Add mechanisms to detect task failures and restart individual tasks
3. Add error handling strategy - Decide between immediate restart vs. backoff strategies
4. Maintain shutdown coordination - Ensure graceful shutdown still works with the new restart mechanisms

Architecture Impact:

• Task lifecycle management - Need to handle task restart without losing shared resources (channels, configs, etc.)
• State management - Ensure clean state when restarting tasks
• Resource cleanup - Prevent resource leaks during task restarts
• Testing complexity - New restart logic needs comprehensive testing

Documentation Gap

The documentation mentions resilience features like "fault-tolerant service" and "automatic restarts on failure" in the README and design docs, but these refer to systemd-level process restarts, not internal task resilience.

Recommendation

Issue #7 remains highly relevant. The current implementation will cause the daemon to exit if either the listener or worker task encounters an unrecoverable error, reducing overall system reliability. Implementing task-level restart mechanisms would significantly improve daemon resilience without requiring external process management.

ƪ(˘⌣˘)ʃ