Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/CLI_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,22 @@ If you want to remove those guards manually (optional):
- `--log-level` (CLI flag): Controls logging verbosity
- `DEBUG_LEVEL` (config): Controls operation detail level (`standard`/`advanced`/`extreme`)

### Log Labels (PHASE/STEP/SKIP)

Some log lines use a label to make the output easier to scan:

| Label | Level | Meaning |
|-------|-------|---------|
| `PHASE` | `info` | High-level workflow phase marker |
| `STEP` | `info` | A notable step within a phase |
| `SKIP` | `info` | Optional item intentionally skipped or not applicable |

**Common `SKIP` examples**:
- A feature is disabled by configuration.
- A non-critical CLI tool is not installed.
- Running in an **unprivileged container/rootless** environment where low-level inventory commands are expected to fail (for example `dmidecode` or `blkid`). In this case, ProxSave still attempts the collection, but logs a `SKIP` (not a `WARNING`) when the failure matches known “missing privileges” patterns.
- For `blkid`, the skip reason also includes a restore hint: `/etc/fstab` remap may be limited.

### Flag Reference

| Flag | Short | Description |
Expand Down
2 changes: 1 addition & 1 deletion docs/RESTORE_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1922,7 +1922,7 @@ If the restore includes filesystem configuration (notably `/etc/fstab`), ProxSav
- Compares the current `/etc/fstab` with the backup copy.
- Keeps existing critical entries (for example, root and swap) when they already match the running system.
- Detects **safe mount candidates** from the backup (for example, additional NFS mounts) and offers to add them.
- If ProxSave inventory data is present in the backup, ProxSave can remap **unstable** `/dev/*` devices from the backup (for example `/dev/sdb1`) to stable `UUID=`/`PARTUUID=`/`LABEL=` references **on the restore host** (only when the stable reference exists on the system).
- If ProxSave inventory data is present in the backup, ProxSave can remap **unstable** `/dev/*` devices from the backup (for example `/dev/sdb1`) to stable `UUID=`/`PARTUUID=`/`LABEL=` references **on the restore host** (only when the stable reference exists on the system). Note: backups taken from an **unprivileged container/rootless** environment may not include usable block-device inventory, so automated remap can be limited/unavailable.
- Normalizes restored entries by adding `nofail` (and `_netdev` for network mounts) so offline storage does not block boot/restore.

**Safety behavior**:
Expand Down
1 change: 1 addition & 0 deletions docs/RESTORE_TECHNICAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -1510,6 +1510,7 @@ When restoring to the real system root (`/`), ProxSave avoids blindly overwritin
- If the backup contains ProxSave inventory (`var/lib/proxsave-info/commands/system/{blkid.txt,lsblk_json.json,lsblk.txt}` or PBS datastore inventory),
ProxSave can remap unstable device paths from the backup (e.g. `/dev/sdb1`) to stable references (`UUID=`/`PARTUUID=`/`LABEL=`) **when the stable reference exists on the restore host**.
- This reduces the risk of mounting the wrong disk after a reinstall where `/dev/sdX` ordering changes.
- Note: backups taken from an **unprivileged container/rootless** environment may not include usable block-device inventory (for example `blkid` output can be empty/skipped). In that case, automated device remap is limited/unavailable and `/etc/fstab` entries may require manual review during restore.

**Normalization**:
- Entries written by the merge are normalized to include `nofail` (and `_netdev` for network mounts) to prevent offline storage from blocking boot/restore.
Expand Down
29 changes: 29 additions & 0 deletions docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,35 @@ COMPRESSION_TYPE=xz # Valid: xz, zstd, gzip, bzip2, lz4

---

#### Notice: `SKIP ... Expected in unprivileged containers` (LXC/rootless)

**Symptoms**:
- Running ProxSave inside an **unprivileged** LXC container (or a rootless container) produces log lines like:
- `SKIP Skipping Hardware DMI information: DMI tables not accessible (Expected in unprivileged containers).`
- `SKIP Skipping Block device identifiers (blkid): block devices not accessible (restore hint: fstab remap may be limited) (Expected in unprivileged containers).`

**Cause**: In unprivileged containers, access to low-level system interfaces is intentionally restricted (for example `/dev/mem` and most block devices). Some inventory commands can fail even though the backup itself is working correctly.

**Behavior**:
- ProxSave still attempts the collection.
- Only a small allowlist of **privilege-sensitive** commands is downgraded from `WARNING` to `SKIP` when failure is expected in this environment (`dmidecode`, `blkid`, `sensors`, `smartctl`).
- Other failures are **not** downgraded and still appear as warnings/errors.

**Impact**:
- Hardware inventory output may be missing/empty.
- If `blkid` is skipped, ProxSave restore may have **limited** ability to automatically remap `/etc/fstab` devices (UUID/PARTUUID/LABEL). You may need to review mounts manually during restore.

**How to verify** (shifted user namespace mapping):
```bash
cat /proc/self/uid_map
cat /proc/self/gid_map
# If the second column is non-zero (e.g. "0 100000 65536"), you're in a shifted/unprivileged mapping.
```

**Optional**: If you want to hide `SKIP` lines on the console, run with `--log-level warning` (this also hides normal info logs).

---

### 3. Cloud Storage Issues

#### Error: `rclone not found in PATH`
Expand Down
85 changes: 76 additions & 9 deletions internal/backup/collector.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,16 @@ type FileSummary struct {

// Collector handles backup data collection
type Collector struct {
logger *logging.Logger
config *CollectorConfig
stats *CollectionStats
statsMu sync.Mutex
tempDir string
proxType types.ProxmoxType
dryRun bool
deps CollectorDeps
logger *logging.Logger
config *CollectorConfig
stats *CollectionStats
statsMu sync.Mutex
tempDir string
proxType types.ProxmoxType
dryRun bool
deps CollectorDeps
unprivilegedOnce sync.Once
unprivilegedCtx unprivilegedContainerContext

// clusteredPVE records whether cluster mode was detected during PVE collection.
clusteredPVE bool
Expand Down Expand Up @@ -915,11 +917,47 @@ func (c *Collector) safeCmdOutput(ctx context.Context, cmd, output, description
c.incFilesFailed()
return fmt.Errorf("critical command `%s` failed for %s: %w (output: %s)", cmdString, description, err, summarizeCommandOutputText(string(out)))
}
exitCode := -1
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
exitCode = exitErr.ExitCode()
}
outputText := strings.TrimSpace(string(out))

c.logger.Debug("Non-critical command failed (safeCmdOutput): description=%q cmd=%q exitCode=%d err=%v", description, cmdString, exitCode, err)
c.logger.Debug("Non-critical command output summary (safeCmdOutput): %s", summarizeCommandOutputText(outputText))

ctxInfo := c.depDetectUnprivilegedContainer()
c.logger.Debug("Unprivileged context evaluation: detected=%t details=%q", ctxInfo.Detected, strings.TrimSpace(ctxInfo.Details))

reason := ""
if ctxInfo.Detected {
c.logger.Debug("Privilege-sensitive allowlist: command=%q allowlisted=%t", cmdParts[0], isPrivilegeSensitiveCommand(cmdParts[0]))
match := privilegeSensitiveFailureMatch(cmdParts[0], exitCode, outputText)
reason = match.Reason
c.logger.Debug("Privilege-sensitive classification: command=%q matched=%t match=%q reason=%q", cmdParts[0], reason != "", match.Match, reason)
Comment on lines +936 to +938
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exitCode is set to -1 when err is not an *exec.ExitError (e.g., context cancellation/timeout). In that case, privilegeSensitiveFailureMatch() can still match dmidecode via exitCode != 0 && outputText == "", causing timeouts/cancellations to be incorrectly downgraded to SKIP. Consider passing an additional flag indicating whether the exit code is known, or only applying the exit!=0 && empty output heuristic when errors.As(err, *exec.ExitError) succeeds (or when exitCode >= 0).

Suggested change
match := privilegeSensitiveFailureMatch(cmdParts[0], exitCode, outputText)
reason = match.Reason
c.logger.Debug("Privilege-sensitive classification: command=%q matched=%t match=%q reason=%q", cmdParts[0], reason != "", match.Match, reason)
if exitCode >= 0 {
match := privilegeSensitiveFailureMatch(cmdParts[0], exitCode, outputText)
reason = match.Reason
c.logger.Debug("Privilege-sensitive classification: command=%q matched=%t match=%q reason=%q", cmdParts[0], reason != "", match.Match, reason)
} else {
c.logger.Debug("Privilege-sensitive classification skipped: unknown or unavailable exit code (command=%q exitCode=%d)", cmdParts[0], exitCode)
}

Copilot uses AI. Check for mistakes.
} else {
c.logger.Debug("Privilege-sensitive downgrade not considered: unprivileged context not detected (command=%q)", cmdParts[0])
}

if ctxInfo.Detected && reason != "" {
c.logger.Debug("Downgrading WARNING->SKIP: description=%q cmd=%q exitCode=%d", description, cmdString, exitCode)

c.logger.Skip("Skipping %s: %s (Expected in unprivileged containers).", description, reason)
c.logger.Debug("SKIP context (privilege-sensitive): description=%q cmd=%q exitCode=%d err=%v unprivilegedDetails=%q", description, cmdString, exitCode, err, strings.TrimSpace(ctxInfo.Details))
c.logger.Debug("SKIP output summary for %s: %s", description, summarizeCommandOutputText(outputText))
return nil
}

if ctxInfo.Detected {
c.logger.Debug("No privilege-sensitive downgrade applied: command=%q did not match known patterns; emitting WARNING", cmdParts[0])
}

Comment on lines +930 to +955
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The privilege-sensitive downgrade/logging block is duplicated in both safeCmdOutput and captureCommandOutput (context detection, matching, SKIP logging). This duplication increases the risk of the two paths drifting (e.g., new patterns or fixes applied to only one). Consider extracting a small helper that takes (command, exitCode, outputText, description, err) and returns whether to downgrade + the reason/match.

Suggested change
ctxInfo := c.depDetectUnprivilegedContainer()
c.logger.Debug("Unprivileged context evaluation: detected=%t details=%q", ctxInfo.Detected, strings.TrimSpace(ctxInfo.Details))
reason := ""
if ctxInfo.Detected {
c.logger.Debug("Privilege-sensitive allowlist: command=%q allowlisted=%t", cmdParts[0], isPrivilegeSensitiveCommand(cmdParts[0]))
match := privilegeSensitiveFailureMatch(cmdParts[0], exitCode, outputText)
reason = match.Reason
c.logger.Debug("Privilege-sensitive classification: command=%q matched=%t match=%q reason=%q", cmdParts[0], reason != "", match.Match, reason)
} else {
c.logger.Debug("Privilege-sensitive downgrade not considered: unprivileged context not detected (command=%q)", cmdParts[0])
}
if ctxInfo.Detected && reason != "" {
c.logger.Debug("Downgrading WARNING->SKIP: description=%q cmd=%q exitCode=%d", description, cmdString, exitCode)
c.logger.Skip("Skipping %s: %s (Expected in unprivileged containers).", description, reason)
c.logger.Debug("SKIP context (privilege-sensitive): description=%q cmd=%q exitCode=%d err=%v unprivilegedDetails=%q", description, cmdString, exitCode, err, strings.TrimSpace(ctxInfo.Details))
c.logger.Debug("SKIP output summary for %s: %s", description, summarizeCommandOutputText(outputText))
return nil
}
if ctxInfo.Detected {
c.logger.Debug("No privilege-sensitive downgrade applied: command=%q did not match known patterns; emitting WARNING", cmdParts[0])
}
// Centralized privilege-sensitive downgrade evaluation.
shouldDowngrade, reason, unprivDetails := c.evaluatePrivilegeSensitiveDowngrade(
cmdParts[0],
exitCode,
outputText,
description,
err,
"safeCmdOutput",
)
if shouldDowngrade {
c.logger.Debug("Downgrading WARNING->SKIP: description=%q cmd=%q exitCode=%d", description, cmdString, exitCode)
c.logger.Skip("Skipping %s: %s (Expected in unprivileged containers).", description, reason)
c.logger.Debug("SKIP context (privilege-sensitive): description=%q cmd=%q exitCode=%d err=%v unprivilegedDetails=%q", description, cmdString, exitCode, err, unprivDetails)
c.logger.Debug("SKIP output summary for %s: %s", description, summarizeCommandOutputText(outputText))
return nil
}

Copilot uses AI. Check for mistakes.
c.logger.Warning("Skipping %s: command `%s` failed (%v). Non-critical; backup continues. Ensure the required CLI is available and has proper permissions. Output: %s",
description,
cmdString,
err,
summarizeCommandOutputText(string(out)),
summarizeCommandOutputText(outputText),
)
return nil // Non-critical failure
}
Expand Down Expand Up @@ -1236,6 +1274,35 @@ func (c *Collector) captureCommandOutput(ctx context.Context, cmd, output, descr
}
outputText := strings.TrimSpace(string(out))

c.logger.Debug("Non-critical command failed (captureCommandOutput): description=%q cmd=%q exitCode=%d err=%v", description, cmdString, exitCode, err)
c.logger.Debug("Non-critical command output summary (captureCommandOutput): %s", summarizeCommandOutputText(outputText))

ctxInfo := c.depDetectUnprivilegedContainer()
c.logger.Debug("Unprivileged context evaluation: detected=%t details=%q", ctxInfo.Detected, strings.TrimSpace(ctxInfo.Details))

reason := ""
if ctxInfo.Detected {
c.logger.Debug("Privilege-sensitive allowlist: command=%q allowlisted=%t", parts[0], isPrivilegeSensitiveCommand(parts[0]))
match := privilegeSensitiveFailureMatch(parts[0], exitCode, outputText)
reason = match.Reason
c.logger.Debug("Privilege-sensitive classification: command=%q matched=%t match=%q reason=%q", parts[0], reason != "", match.Match, reason)
} else {
c.logger.Debug("Privilege-sensitive downgrade not considered: unprivileged context not detected (command=%q)", parts[0])
}

if ctxInfo.Detected && reason != "" {
c.logger.Debug("Downgrading WARNING->SKIP: description=%q cmd=%q exitCode=%d", description, cmdString, exitCode)

c.logger.Skip("Skipping %s: %s (Expected in unprivileged containers).", description, reason)
c.logger.Debug("SKIP context (privilege-sensitive): description=%q cmd=%q exitCode=%d err=%v unprivilegedDetails=%q", description, cmdString, exitCode, err, strings.TrimSpace(ctxInfo.Details))
c.logger.Debug("SKIP output summary for %s: %s", description, summarizeCommandOutputText(outputText))
return nil, nil
}

if ctxInfo.Detected {
c.logger.Debug("No privilege-sensitive downgrade applied: command=%q did not match known patterns; continuing with standard handling", parts[0])
}

if parts[0] == "systemctl" && len(parts) >= 2 && parts[1] == "status" {
unit := parts[len(parts)-1]
if exitCode == 4 || strings.Contains(outputText, "could not be found") {
Expand Down
9 changes: 5 additions & 4 deletions internal/backup/collector_deps.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,11 @@ var (

// CollectorDeps allows injecting external dependencies for the Collector.
type CollectorDeps struct {
LookPath func(string) (string, error)
RunCommandWithEnv func(context.Context, []string, string, ...string) ([]byte, error)
RunCommand func(context.Context, string, ...string) ([]byte, error)
Stat func(string) (os.FileInfo, error)
LookPath func(string) (string, error)
RunCommandWithEnv func(context.Context, []string, string, ...string) ([]byte, error)
RunCommand func(context.Context, string, ...string) ([]byte, error)
Stat func(string) (os.FileInfo, error)
DetectUnprivilegedContainer func() (bool, string)
}

func defaultCollectorDeps() CollectorDeps {
Expand Down
89 changes: 89 additions & 0 deletions internal/backup/collector_privilege_sensitive.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
package backup

import "strings"

type privilegeSensitiveMatch struct {
Reason string
Match string
}

func isPrivilegeSensitiveCommand(command string) bool {
switch strings.TrimSpace(command) {
case "dmidecode", "blkid", "sensors", "smartctl":
return true
default:
return false
}
}

func privilegeSensitiveFailureMatch(command string, exitCode int, outputText string) privilegeSensitiveMatch {
command = strings.TrimSpace(command)
if command == "" {
return privilegeSensitiveMatch{}
}

outputText = strings.TrimSpace(outputText)
lower := strings.ToLower(outputText)

switch command {
case "dmidecode":
// dmidecode typically fails due to restricted access to DMI tables (/sys/firmware/dmi or /dev/mem).
switch {
case strings.Contains(lower, "/dev/mem"):
return privilegeSensitiveMatch{Reason: "DMI tables not accessible", Match: "stderr contains /dev/mem"}
case strings.Contains(lower, "permission denied"):
return privilegeSensitiveMatch{Reason: "DMI tables not accessible", Match: "stderr contains permission denied"}
case strings.Contains(lower, "operation not permitted"):
return privilegeSensitiveMatch{Reason: "DMI tables not accessible", Match: "stderr contains operation not permitted"}
case exitCode != 0 && outputText == "":
return privilegeSensitiveMatch{Reason: "DMI tables not accessible", Match: "exit!=0 and empty output"}
}
return privilegeSensitiveMatch{}
case "blkid":
// In unprivileged LXC, blkid often exits 2 with empty output when block devices are not accessible.
const blkidReason = "block devices not accessible (restore hint: fstab remap may be limited)"
switch {
case exitCode == 2 && outputText == "":
return privilegeSensitiveMatch{
Reason: blkidReason,
Match: "exit=2 and empty output",
}
case strings.Contains(lower, "permission denied"):
return privilegeSensitiveMatch{
Reason: blkidReason,
Match: "stderr contains permission denied",
}
case strings.Contains(lower, "operation not permitted"):
return privilegeSensitiveMatch{
Reason: blkidReason,
Match: "stderr contains operation not permitted",
}
}
return privilegeSensitiveMatch{}
case "sensors":
// In containers, sensors may not be available or may report no sensors found.
switch {
case strings.Contains(lower, "permission denied"):
return privilegeSensitiveMatch{Reason: "hardware sensors not accessible", Match: "stderr contains permission denied"}
case strings.Contains(lower, "operation not permitted"):
return privilegeSensitiveMatch{Reason: "hardware sensors not accessible", Match: "stderr contains operation not permitted"}
case strings.Contains(lower, "no sensors found"):
return privilegeSensitiveMatch{Reason: "hardware sensors not accessible", Match: "stderr contains no sensors found"}
}
return privilegeSensitiveMatch{}
case "smartctl":
switch {
case strings.Contains(lower, "permission denied"):
return privilegeSensitiveMatch{Reason: "SMART devices not accessible", Match: "stderr contains permission denied"}
case strings.Contains(lower, "operation not permitted"):
return privilegeSensitiveMatch{Reason: "SMART devices not accessible", Match: "stderr contains operation not permitted"}
}
return privilegeSensitiveMatch{}
default:
return privilegeSensitiveMatch{}
}
}

func privilegeSensitiveFailureReason(command string, exitCode int, outputText string) string {
return privilegeSensitiveFailureMatch(command, exitCode, outputText).Reason
}
Loading
Loading