feature: logging redesign

README.md

@@ -143,7 +143,9 @@ conductor run <workflow.yaml> [OPTIONS]
 | `-p, --provider PROVIDER` | Override provider |
 | `--dry-run` | Preview execution plan |
 | `--skip-gates` | Auto-select at human gates |
-| `-V, --verbose` | Show detailed progress |
+| `-q, --quiet` | Suppress progress output |
+| `-s, --silent` | Suppress all output except errors |
+| `-l, --log-file PATH` | Write logs to file |
 
 ### `conductor validate`
 

docs/cli-reference.md

@@ -26,7 +26,9 @@ conductor run <workflow.yaml> [OPTIONS]
 | `--provider PROVIDER` | `-p` | Override provider (copilot, claude) |
 | `--dry-run` | | Show execution plan without running |
 | `--skip-gates` | | Auto-select first option at human gates |
-| `--verbose` | `-V` | Show detailed execution progress |
+| `--quiet` | `-q` | Minimal output (agent lifecycle and routing only) |
+| `--silent` | `-s` | No progress output (JSON result only) |
+| `--log-file <auto\|PATH>` | `-l` | Write full debug output to a file |
 
 ### Examples
 
@@ -59,11 +61,11 @@ conductor run workflow.yaml -p copilot
 # Preview execution plan without running
 conductor run workflow.yaml --dry-run
 
-# Verbose output for debugging
-conductor -V run workflow.yaml --input question="Test"
+# Quiet output (agent lifecycle only)
+conductor run workflow.yaml --quiet --input question="Test"
 
-# Combine dry-run with verbose
-conductor -V run workflow.yaml --dry-run
+# Write full debug log to a file
+conductor run workflow.yaml --log-file debug.log
 ```
 
 #### Automation Mode
@@ -72,8 +74,8 @@ conductor -V run workflow.yaml --dry-run
 # Skip human gates (auto-select first option)
 conductor run workflow.yaml --skip-gates
 
-# Useful for CI/CD pipelines
-conductor run workflow.yaml --skip-gates --input question="Automated test"
+# CI/CD pattern: silent console + full file log
+conductor run workflow.yaml --silent --log-file auto --skip-gates --input question="Automated test"
 ```
 
 #### Complex Inputs

docs/dynamic-parallel.md

@@ -883,7 +883,7 @@ agents:
 1. Check agent prompt is valid for the item structure
 2. Verify items match expected schema
 3. Test with a small sample first
-4. Check verbose logs: `conductor run --verbose`
+4. Check verbose logs: `conductor run --log-file debug.log`
 
 ### Memory Issues with Large Arrays
 

docs/projects/planned-features.md

@@ -0,0 +1,151 @@
+# Planned Features
+
+## 1. Logging Redesign (Console + File Output)
+
+Replaces the current `--verbose`/`-V` flag with a cleaner two-dimensional model: console verbosity and file output are independent.
+
+### Console Output
+
+| Level | Flag | Behavior |
+|---|---|---|
+| **full** (default) | *(none)* | Untruncated prompts, tool args, timing, routing — everything |
+| **minimal** | `--quiet` / `-q` | Agent start/complete, routing decisions, timing — no prompt/tool detail |
+| **silent** | `--silent` / `-s` | No progress output — only final JSON result on stdout |
+
+### File Output
+
+| Mode | Flag | Behavior |
+|---|---|---|
+| **none** (default) | *(none)* | No file logging |
+| **auto** | `--log-file` / `-l` | Writes to `$TMPDIR/conductor/conductor-<workflow>-<timestamp>.log` |
+| **explicit** | `--log-file PATH` / `-l PATH` | Writes to specified path |
+
+File output is **always full/untruncated** regardless of console level. This enables CI usage like `--silent --log-file` for clean stdout with full debug log in a file.
+
+### Removed Flags
+
+- `--verbose` / `-V` — removed entirely (full output is now the default)
+
+### Implementation Notes
+
+- The existing `verbose_mode` and `full_mode` ContextVars in `src/conductor/cli/app.py` still work internally; the new flags just set them differently
+- File console uses `no_color=True` for plain text output
+- File console bypasses the 500-char truncation in `verbose_log_section()`
+- At workflow completion, print the log file path to stderr
+
+### Short Flag Summary
+
+| Flag | Short | Scope |
+|---|---|---|
+| `--version` | `-v` | global |
+| `--quiet` | `-q` | global |
+| `--silent` | `-s` | global |
+| `--log-file` | `-l` | run command |
+| `--provider` | `-p` | run command |
+| `--input` | `-i` | run command |
+| `--template` | `-t` | init command |
+| `--output` | `-o` | init command |
+
+---
+
+## 2. Async Stdin Input During Workflow Execution
+
+Allow users to type guidance into the terminal while a workflow is running. Input is captured asynchronously and injected into context for the next agent.
+
+### Design
+
+- Spawn a background asyncio task that reads stdin lines via `loop.run_in_executor(None, sys.stdin.readline)` into an `asyncio.Queue`
+- Between each agent step (after route evaluation, before next agent starts), drain the queue
+- Store user input in context under `_user_guidance` key, accessible to agents via `{{ _user_guidance }}`
+- Only activate when stdin is a TTY (`sys.stdin.isatty()`)
+- Display a subtle indicator at workflow start: "Type to provide guidance at any time"
+- Add `--no-interactive` flag to disable for CI/piped usage
+
+### Key Files
+
+- `src/conductor/engine/workflow.py` — queue integration in main `run()` loop (~L519)
+- `src/conductor/cli/run.py` — queue creation and stdin reader task in `run_workflow_async()`
+- `src/conductor/engine/context.py` — ensure `_user_guidance` included in `build_for_agent()`
+
+---
+
+## 3. `$file` Reference Resolution in YAML
+
+Allow any YAML field value to reference an external file using the `$file: path/to/file` pattern. Resolved during loading before Pydantic validation.
+
+### Syntax
+
+```yaml
+agents:
+  reviewer:
+    prompt: "$file: prompts/review-prompt.md"
+    tools:
+      - "$file: tools/review-tools.yaml"
+```
+
+### Design
+
+- Add `_resolve_file_refs_recursive(data, base_path)` in `src/conductor/config/loader.py`, following the same recursive dict-walking pattern as `_resolve_env_vars_recursive()`
+- Runs **after** env var resolution so paths can contain `${VAR}` references
+- Paths are relative to the parent YAML file's directory
+- If loaded content parses as a YAML dict/list, use the parsed structure; if scalar, use as raw string
+- Supports nested `$file` references (files referencing other files)
+- Cycle detection via tracked set of resolved absolute paths
+- For `load_string()`, uses `source_path.parent` if provided, otherwise CWD
+
+### Key Files
+
+- `src/conductor/config/loader.py` — new resolver function, called at ~L181 after env var resolution
+- `src/conductor/config/validator.py` — may need awareness of included files for cross-reference validation
+- `docs/workflow-syntax.md` — documentation
+
+---
+
+## 4. Script Execution Steps
+
+Add `type: script` as a new workflow step type that runs shell commands, captures stdout, and stores it in context like agent outputs.
+
+### YAML Syntax
+
+```yaml
+agents:
+  run-tests:
+    type: script
+    command: pytest
+    args: ["tests/", "--tb=short"]
+    env:
+      PYTHONPATH: ./src
+    working_dir: .
+    timeout: 300
+    routes:
+      - when: "{{ exit_code == 0 }}"
+        next: summarize-results
+      - next: fix-failures
+```
+
+### Design
+
+- Extend `AgentDef.type` to `Literal["agent", "human_gate", "script"]` in `src/conductor/config/schema.py`
+- Add fields: `command` (required for scripts), `args`, `env`, `working_dir`, `timeout`
+- Model validator: if `type == "script"`, `command` is required, `prompt`/`provider`/`model` are forbidden
+- Follow `MCPServerDef` pattern (~L415-L455 in schema.py) for command/args/env structure
+- Create `src/conductor/executor/script.py` with `ScriptExecutor` using `asyncio.create_subprocess_exec()`
+- Capture stdout as text output (not JSON-parsed)
+- `exit_code` exposed in route evaluation context
+- Jinja2 template rendering supported in `command` and `args` for context injection
+
+### Key Files
+
+- `src/conductor/config/schema.py` — schema changes
+- `src/conductor/executor/script.py` — new file
+- `src/conductor/engine/workflow.py` — dispatch logic in main loop (~L728-L735)
+- `src/conductor/config/validator.py` — validation for script steps
+
+---
+
+## Implementation Order
+
+1. **Logging Redesign** — smallest diff, foundational for everything else
+2. **`$file` References** — isolated to loader, well-scoped
+3. **Script Steps** — new executor + schema, moderate scope
+4. **Async Stdin** — most experimental, depends on logging being settled

src/conductor/cli/app.py

@@ -6,6 +6,7 @@
 from __future__ import annotations
 
 import contextvars
+from enum import Enum
 from pathlib import Path
 from typing import Annotated, Any
 
@@ -16,6 +17,15 @@
 
 from conductor import __version__
 
+
+class ConsoleVerbosity(str, Enum):
+    """Console output verbosity level."""
+
+    FULL = "full"  # Default: everything, untruncated
+    MINIMAL = "minimal"  # Agent lifecycle + routing + timing only
+    SILENT = "silent"  # No progress output at all
+
+
 # Create the main Typer app
 app = typer.Typer(
     name="conductor",
@@ -31,8 +41,13 @@
 # Context variable for verbose mode (default True - show progress output)
 verbose_mode: contextvars.ContextVar[bool] = contextvars.ContextVar("verbose_mode", default=True)
 
-# Context variable for full verbose mode (--verbose flag - show full details)
-full_mode: contextvars.ContextVar[bool] = contextvars.ContextVar("full_mode", default=False)
+# Context variable for full verbose mode (default True - show full details)
+full_mode: contextvars.ContextVar[bool] = contextvars.ContextVar("full_mode", default=True)
+
+# Context variable for console verbosity level
+console_verbosity: contextvars.ContextVar[ConsoleVerbosity] = contextvars.ContextVar(
+    "console_verbosity", default=ConsoleVerbosity.FULL
+)
 
 
 def is_verbose() -> bool:
@@ -41,10 +56,11 @@ def is_verbose() -> bool:
 
 
 def is_full() -> bool:
-    """Check if full verbose mode is enabled (--verbose flag).
+    """Check if full verbose mode is enabled.
 
-    When full mode is enabled, prompts are shown untruncated and
+    Full mode is the default. When enabled, prompts are shown untruncated and
     additional details like tool arguments and reasoning are displayed.
+    Use --quiet to disable full mode while keeping progress output.
     """
     return full_mode.get()
 
@@ -149,17 +165,35 @@ def main(
             is_eager=True,
         ),
     ] = False,
-    verbose: Annotated[
+    quiet: Annotated[
         bool,
         typer.Option(
-            "--verbose",
-            "-V",
-            help="Show full prompts and detailed tool call information.",
+            "--quiet",
+            "-q",
+            help="Minimal output: agent lifecycle and routing only.",
+        ),
+    ] = False,
+    silent: Annotated[
+        bool,
+        typer.Option(
+            "--silent",
+            "-s",
+            help="No progress output. Only JSON result on stdout.",
         ),
     ] = False,
 ) -> None:
     """Conductor - Orchestrate multi-agent workflows defined in YAML."""
-    full_mode.set(verbose)
+    if quiet and silent:
+        raise typer.BadParameter("--quiet and --silent are mutually exclusive")
+    if silent:
+        verbosity = ConsoleVerbosity.SILENT
+    elif quiet:
+        verbosity = ConsoleVerbosity.MINIMAL
+    else:
+        verbosity = ConsoleVerbosity.FULL
+    console_verbosity.set(verbosity)
+    verbose_mode.set(verbosity != ConsoleVerbosity.SILENT)
+    full_mode.set(verbosity == ConsoleVerbosity.FULL)
 
 
 @app.command()
@@ -205,6 +239,17 @@ def run(
             help="Auto-select first option at human gates (for automation).",
         ),
     ] = False,
+    log_file: Annotated[
+        str | None,
+        typer.Option(
+            "--log-file",
+            "-l",
+            help=(
+                "Write full debug output to a file. "
+                "Pass a file path or 'auto' for auto-generated temp file."
+            ),
+        ),
+    ] = None,
 ) -> None:
     """Run a workflow from a YAML file.
 
@@ -219,6 +264,9 @@ def run(
         conductor run workflow.yaml --provider copilot
         conductor run workflow.yaml --dry-run
         conductor run workflow.yaml --skip-gates
+        conductor run workflow.yaml --log-file auto
+        conductor run workflow.yaml --log-file debug.log
+        conductor run workflow.yaml --silent --log-file auto
     """
     import asyncio
     import json
@@ -228,6 +276,7 @@ def run(
         InputCollector,
         build_dry_run_plan,
         display_execution_plan,
+        generate_log_path,
         parse_input_flags,
         run_workflow_async,
     )
@@ -252,9 +301,19 @@ def run(
     # Also parse --input.name=value style from sys.argv
     inputs.update(InputCollector.extract_from_args())
 
+    # Resolve log file path
+    resolved_log_file: Path | None = None
+    if log_file is not None:
+        if log_file.lower() == "auto":
+            resolved_log_file = generate_log_path(workflow.stem)
+        else:
+            resolved_log_file = Path(log_file)
+
     try:
         # Run the workflow
-        result = asyncio.run(run_workflow_async(workflow, inputs, provider, skip_gates))
+        result = asyncio.run(
+            run_workflow_async(workflow, inputs, provider, skip_gates, resolved_log_file)
+        )
 
         # Output as JSON to stdout
         output_console.print_json(json.dumps(result))