Use structlog and store JSON logs on disk

[paul-nechifor]

• I think we should be storing detailed logs on disk. If people have issues we'll be able to tell them to give us the stored logs.
• This PR uses structlog. It stores JSON logs on disk and colored logs on the console. The JSON logs include more data (currently just func_name and lineno, but we can add more.
• It allows us to store logs more logically and easy to analyze. For example, instead of

logger.info(
    f"RecoveryServer initialized with position_threshold={position_threshold}, "
    f"stuck_duration={stuck_duration}"
)

we can use:

logger.info(
    "RecoveryServer initialized",
    position_threshold=position_threshold,
    stuck_duration=stuck_duration,
)

---

Console logs look like this:

JSON logs get stored like this:

{"module": "SpatialMemory", "worker_id": 1, "event": "Deployed module.", "level": "info", "logger": "dimos/core/__init__.py", "timestamp": "2025-10-31T03:13:40.707846Z", "func_name": "deploy", "lineno": 106}
{"module": "ObjectTracking", "worker_id": 4, "event": "Deployed module.", "level": "info", "logger": "dimos/core/__init__.py", "timestamp": "2025-10-31T03:13:42.262636Z", "func_name": "deploy", "lineno": 106}
{"module": "UtilizationModule", "worker_id": 6, "event": "Deployed module.", "level": "info", "logger": "dimos/core/__init__.py", "timestamp": "2025-10-31T03:13:42.291749Z", "func_name": "deploy", "lineno": 106}
{"event": "Local planner module initialized", "level": "info", "logger": "dimos/navigation/local_planner/local_planner.py", "timestamp": "2025-10-31T03:13:35.151063Z", "func_name": "__init__", "lineno": 89}
{"event": "WavefrontFrontierExplorer module initialized", "level": "info", "logger": "dimos/navigation/frontier_exploration/wavefront_frontier_goal_selector.py", "timestamp": "2025-10-31T03:13:35.239683Z", "func_name": "__init__", "lineno": 151}
{"event": "Creating new ChromaDB database (new_memory=True)", "level": "info", "logger": "dimos/perception/spatial_perception.py", "timestamp": "2025-10-31T03:13:35.462126Z", "func_name": "__init__", "lineno": 121}
{"event": "Removed existing ChromaDB files from /home/p/pro/dimensional/dimos/assets/output/memory/spatial_memory/chromadb_data", "level": "info", "logger": "dimos/perception/spatial_perception.py", "timestamp": "2025-10-31T03:13:35.462528Z", "func_name": "__init__", "lineno": 131}
{"event": "Creating new visual memory", "level": "info", "logger": "dimos/perception/spatial_perception.py", "timestamp": "2025-10-31T03:13:35.593380Z", "func_name": "__init__", "lineno": 146}
{"event": "VisualMemory initialized with output directory: /home/p/pro/dimensional/dimos/assets/output/memory/spatial_memory", "level": "info", "logger": "dimos/agents/memory/visual_memory.py", "timestamp": "2025-10-31T03:13:35.593647Z", "func_name": "__init__", "lineno": 52}

Exceptions get logged on the console as:

...and in the JSON file as:

{"exception_type": "ValueError", "exception_message": "showing an example", "traceback_lines": ["Traceback (most recent call last):\n", "  File \"/home/p/pro/dimensional/dimos/.venv/bin/dimos-robot\", line 8, in <module>\n    sys.exit(main())\n             ^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/main.py\", line 325, in __call__\n    raise e\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/main.py\", line 308, in __call__\n    return get_command(self)(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 1462, in __call__\n    return self.main(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/core.py\", line 784, in main\n    return _main(\n           ^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/core.py\", line 192, in _main\n    rv = self.invoke(ctx)\n         ^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 1850, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 1246, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 814, in invoke\n    return callback(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/main.py\", line 685, in wrapper\n    return callback(**use_params)\n           ^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/dimos/robot/cli/dimos_robot.py\", line 120, in run\n    dimos = blueprint.build(global_config=config)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/dimos/core/blueprints.py\", line 127, in build\n    module_coordinator.deploy(blueprint.module, *blueprint.args, **kwargs)\n", "  File \"/home/p/pro/dimensional/dimos/dimos/core/module_coordinator.py\", line 55, in deploy\n    module = self._client.deploy(module_class, *args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/dimos/core/__init__.py\", line 103, in deploy\n    ).result()\n      ^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/distributed/client.py\", line 392, in result\n    return self.client.sync(self._result, callback_timeout=timeout)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", "  File \"/home/p/pro/dimensional/dimos/dimos/utils/monitoring.py\", line 154, in __init__\n    raise ValueError(\"showing an example\")\n    ^^^^^^^^^^^^^^^^^\n", "ValueError: showing an example\n"], "event": "Uncaught exception occurred", "level": "error", "logger": "uncaught_exception", "timestamp": "2025-10-31T04:04:39.630563Z", "func_name": "handle_exception", "lineno": 186, "exception": "Traceback (most recent call last):\n  File \"/home/p/pro/dimensional/dimos/.venv/bin/dimos-robot\", line 8, in <module>\n    sys.exit(main())\n             ^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/main.py\", line 325, in __call__\n    raise e\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/main.py\", line 308, in __call__\n    return get_command(self)(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 1462, in __call__\n    return self.main(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/core.py\", line 784, in main\n    return _main(\n           ^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/core.py\", line 192, in _main\n    rv = self.invoke(ctx)\n         ^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 1850, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 1246, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/click/core.py\", line 814, in invoke\n    return callback(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/typer/main.py\", line 685, in wrapper\n    return callback(**use_params)\n           ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/dimos/robot/cli/dimos_robot.py\", line 120, in run\n    dimos = blueprint.build(global_config=config)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/dimos/core/blueprints.py\", line 127, in build\n    module_coordinator.deploy(blueprint.module, *blueprint.args, **kwargs)\n  File \"/home/p/pro/dimensional/dimos/dimos/core/module_coordinator.py\", line 55, in deploy\n    module = self._client.deploy(module_class, *args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/dimos/core/__init__.py\", line 103, in deploy\n    ).result()\n      ^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/.venv/lib/python3.12/site-packages/distributed/client.py\", line 392, in result\n    return self.client.sync(self._result, callback_timeout=timeout)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/p/pro/dimensional/dimos/dimos/utils/monitoring.py\", line 154, in __init__\n    raise ValueError(\"showing an example\")\n    ^^^^^^^^^^^^^^^^^\nValueError: showing an example"}

Greptile Overview

Greptile Summary

This PR migrates the logging system from colorlog to structlog, enabling structured JSON logging to disk while maintaining colored console output.

Key Changes:

• Replaced colorlog with structlog for structured logging
• JSON logs stored in logs/dimos_{timestamp}_{pid}.jsonl with rotation (10MB max, 20 backups)
• Console logs remain colored but exclude metadata like func_name and lineno
• JSON logs include full metadata: timestamp, logger name, function name, line number, and structured key-value pairs
• Changed setup_logger() signature to auto-detect logger name from caller's filename instead of requiring explicit name parameter
• Added global exception handler that logs uncaught exceptions to both JSON and console (with Rich formatting)
• Updated all 131 files that use logging to call setup_logger() without name argument

Critical Issue Found:

• Line 203 in dimos/utils/logging_config.py calls setup_logger("uncaught_exception") but the new signature only accepts level: int | None - this will cause errors when any uncaught exception occurs

Confidence Score: 2/5

• This PR has a critical bug that will cause runtime failure when exceptions occur
• The logging refactoring is well-structured and the migration to structlog is clean across 131 files, but line 203 in dimos/utils/logging_config.py calls setup_logger("uncaught_exception") when the function signature was changed to only accept level: int | None. This will cause a TypeError or misinterpret the string as a level parameter when any uncaught exception occurs.
• dimos/utils/logging_config.py - line 203 must be fixed before merge

Important Files Changed

File Analysis

Filename	Score	Overview
dimos/utils/logging_config.py	2/5	Complete rewrite from colorlog to structlog with JSON file logging; CRITICAL BUG on line 203 - passes string to setup_logger() which expects int or None
pyproject.toml	5/5	Added structlog dependency (>=25.5.0,<26)
dimos/core/init.py	5/5	Changed setup_logger() call signature (removed name argument) and updated print statement to logger.info() with structured data
dimos/robot/cli/dimos.py	5/5	Added setup_exception_handler() call at start of run command
dimos/constants.py	5/5	Added DIMOS_LOG_DIR constant pointing to logs directory
.gitignore	5/5	Added /logs/ directory to gitignore

Sequence Diagram

sequenceDiagram
    participant App as Application Code
    participant SetupLogger as setup_logger
    participant Configure as configure_structlog
    participant Structlog as structlog
    participant StdLib as stdlib.logging
    participant Console as Console Handler
    participant File as File Handler
    
    App->>SetupLogger: Call with optional level
    SetupLogger->>SetupLogger: Get caller filename from stack
    SetupLogger->>SetupLogger: Convert to relative path
    SetupLogger->>Configure: Request configuration
    
    alt First call
        Configure->>Configure: Generate log file path
        Configure->>Structlog: Configure processors
        Configure-->>SetupLogger: Return log path
    else Cached
        Configure-->>SetupLogger: Return cached path
    end
    
    SetupLogger->>StdLib: Get or create logger
    SetupLogger->>StdLib: Clear existing handlers
    SetupLogger->>Console: Add colored console handler
    SetupLogger->>File: Add JSON file handler
    SetupLogger->>Structlog: Get configured logger
    Structlog-->>App: Return logger instance
    
    App->>Structlog: Log message with structured data
    Structlog->>Console: Colored console output
    Structlog->>File: JSON file output

Loading

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[leshy]

you know better probably - do we even need this at all?

setup_logger(__file__) 

I figure should be implicit given we know not only the file but the actual function that calls the .log?

is it difficult to get
from dimos import logger
?

[leshy]

for next steps we can use bind_contextvars with module info (worker, module name) and make logger available on self for modules, I can do this pass

[paul-nechifor]

@leshy

I've changed it to remove __file__ and infer it from the callee.

I don't think it's a good idea to use from dimos import logger. Internally it's better to full paths to avoid circular dependencies.

[leshy]

very good and non contraversial

[spomichter]

@greptile

[greptile-apps]

_{131 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}