Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
285 changes: 80 additions & 205 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion dashboard/src/components/GraphTemplateDetailModal.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ export const GraphTemplateDetailModal: React.FC<GraphTemplateDetailModalProps> =
<div className="fixed inset-0 bg-white/30 backdrop-blur-sm flex items-center justify-center z-50">
<div className="bg-white rounded-lg shadow-xl max-w-6xl w-full mx-4 max-h-[90vh] overflow-y-auto">
{/* Header */}
<div className="bg-gradient-to-r from-green-500 to-[#031035] p-6 text-white">
<div className="bg-[#031035] p-6 text-white">
<div className="flex items-center justify-between">
<div>
<h2 className="text-2xl font-bold">Graph Template</h2>
Expand Down
2 changes: 1 addition & 1 deletion dashboard/src/components/NodeDetailModal.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ export const NodeDetailModal: React.FC<NodeDetailModalProps> = ({
<div className="fixed inset-0 bg-white/30 backdrop-blur-sm flex items-center justify-center z-50">
<div className="bg-white rounded-lg shadow-xl max-w-4xl w-full mx-4 max-h-[90vh] overflow-y-auto">
{/* Header */}
<div className="bg-gradient-to-r from-[#031035] to-purple-600 p-6 text-white">
<div className="bg-[#031035] p-6 text-white">
<div className="flex items-center justify-between">
<div>
<h2 className="text-2xl font-bold">{node.name}</h2>
Expand Down
147 changes: 0 additions & 147 deletions docs/docs/api-server/index.md

This file was deleted.

2 changes: 0 additions & 2 deletions docs/docs/api-server/swagger.md

This file was deleted.

34 changes: 0 additions & 34 deletions docs/docs/architecture.md

This file was deleted.

Binary file added docs/docs/assets/DashboardSS-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/docs/assets/DashboardSS-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/docs/assets/DashboardSS-3.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 0 additions & 4 deletions docs/docs/cluster.md

This file was deleted.

149 changes: 149 additions & 0 deletions docs/docs/exosphere/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# Exosphere Architecture

This document provides a comprehensive overview of Exosphere's top-level architecture, focusing on state execution, fanout mechanisms, and the unites keyword for stage unification.

## Overview

Exosphere is built around a **state-based execution model** where workflows are composed of discrete states that can be executed independently. The architecture consists of several key components:

- **State Manager**: Central orchestrator that manages state lifecycle and transitions
- **Runtimes**: Distributed execution engines that process individual states
- **Graph Templates**: Declarative workflow definitions
- **States**: Individual execution units with inputs, outputs, and metadata

## State Execution Model

### State Lifecycle

Each state in Exosphere follows a well-defined lifecycle:

```
CREATED → QUEUED → EXECUTED/ERRORED → SUCCESS
```
Comment on lines +20 to +22
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Add a language tag to the lifecycle fence (MD040).

Apply:

-```
+```text
 CREATED → QUEUED → EXECUTED/ERRORED → SUCCESS

<details>
<summary>🤖 Prompt for AI Agents</summary>

In docs/docs/exosphere/architecture.md around lines 20 to 22, the lifecycle
fence is missing a language tag (MD040); update the code block opening from totext and ensure the block content reads " CREATED → QUEUED →
EXECUTED/ERRORED → SUCCESS" (preserve the leading space as shown), then close
the block with ``` so the fence is explicitly tagged as text.


</details>

<!-- fingerprinting:phantom:triton:chinchilla -->

<!-- This is an auto-generated comment by CodeRabbit -->


1. **CREATED**: State is initialized with inputs and dependencies
2. **QUEUED**: State is ready for execution and waiting for a runtime
3. **EXECUTED**: State finished (with outputs) — terminal for that state
4. **ERRORED**: State failed during executionState and all its dependencies are complete
5. **SUCCESS**: Workflow/branch-level success once all dependent states complete

Comment on lines +24 to +29
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Clarify lifecycle text; fix concatenated sentence.

Tighten definitions and remove the stray “executionState …” fragment.

Apply:

 1. **CREATED**: State is initialized with inputs and dependencies
 2. **QUEUED**: State is ready for execution and waiting for a runtime
-3. **EXECUTED**: State finished (with outputs) — terminal for that state
-4. **ERRORED**: State failed during executionState and all its dependencies are complete
-5. **SUCCESS**: Workflow/branch-level success once all dependent states complete
+3. **EXECUTED**: State finished (with outputs) — terminal for that state
+4. **ERRORED**: State failed during execution
+5. **SUCCESS**: Workflow/branch-level success once all dependent states complete
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
1. **CREATED**: State is initialized with inputs and dependencies
2. **QUEUED**: State is ready for execution and waiting for a runtime
3. **EXECUTED**: State finished (with outputs) — terminal for that state
4. **ERRORED**: State failed during executionState and all its dependencies are complete
5. **SUCCESS**: Workflow/branch-level success once all dependent states complete
1. **CREATED**: State is initialized with inputs and dependencies
2. **QUEUED**: State is ready for execution and waiting for a runtime
3. **EXECUTED**: State finished (with outputs) — terminal for that state
4. **ERRORED**: State failed during execution
5. **SUCCESS**: Workflow/branch-level success once all dependent states complete
🧰 Tools
🪛 LanguageTool

[grammar] ~24-~24: There might be a mistake here.
Context: ...initialized with inputs and dependencies 2. QUEUED: State is ready for execution a...

(QB_NEW_EN_OTHER)


[grammar] ~26-~26: There might be a mistake here.
Context: ...e 3. EXECUTED: State finished (with outputs) — terminal for that state 4. ERRORED: State fa...

(QB_NEW_EN_OTHER)


[grammar] ~26-~26: There might be a mistake here.
Context: ...(with outputs) — terminal for that state 4. ERRORED: State failed during execution...

(QB_NEW_EN_OTHER)


[grammar] ~27-~27: There might be a mistake here.
Context: ...ate 4. ERRORED: State failed during executionState and all its dependencies are complete 5...

(QB_NEW_EN_OTHER)


[grammar] ~27-~27: There might be a mistake here.
Context: ...te and all its dependencies are complete 5. SUCCESS: Workflow/branch-level success...

(QB_NEW_EN_OTHER)


[grammar] ~28-~28: There might be a mistake here.
Context: ...ccess once all dependent states complete ## Fanout Mechanism ### Single vs Multiple...

(QB_NEW_EN_OTHER)

🤖 Prompt for AI Agents
In docs/docs/exosphere/architecture.md around lines 24 to 29, the lifecycle list
contains a concatenated/stray fragment ("executionState …") and needs tightened
wording; update the five bullet definitions to be concise and grammatically
correct — e.g., "CREATED: State is initialized with inputs and dependencies",
"QUEUED: State is ready for execution and awaiting a runtime", "EXECUTED: State
finished (with outputs) — terminal for that state", "ERRORED: State failed
during execution and all its dependencies are complete", and "SUCCESS: Workflow
or branch-level success once all dependent states complete", removing the stray
fragment and ensuring consistent punctuation and parallel structure.


## Fanout Mechanism

### Single vs Multiple Outputs

Exosphere supports two execution patterns:

1. **Single Output**: A state produces one output and continues to the next stage
2. **Multiple Outputs (Fanout)**: A state produces multiple outputs, creating parallel execution paths


### Fanout Example

Consider a data processing workflow:

```python hl_lines="9-11"
class DataSplitterNode(BaseNode):
async def execute(self) -> list[Outputs]:
data = json.loads(self.inputs.data)
chunk_size = 100

outputs = []
for i in range(0, len(data), chunk_size):
chunk = data[i:i + chunk_size]
outputs.append(self.Outputs(
chunk=json.dumps(chunk)
))

return outputs # This creates fanout on each output
```

When this node executes:
1. **Original state** gets the first chunk as output
2. **Additional states** are created for each remaining chunk
3. **All states** are marked as EXECUTED
4. **Next stages** are created for each state independently

**This enables parallel processing of data chunks across multiple runtime instances.**

## Unites Keyword

### Purpose

The `unites` keyword is a powerful mechanism for **synchronizing parallel execution paths**. It allows a node to wait for multiple parallel states to complete before executing.

### Unites Logic

When a node has a `unites` configuration:

1. **Execution is deferred** until all states with the specified identifier are complete
2. **State fingerprinting** ensures only one unites state is created per unique combination
3. **Dependency validation** ensures the unites node depends on the specified identifier

### Unites Example

```json hl_lines="22-24"
{
"nodes": [
{
"node_name": "DataSplitterNode",
"identifier": "data_splitter",
"next_nodes": ["processor_1"]
},
{
"node_name": "DataProcessorNode",
"identifier": "processor_1",
"inputs":{
"x":"${{data_splitter.outputs.data_chunk}}"
}
"next_nodes": ["result_merger"]
},
Comment on lines +96 to +100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix invalid JSON: add missing commas after inputs objects.

This currently fails JSON parsing and will trip CI linters.

Apply:

       "inputs":{
         "x":"${{data_splitter.outputs.data_chunk}}"
-      }
+      },
       "next_nodes": ["result_merger"]
@@
       "inputs":{
         "x_processed":"${{processor_1.outputs.processed_data}}"
-      }
+      },
       "unites": {
         "identifier": "data_splitter"
       },

Also applies to: 104-107

🤖 Prompt for AI Agents
In docs/docs/exosphere/architecture.md around lines 96-100 (and similarly
104-107) there are JSON objects missing commas after the "inputs" object entries
before the "next_nodes" key, causing invalid JSON that fails parsing; add a
trailing comma after the closing brace of the "inputs" object (or otherwise
ensure proper comma separation between object properties) for each occurrence so
the JSON is syntactically valid.

{
"node_name": "ResultMergerNode",
"identifier": "result_merger",
"inputs":{
"x_processed":"${{processor_1.outputs.processed_data}}"
}
"unites": {
"identifier": "data_splitter"
},
"next_nodes": []
}
]
}
```

In this example:
1. `data_splitter` creates fanout with 3 outputs
2. `processor_1` executes in parallel for all three data chunks
3. `result_merger` waits for all processors to complete (unites with `data_splitter`)
4. Only one `result_merger` state is created due to fingerprinting

## Architecture Benefits

### Scalability

- **Horizontal scaling**: Add more runtime instances to handle increased load
- **Parallel processing**: Fanout enables concurrent execution
- **Load distribution**: State manager distributes work across available runtimes

### Fault Tolerance

- **State persistence**: All states are stored in the database
- **Retry mechanisms**: Failed states can be retried automatically
- **Recovery**: Workflows can resume from where they left off

### Flexibility

- **Dynamic fanout**: Nodes can produce variable numbers of outputs
- **Synchronization**: Unites keyword provides precise control over parallel execution
- **Dependency management**: Automatic resolution of complex dependencies

### Observability

- **State tracking**: Complete visibility into execution progress
- **Error handling**: Detailed error information and retry logic
- **Performance monitoring**: Track execution times and resource usage


Exosphere's architecture provides a robust foundation for building distributed, scalable workflows. The combination of state-based execution, fanout mechanisms, and the unites keyword enables complex parallel processing patterns while maintaining simplicity and reliability.
Loading