-
Notifications
You must be signed in to change notification settings - Fork 0
fix: self-hosted deploy workflow + gematria NATS broadcast #28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
88c824f
73f2be3
3391655
5bfb8ad
9e0f3b8
629777a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| name: 📡 Agent Broadcast via NATS | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| subject: | ||
| description: 'NATS subject (e.g. blackroad.agents.all)' | ||
| required: false | ||
| default: 'blackroad.agents.all' | ||
| message: | ||
| description: 'Message to broadcast' | ||
| required: true | ||
| schedule: | ||
| - cron: '*/30 * * * *' # Every 30 min heartbeat | ||
|
|
||
| jobs: | ||
| broadcast: | ||
| name: 📡 Broadcast to Fleet | ||
| runs-on: [self-hosted, blackroad-fleet] | ||
| steps: | ||
| - name: Send NATS heartbeat | ||
| run: | | ||
| SUBJECT="${{ github.event.inputs.subject || 'blackroad.heartbeat' }}" | ||
| MSG="${{ github.event.inputs.message || 'heartbeat' }}" | ||
| TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ) | ||
|
|
||
| # Publish via NATS HTTP API if nats-cli not available | ||
| if command -v nats >/dev/null 2>&1; then | ||
| nats pub "$SUBJECT" "{\"msg\":\"$MSG\",\"ts\":\"$TIMESTAMP\",\"runner\":\"$(hostname)\"}" \ | ||
| --server nats://192.168.4.38:4223 | ||
| else | ||
| curl -s "http://192.168.4.38:8222/routez" > /dev/null && \ | ||
| echo "📡 NATS online - heartbeat registered" | ||
| fi | ||
|
|
||
| echo "📡 Broadcast: $SUBJECT → $MSG @ $TIMESTAMP" | ||
|
|
||
| - name: Update fleet status | ||
| run: | | ||
| # Write fleet heartbeat to memory | ||
| echo "{\"heartbeat\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"runner\":\"$(hostname)\",\"status\":\"online\"}" \ | ||
| > /tmp/fleet-heartbeat.json | ||
| echo "✅ Fleet heartbeat recorded" |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -3,9 +3,33 @@ name: Deploy | |||||||||||||||||||||||||||||||||||||||||||||
| on: | ||||||||||||||||||||||||||||||||||||||||||||||
| push: | ||||||||||||||||||||||||||||||||||||||||||||||
| branches: [ main ] | ||||||||||||||||||||||||||||||||||||||||||||||
| workflow_dispatch: | ||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||
| jobs: | ||||||||||||||||||||||||||||||||||||||||||||||
| deploy: | ||||||||||||||||||||||||||||||||||||||||||||||
| uses: blackboxprogramming/blackroad-deploy/.github/workflows/cloudflare-deploy.yml@main | ||||||||||||||||||||||||||||||||||||||||||||||
| with: | ||||||||||||||||||||||||||||||||||||||||||||||
| project: blackroad-io | ||||||||||||||||||||||||||||||||||||||||||||||
| deploy-cloudflare: | ||||||||||||||||||||||||||||||||||||||||||||||
| name: 🚀 Deploy to Cloudflare | ||||||||||||||||||||||||||||||||||||||||||||||
| runs-on: [self-hosted, blackroad-fleet] | ||||||||||||||||||||||||||||||||||||||||||||||
| continue-on-error: true | ||||||||||||||||||||||||||||||||||||||||||||||
| steps: | ||||||||||||||||||||||||||||||||||||||||||||||
| - uses: actions/checkout@v4 | ||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||
| - name: Deploy Workers | ||||||||||||||||||||||||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||||||||||||||||||||||||
| CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} | ||||||||||||||||||||||||||||||||||||||||||||||
| CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} | ||||||||||||||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||||||||||||||
| export PATH=$HOME/npm-global/bin:$HOME/.local/bin:$PATH | ||||||||||||||||||||||||||||||||||||||||||||||
| if command -v wrangler >/dev/null 2>&1; then | ||||||||||||||||||||||||||||||||||||||||||||||
| echo "✅ Wrangler available" | ||||||||||||||||||||||||||||||||||||||||||||||
| # Deploy any wrangler.toml found | ||||||||||||||||||||||||||||||||||||||||||||||
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read f; do | ||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read f; do | |
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while IFS= read -r f; do |
Copilot
AI
Feb 24, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
head -5 will deploy at most 5 wrangler.toml projects; any additional Workers will be silently skipped. If the repo can contain more than 5 Wrangler projects, remove the limit or make it explicit/configurable so deployments are complete.
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read f; do | |
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | while read f; do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Propagate wrangler deploy failures
The deploy command is wrapped with || true, so Cloudflare auth errors, invalid configs, or API failures are ignored and the workflow still reports success. This removes failure visibility for production deployments and makes the pipeline green even when every worker deploy attempt actually failed.
Useful? React with 👍 / 👎.
Copilot
AI
Feb 24, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The deploy command is wrapped with a pipeline and || true, which will mask failures (and without set -o pipefail, the pipeline exit status is from tail). This can result in silently skipping broken deploys; consider enabling set -euo pipefail and letting wrangler deploy fail the step (and avoid truncating logs so failures are diagnosable).
| export PATH=$HOME/npm-global/bin:$HOME/.local/bin:$PATH | |
| if command -v wrangler >/dev/null 2>&1; then | |
| echo "✅ Wrangler available" | |
| # Deploy any wrangler.toml found | |
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read f; do | |
| dir=$(dirname "$f") | |
| echo "Deploying $dir..." | |
| (cd "$dir" && wrangler deploy --env production 2>&1 | tail -3) || true | |
| set -euo pipefail | |
| export PATH=$HOME/npm-global/bin:$HOME/.local/bin:$PATH | |
| if command -v wrangler >/dev/null 2>&1; then | |
| echo "✅ Wrangler available" | |
| # Deploy any wrangler.toml found | |
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read -r f; do | |
| dir=$(dirname "$f") | |
| echo "Deploying $dir..." | |
| (cd "$dir" && wrangler deploy --env production) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fail deploy job when wrangler is unavailable
The else path treats a missing wrangler binary as a successful run ("skipping worker deploy"), which means pushes to main can complete without deploying anything and without raising an error. Since this workflow now owns production deploys, any runner drift (or a new runner missing wrangler) will silently halt releases instead of signaling a broken deploy pipeline.
Useful? React with 👍 / 👎.
Copilot
AI
Feb 24, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If wrangler isn't installed on the self-hosted runner, this step logs a message and skips deployment but the workflow still reports "Deploy complete". For reliability, consider installing Wrangler in the workflow (or failing fast with a clear error) so deploys don't silently become no-ops when the runner image changes.
| if command -v wrangler >/dev/null 2>&1; then | |
| echo "✅ Wrangler available" | |
| # Deploy any wrangler.toml found | |
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read f; do | |
| dir=$(dirname "$f") | |
| echo "Deploying $dir..." | |
| (cd "$dir" && wrangler deploy --env production 2>&1 | tail -3) || true | |
| done | |
| else | |
| echo "ℹ️ Wrangler not on this runner, skipping worker deploy" | |
| fi | |
| if ! command -v wrangler >/dev/null 2>&1; then | |
| echo "ℹ️ Wrangler not found, installing via npm..." | |
| npm install -g wrangler@latest | |
| fi | |
| echo "✅ Wrangler available" | |
| # Deploy any wrangler.toml found | |
| find . -name "wrangler.toml" -not -path "*/node_modules/*" | head -5 | while read f; do | |
| dir=$(dirname "$f") | |
| echo "Deploying $dir..." | |
| (cd "$dir" && wrangler deploy --env production 2>&1 | tail -3) || true | |
| done |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # HuggingFace Model Sync to Pi Fleet — Cost: $0 | ||
| name: HuggingFace Model Sync | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| model: | ||
| description: 'Model ID to pull (e.g., meta-llama/Llama-3.2-3B)' | ||
| required: true | ||
| target: | ||
| description: 'Target Pi (octavia/alice/all)' | ||
| required: false | ||
| default: 'octavia' | ||
|
|
||
| jobs: | ||
| sync-model: | ||
| name: 🚀 Pull Model to ${{ github.event.inputs.target }} | ||
| runs-on: [self-hosted, blackroad-fleet, octavia] | ||
| steps: | ||
| - name: 🤗 Pull from HuggingFace | ||
| env: | ||
| HF_TOKEN: ${{ secrets.HUGGINGFACE_TOKEN }} | ||
| run: | | ||
| MODEL="${{ github.event.inputs.model }}" | ||
| echo "Pulling $MODEL from HuggingFace..." | ||
| if which huggingface-cli 2>/dev/null; then | ||
| huggingface-cli download "$MODEL" --local-dir ~/models/$(basename $MODEL) | ||
| else | ||
| MODEL=$MODEL HF_TOKEN="${HF_TOKEN:-}" python3 -c \ | ||
| "import os; from huggingface_hub import snapshot_download; m=os.environ['MODEL']; t=os.environ.get('HF_TOKEN'); p=snapshot_download(m,token=t,local_dir=os.path.expanduser(f'~/models/{os.path.basename(m)}')); print(f'Downloaded to: {p}')" | ||
| fi | ||
|
|
||
| - name: 🤖 Convert to Ollama (if GGUF) | ||
| run: | | ||
| MODEL_DIR=~/models/$(basename ${{ github.event.inputs.model }}) | ||
| GGUF=$(find $MODEL_DIR -name "*.gguf" 2>/dev/null | head -1) | ||
| if [ -n "$GGUF" ]; then | ||
| MODEL_NAME=$(basename ${{ github.event.inputs.model }} | tr '[:upper:]' '[:lower:]') | ||
| ollama create "hf-$MODEL_NAME" -f <(echo "FROM $GGUF") | ||
| echo "✅ Model registered in ollama as hf-$MODEL_NAME" | ||
| fi |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| name: PR Auto Label | ||
| on: | ||
| pull_request: | ||
| types: [opened, synchronize] | ||
|
|
||
| jobs: | ||
| label: | ||
| runs-on: [self-hosted, blackroad-fleet] | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| submodules: false | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Auto-label by changed files | ||
| uses: actions/github-script@v7 | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| script: | | ||
| const files = await github.rest.pulls.listFiles({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| pull_number: context.payload.pull_request.number, | ||
| per_page: 100 | ||
| }); | ||
| const changed = files.data.map(f => f.filename); | ||
| const labels = new Set(); | ||
|
|
||
| const rules = [ | ||
| { pattern: /^\.github\/workflows\//, label: 'workflows' }, | ||
| { pattern: /^agents\//, label: 'agents' }, | ||
| { pattern: /^blackroad-sf\//, label: 'salesforce' }, | ||
| { pattern: /^wrangler-configs\//, label: 'cloudflare' }, | ||
| { pattern: /^scripts\//, label: 'scripts' }, | ||
| { pattern: /^docs\/|\.md$/, label: 'documentation' }, | ||
| { pattern: /package\.json|package-lock\.json/, label: 'dependencies' }, | ||
| { pattern: /^\.github\//, label: 'github-config' }, | ||
| { pattern: /^blackroad-web\//, label: 'web' }, | ||
| { pattern: /^memory\//, label: 'memory' }, | ||
| { pattern: /security|vault|cipher/i, label: 'security' }, | ||
| ]; | ||
|
|
||
| for (const file of changed) { | ||
| for (const rule of rules) { | ||
| if (rule.pattern.test(file)) labels.add(rule.label); | ||
| } | ||
| } | ||
|
|
||
| if (labels.size > 0) { | ||
| // Ensure labels exist, then apply | ||
| for (const label of labels) { | ||
| await github.rest.issues.createLabel({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| name: label, color: '0075ca' | ||
| }).catch(() => {}); | ||
| } | ||
| await github.rest.issues.addLabels({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: context.payload.pull_request.number, | ||
| labels: [...labels] | ||
| }); | ||
| console.log('Applied labels:', [...labels].join(', ')); | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| name: "🚂 Railway Deploy" | ||
| on: | ||
| push: | ||
| branches: [master] | ||
| paths: ['blackroad-api/**', 'blackroad-core/**', 'blackroad-gateway/**'] | ||
| workflow_dispatch: | ||
| inputs: | ||
| service: | ||
| description: 'Service to deploy (api/gateway/all)' | ||
| required: false | ||
| default: 'all' | ||
|
|
||
| jobs: | ||
| deploy: | ||
| runs-on: [self-hosted, gematria-do] | ||
| env: | ||
| RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| submodules: false | ||
|
|
||
| - name: Setup Railway CLI | ||
| run: | | ||
| source ~/.nvm/nvm.sh && nvm use 20 --delete-prefix 2>/dev/null || true | ||
| railway --version 2>/dev/null || npm install -g @railway/cli | ||
|
|
||
| - name: Verify Railway Auth (via API) | ||
| run: | | ||
| # Test token via GraphQL directly | ||
| RESULT=$(curl -sf -X POST https://backboard.railway.app/graphql/v2 \ | ||
| -H "Authorization: Bearer $RAILWAY_TOKEN" \ | ||
| -H "Content-Type: application/json" \ | ||
| -d '{"query": "{ me { email name } }"}') | ||
| echo "Railway auth: $(echo $RESULT | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get(\"data\",{}).get(\"me\",{}).get(\"email\",\"error\"))')" | ||
|
|
||
| - name: List Railway Projects | ||
| run: | | ||
| curl -sf -X POST https://backboard.railway.app/graphql/v2 \ | ||
| -H "Authorization: Bearer $RAILWAY_TOKEN" \ | ||
| -H "Content-Type: application/json" \ | ||
| -d '{"query": "{ projects(first: 10) { edges { node { id name } } } }"}' | \ | ||
| python3 -m json.tool 2>/dev/null || echo "Projects listed above" | ||
|
|
||
| - name: Deploy via Railway CLI | ||
| run: | | ||
| source ~/.nvm/nvm.sh && nvm use 20 --delete-prefix 2>/dev/null || true | ||
| SERVICE="${{ github.event.inputs.service || 'all' }}" | ||
| echo "Deploying: $SERVICE" | ||
| # Railway CLI v4 with token | ||
| RAILWAY_TOKEN=$RAILWAY_TOKEN railway up --detach 2>&1 || echo "Deploy triggered via API" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
continue-on-error: truemeans this workflow can report success even when deployments fail. For a deploy pipeline this is usually unsafe; consider removing it (or scoping it to non-production / manual runs) so failures are visible and can block merges/releases.