Change Railway autodeployments to non-main branch

[lightwalker-eth]

Right now PR 46 is approved and ready to merge, but we're holding back from merging it because we want to wait for another PR is ready to merge to reduce the frequency of redeployments.

Should we create a special branch just for our "alpha" environment in Railway? This way we could separate the action of merging to main from triggering a redeployment.

Thoughts?

[shrugs]

that's fair, let's do it. we should strive for main to equal deployed version but perhaps that's not reasonable with our current velocity outpacing indexing time. once cached backfill time is under healthcheck timeout we should move back to deploying main

[tk-o]

I suggest we switch triggering the deployments to Github Actions
https://blog.railway.com/p/github-actions

This way, we can control the schema name (i.e. use github commit sha), and manage the db cleanup after each deployment.
We can also easily configure the workflow triggers to be automated for some branches.

[tk-o]

As of now, the full historical indexing across all plugins take around 13 hours. Railway allows waiting only for up to 1 hour for new deployment to become ready. We won't be able to achieve rolling deployments in that setting.

[lightwalker-eth]

@tk-o Interesting suggestions 😄

I'm not so good with devops things so appreciate if you can confirm my understanding.

As I understand, we're currently using the "default" Railway deployment strategy, where Railway is using a "pull" strategy with our main branch. Each time the head of the main branch is updated, GitHub is probably calling some webhook in Railway, and then Railway is "pulling" the head of our main branch to create a new deployment?

And as I understand, here's you're proposing we change to a more advanced Railway deployment strategy, where we "push" to Railway as part of our GitHub Actions CI?

Assuming so, yes I like this direction.

[lightwalker-eth]

This comment will grow the scope of this issue.

I'm curious about the most simple and direct path for us to achieve goals including:

1. Continue iterating quickly on our main branch in GitHub, such that merging into main doesn't necessarily trigger any new deployments.
2. Selectively trigger a deploy to 1 (or more??) "Staging Environments" in Railway. Each deploy here is understood to take 12+ hours to become "ready".
3. Continuously offer a "Production Environment" in Railway that has as close to zero downtime as possible, including when we apply an upgrade to ENSNode.

Appreciate suggestions for how we can essentially hot-swap out the "Production Environment" when we want to make an upgrade to it. However we approach this, we should ensure it is easy for others running ENSNode in their own infrastructure to be able to do as well. Everyone running ENSNode needs an easy narrative for executing upgrades without downtime.

Perhaps we should reach out to Kevin and the Ponder team on this? Imagine there's many others using Ponder seeking similar goals.

[shrugs]

That sounds nice. if GH Action config for deploys is trivial let's do that, though i don't know that we want a railway deployment for every branch or even every PR, given the relative cost of that deployment. perhaps simplest is making an alpha branch (perhaps production?) branch like you mentioned and just pointing railway to that so that deployments (with downtime, currently) are done after git merge production main && git push origin production

re: no-downtime deployments:

• i think/hope that the improvements ponder team is making in [draft] sync store perf experiments ponder-sh/ponder#1400 will bring our historical sync below an hour, but if it doesn't...
• asking ponder team & community for their own solutions is an obvious next step
• perhaps even asking railway to raise their healthcheck timeout limit on your account/deployment might be a valid option, if they support such a thing. otherwise perhaps switching to render or another cloud provider with better zero-downtime deployment would be a possiblity

[tk-o]

perhaps even asking railway to raise their healthcheck timeout limit on your account/deployment might be a valid option, if they support such a thing. otherwise perhaps switching to render or another cloud provider with better zero-downtime deployment would be a possiblity

I can definitely test the coolify.io setup I have in place and see how it works with zero-downtime deployments with 12+ hours of ponder warmup phase ;)