Add live site s3 docs bucket

[gopidesupavan]

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[eladkal]

When would we use the stage option?

[gopidesupavan]

When would we use the stage option?

Yeah, we can use it for testing docs and @potiuk mentioned in dev channel we could publish RC provider docs to staging location and verify , thats a good idea :)

[potiuk]

Yeah. I think we can work out the workflows for this step-by-step with @kaxil and @ephraimbuddy and others who would be the release managers.

Right after we improve and finalize the docs for regular workflows, we can work out how to do the "staging" ones. For now Kaxil for example publishes Airflow docs with surge.sh - which is fine but it would be better if we can do the same publishing for RC as for the regular releases.

For example when we are relasing the images we have the same workflow -> just "Release -> type version-> workflow will appropriately build the images with or without --pre option and automatically (based on the presence of pre-release rc/alpha/beta in the version) using the rc versions of the providers (and handles all the complex cases) and with or without updating the "katest" image link (for example when we release an RC image we never update "latest" aliases in DockerHub.

Here, ti's a bit different story, because we have to first work out how we are publishing the staging site as well. Not only the documentation for airflow and providers.

And we do not have it automated yet, but we can - and we planned to have a dedicated branch "staging" that you can push in "airflow-site" and then the site will automatically be published at "staging.airflow.apache.org" - that will allow us to iterate on "site" itself. Then I imagine when you are releasing RC candidates, you could choose "staging" from the list (that might be even a bit simpler than now - just additional "staging" checkbox that might even be selected by default) or have a separate "Staging Publishing' workflow - and then the RC docs would be published at "staging.airflow.apache.org" as well. Also we could always select staging by default when tag contains pre-release version (rc/alpha etc.) and do not give freedom to the RM to select staging.

One thing to note there is a question of RC links in pypi - currently the links are nicely redirecting Documentation for example - and ... it nicely redirects to 1.0.1 (I thiink it 's a side effect of the index.html fixing that @gopidesupavan implemented.

Probably we could do the same with future links that are for versions we have not released yet - but we could redirect them to staging - so for example: https://airflow.apache.org/docs/apache-airflow-providers-common-messaging/1.0.2rc1/ (when 1.0.2 is not yet released and docs not published) would redirect to https://staging.airflow.apache.org/docs/apache-airflow-providers-common-messaging/1.0.2/ -> and use the documentation from there.

In this case the workflows would look like this:

1. When you want to check a new "site" modification -> push to "staging" branch in "airflow-site" and it will be visible under "staging.airflow.apache.org". Later merging it to "main" branch will publish them to "airflow.apache.org" - so far, so good.

2. When you want to make a release candidate for providers, airlfow , docker-stack, all-providers -> you run the same workflow as for final publishing (choose rc* tag to use) -> they will also land at staging.airlfow.apache.org, links in PyPi packages will point to "apache.airflow.org/ .... x.y.zrc2" - but since documentation will not be published in "airflow.apache.org" - that documentation will get riderected to "staging.airflow.apache.org/.....x.y.z/" which will be already published by release manager.

3. once we publish the final release, the RM will re-run the docs push (but will select "no staging" option) and the docs will land in "airflow.apache.org" -> and the links from RC candidates in PyPI will no longer redirect to staging, they will point to the final release (of airflow or providers).

Does it sound like a good plan - @eladkal @kaxil @ephraimbuddy ?

What would be the preference for workflows of publishing airlfow/provider docs ?

• two separate workflow (PROD/STAGING push?)
• one workflow with checkbox? (what should be the default)
• or maybe use the TAG and take away the freedom of choosing staging or not

If we get answer to those questions - we can proceed with implementing staging the chosen way :)

[potiuk]

Also there is another option - we could always publish latest main to staging when main succeeds. But that will go fine for "providers" but not too fine for airflow (we want to push airflow docs from v-* branch). And we already have the "main" docs published using the s3 main bucket which we use to pull inventories from, and I think what is at staging should be a deliberate act of release manager rather than what is "latest".

[gopidesupavan]

Thanks Jarek for such a details process, this approach is fine.

But will leave it release managers, @eladkal @kaxil @ephraimbuddy WDYT?

[potiuk]

I think it was a bit "abstract" description so far but with #50464 which describes how things work and show screenshots and workflows - it might be a little more digestible (albeit it's a longer read).

@eladkal @kaxil @ephraimbuddy -> the #50464 is for you to take a look and see if you are fine with the processes and workflows.

Also @eladkal -> I do not think it's worth to cover the case that happened this time (when we had to build docs for Amazon from main instead of tag) is worth automating. We should aim to be able to release provider docs directly from the tag, and in case we need to apply some extra fixes after the tag is already created, we always have manual publishing option - there are two ways described in #50464 and rather than trying to automate those cases, i'd keep the manual ways as "escape-hatch". We have enough ways to override the "GitHub Actions" regular process - and then it could even be done with help of others. And via apache-airflow-site-archive it does not even need S3 credentials - just building docs locally and pushing them to a branch in apache-airflow-site-archive and then pushing changes to S3 via GH -> S3 workflow.

[kaxil]

Two separate workflow (PROD/STAGING push?) is fine. So we can use it for testing things to when needed

What would be the preference for workflows of publishing airlfow/provider docs ?

• two separate workflow (PROD/STAGING push?)
• one workflow with checkbox? (what should be the default)
• or maybe use the TAG and take away the freedom of choosing staging or not

If we get answer to those questions - we can proceed with implementing staging the chosen way :)

[potiuk]

Two separate workflow (PROD/STAGING push?) is fine. So we can use it for testing things to when needed

One thing about single workflow that I thought about is that it could automatically push to staging when RC tag is used and that saves one decision to make by RM. We could parse the tag and when pre-release is detected, we would publish to staging instead of live.

That would also nicely correspond to building packages - because in pre-release builds in metadata we could point to that staging site rather than production one. Currently when you publish RC in PyPI - the documentation link points to non-existing RC version (and even some people reported that as a bug) but we could change it so that links are pointing to final version but on staging site. and whenever you use RC/ALPHA/BETA tag, we could automatically push to staging.

And we could have a drop-down:

• auto (defaullt)
• live
• staging

That would "close" the loop on our RC packages preparation.

1. prepare RC packages -> staging link in metadata https://staging.airflow.apache.org/...../1.0.1 (even if RC is used)
2. publish docs using RC tag -> push to staging s3 while voting on RC
   .... voting .....
3. final packages -> live link in metadata https://airflow.apache.org/..../1.0.1
4. publish docs using final tag -> push to live s3