[AIRFLOW-4061] Remove incubator in CI process

[zhongjiajie]

Remove incubator as Airflow is Top-Level project.
[ci skip]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-4061], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal(AIP).

Description

• Here are some details about my PR, including screenshots of any UI changes:

We still use airflowci/incubator-airflow-ci:latest in our ci docker-compose, should change it to airflowci/airflow-ci:latest, in this case we have to change airflow-ci setting in docker-hub, let airflowci/airflow-ci:latest work.

[zhongjiajie]

PTAL @ashb @kaxil @XD-DENG

[XD-DENG]

This PR can't be merged as airflowci/airflow-ci:latest is not available (yet).

Ref: https://hub.docker.com/search?q=airflowci&type=image

[XD-DENG]

Sorry missed your description above @zhongjiajie . Yes, change must be made for airflow-ci image on DockerHub accordingly before this PR can be merged.

[zhongjiajie]

@XD-DENG Never mind, I thought you had seen it 😄. just let comment for other committer/PMC member?

[potiuk]

Maybe we can wait with it a bit as I am working now on #4543 which effectively removes the need for this external airflowci image - also something discussed in
AIP-7 (CI tests might run using multi-staging Docker which is part of the main repo). I am going to add some more description/explanation and resume community discussion on it this week.

[Fokko]

I think we should get this in for now. We should get rid of all incubator references. I've also requested to remove the old repository: https://jira.apache.org/jira/browse/INFRA-18084

Did the CI run with this PR?

[Fokko]

@zhongjiajie Please [ci skip] from the commit message.

[Fokko]

https://travis-ci.org/Fokko/incubator-airflow/builds/510390240

[zhongjiajie]

@Fokko I already add [ci skip] in the git comment.

[Fokko]

Sorry, I was a bit too quick with the enter button there. I think we should run the CI to check if the image works.

[zhongjiajie]

@Fokko I remove "skip" and start test. but I could not find any release in https://hub.docker.com/r/airflowci/airflow-ci and I don't think CI will pass

[Fokko]

So it died:

ERROR: for airflow-testing  manifest for airflowci/airflow-ci:latest not found
ERROR: manifest for airflowci/airflow-ci:latest not found
The command "docker-compose -f scripts/ci/docker-compose.yml pull --quiet --parallel" failed and exited with 1 during .

Lets see if we can do a new release of the image. AIP-3 is pending until Sunday. Lets wait until 1.10.3 to make @ashb 's life easier ❤️ and then make a new release of the CI image.

[Fokko]

@kaxil Can you elaborate on what's going on here. I see that the current repository points to kaxil/incubator-airflow-ci. Can we change this for https://github.com/apache/airflow-ci

[kaxil]

@Fokko where exactly is that? Sorry, I am not able to get this

[Fokko]

No, I'm confused. I thought the old incubator repo was pointing to your repository.

[potiuk]

Hey @Fokko @kaxil . Not sure if you are aware of it but the problem is that you need to update DockerHub account rather than GitHub repo. The URL in docker-compose is link to this dockerhub repository : https://cloud.docker.com/repository/docker/airflowci/incubator-airflow-ci and it takes the name of the image from repository name in DockerHub.

In order to make it works, you'd have to create a new repository for 'airflowci' DockerHub user ( airflow-ci). I don't think you can rename incubator-airflow-ci repository (I have not found such option)- yoy need to copy configuration for automated build from incubator to the new repository (and eventually delete the old Dockerhub incubator-airflow-ci repository). No change in https://github.com/apache/airflow-ci github is needed.

BTW. I am rather close to fixing all errors in the multi-stage docker #4938 and once this is merged, we won't need this dual-account/dual repos drama ;). All of this will be happening (automatically) via 'airflow' github repo and 'airflow' dockerhub user.

[Fokko]

@potiuk I was thinking the same, in the end, we would like to use the same image. Having two repositories and trying to keep them in sync is painful, and distracts us from the actual work :-)

[zhongjiajie]

@Fokko Are we going to point to airflowci/airflow-ci:latest right now?

[Fokko]

@zhongjiajie Not yet, airflowci/airflow-ci:latest isn't build right now, I need to figure out how to fix this.

[zhongjiajie]

Get it.

[zhongjiajie]

Will close this PR because of @potiuk 's #4938 try to integrate our CI to our main repo.