When using monorepo, react only on changes done to configured paths

[lachmanfrantisek]

Description

When using monorepo (=more packages in one git repository, defined using the packages key) with many packages, triggering actions for all the packages is not needed.

Let's check the affected paths first and react only if anything has been done to the package-related path.

Benefit

• Safe of resources. (=Only the important/relevant jobs are done.)
• Cleaner UX. (=Just important/relevant results are shown.)

Importance

This has been asked multiple times, but lately by @LecrisUT who wants to use a monorepo with ~500 packages.

What is the impacted category (job)?

General

Workaround

• There is an existing workaround that can be used until this feature is implemented.

Participation

• I am willing to submit a pull request for this issue. (Packit team is happy to help!)

[lachmanfrantisek]

I am preparing some tips/ideas for the implementation but am now not sure if this should be pull-request/merge-request specific. WDYT? Should we run only change-relevant jobs for pull requests but all for branch/release builds so we are sure everything is rebuilt once merged? Or, is it misleading/confusing and the new builds not run on PR/MR could potentially fail?

[lachmanfrantisek]

Option 1:

Add the info to the Event classes and use this when creating job tasks:

• Add a new property to all the relevant event classes:
  

  packit-service/packit_service/worker/events/github.py

  Line 86 in a8dddb3

  class PullRequestGithubEvent(AddPullRequestDbTrigger, AbstractGithubEvent):

• Fill in the info when parsing:
  

  packit-service/packit_service/worker/parser.py

  Line 250 in a8dddb3

  def parse_pr_event(event) -> Optional[PullRequestGithubEvent]:

  • Check if we can actually get that info: https://docs.github.com/en/webhooks-and-events/webhooks/webhook-events-and-payloads?actionType=opened#pull_request (If I've checked correctly, it's not in the event payload and we need to use GitHub API (using this OGR method in our case) to get that info.
• Use the info when filtering the new tasks to be added to our job queue:
  

  packit-service/packit_service/worker/jobs.py

  Line 446 in a8dddb3

  def should_task_be_created_for_job_config_and_handler(

[LecrisUT]

I would say to have it for both PRs and branch commits initially. Later this should be handled by the dependencies logic with rebuild/sync-version logic from here. To bridge the gap between them, maybe the user could run the packit build in-copr for the select packages or use a \packit build in the PR specifying the packages there (later just \packit build dependents).

[lachmanfrantisek]

Option 2:

Probably more simple. Check this info when creating the job tasks:

• Use GitHub API (using this OGR method in our case) to get the info about changed files.
• And use it when filtering the new tasks to be added to our job queue. Probably here:
  

  packit-service/packit_service/worker/jobs.py

  Line 446 in a8dddb3

  def should_task_be_created_for_job_config_and_handler(

[lachmanfrantisek]

I would say to have it for both PRs and branch commits initially.

👍 That makes sense.

[majamassarini]

I am preparing some tips/ideas for the implementation but am now not sure if this should be pull-request/merge-request specific. WDYT? Should we run only change-relevant jobs for pull requests but all for branch/release builds so we are sure everything is rebuilt once merged? Or, is it misleading/confusing and the new builds not run on PR/MR could potentially fail?

I think the triggering method should stay the same, we should always create all jobs and let the jobs themself decide if they are relevant or not, but maybe I am missing something...

[majamassarini]

Option 1:

Add the info to the Event classes and use this when creating job tasks:

• Add a new property to all the relevant event classes:
  

  packit-service/packit_service/worker/events/github.py

  Line 86 in a8dddb3

  class PullRequestGithubEvent(AddPullRequestDbTrigger, AbstractGithubEvent):

• Fill in the info when parsing:
  

  packit-service/packit_service/worker/parser.py

  Line 250 in a8dddb3

  def parse_pr_event(event) -> Optional[PullRequestGithubEvent]:

  • Check if we can actually get that info: https://docs.github.com/en/webhooks-and-events/webhooks/webhook-events-and-payloads?actionType=opened#pull_request (If I've checked correctly, it's not in the event payload and we need to use GitHub API (using this OGR method in our case) to get that info.

• Use the info when filtering the new tasks to be added to our job queue:
  

  packit-service/packit_service/worker/jobs.py

  Line 446 in a8dddb3

  def should_task_be_created_for_job_config_and_handler(

I know that with these solutions we can save some cpu time. But to me this seems like work that should be done by the Job itself.
I think the jobs won't always have the same filtering method; most of them will just look for something that happened inside their path. But others, which are dependent, could also check for changes elsewhere. And maybe there could exist jobs that need to be done always to keep things consistent (I don't really know, just a fear).