Add context about failing branches during error in used_cache()

[courentin]

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

This PR is an implementation of #5037.
It is my first contribution to the dvc project, feel free to challenge it.

[courentin]

This needs to be updated

[efiop]

Seems like using --force would be intuitive here.

Suggested change

	if ignore_dvc_exceptions:
	if ignore_dvc_exceptions or force:

WDYT?

[courentin]

Do you suggest using force and ignore_dvc_exceptions or just force?
Indeed, --force seems more coherent with options across dvc commands

[efiop]

@courentin Seems like --force would be enough.

[courentin]

Hum, --force is already used in the gc command "Force garbage collection - automatically agree to all prompts".

[courentin]

@efiop Don't we want to differentiate --force where the behaviour is "Force garbage collection - automatically agree to all prompts" and --skip-failing-collect so that we do one without the other?

WDYT?

I'd like to merge this PR so that our team can clean their workspace :)

[efiop]

@courentin Do you have a scenario in which those could be used separately?

[courentin]

I'd say if we want to ignore failing commits but not agree to all prompts. But to be honest I'm fine with using only --force, I let you decide what's best for the product ;)

[efiop]

Thank you for the PR! 🙏

Ideally, this should skip particular malformed dvcfiles and not the whole branch. We have similar requests for metrics/params/plots parsing too. But this would be a great start too.

[courentin]

Yes indeed, not sure how to do it however

[skshetry]

Yes indeed, not sure how to do it however

There's not an easy way right now for this. I'd imagine something like the following, as stage collection error can be handled using a callback:

def func(path, exc):
  print(f"Collecting '{path}' failed and was skipped.")

with repo.handle_error("stage_collection", func):
   stages = repo.stages

Something like that should work for the CLI usage, but as an API, it might not as we may have collected the stages before running
metrics.show() for example.

@efiop @pmrowla @pared if you have better suggestions.

[efiop]

@skshetry Maybe just something generic and explicit like a onerror(similar to os.walk) for collect* methods?

[courentin]

Do you mind if after finding a good wording for this option, I try to merge this PR as it is and we can discuss a more general approach in an issue or discussion?
(I'm sorry I can't commit myself too much on dvc even tho I love it)

[skshetry]

Do you mind if after finding a good wording for this option, I try to merge this PR as it is and we can discuss a more general approach in an issue or discussion?

We don't expect you to handle this right away. This was just us discussing a long-term solution, sorry for the noise here.
The PR looks good, I'm fine with either of the --force or the --skip-failing-collect, as long as they are well documented (just don't like the conditional flag on gc, that's all :) ).

It'd be great if you could respond to #5037 (comment) and #5066, that'd help us prioritize (if those features would be helpful for you). Thanks a lot for the contributions, @courentin.

[efiop]

Closing as outdated. @courentin Thank you for your patience 🙏 , we've readjusted error handling for our metrics/plots/etc, and brancher is on our todo list for the near future. So please stay tuned.