Improve performance of package ordering/tagging after version solving

[zweger]

This PR greatly improves the performance of package ordering/tagging which occurs during package solving, after the version solving itself has been completed. In certain cases, the current algorithm has exponential time complexity and starts to really break down when the package dependency graph is dense. I've added a test case for one such scenario, where the dependencies form a transitive tournament. On master this test takes several minutes to run with my hardware, but only takes around 100ms with this patch.

I've noticed on personal projects that it's very easy to run into this issue when using aws_cdk.

At the core of the new algorithm is a depth-first search which is used to compute a topological sort of the packages. Using the topological ordering, the longest path / depth of each package can be computed.

Along the way I noticed that the current algorithm does not correctly check for dependency cycles and in certain cases will trigger a RecursionError. I've added a test case for this scenario as well.

Pull Request Check List

This may be related to #2642 and #2149, which both mention using aws_cdk.

• Added tests for changed code.
• Updated documentation for changed code.

[abn]

@sdispater this might need your review.

[dekoza]

Can I do anything to help this patch reach production asap? It's among the top fixes on my "urgently needed" list.

[dekoza]

This works like a charm. Dependencies that took 40 minutes in total (20 resolving + 20 locking) are now resolved in 18 seconds. Thank you @zweger !

[sdispater]

Great work @zweger!

This part of the code always felt clunky – even though it was doing the job – but I never had the time to improve it so I appreciate that you took the time to do it.

Thanks a lot!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.