github workflow python build and test takes long time

[SarahAlidoost]

Currently each task in https://github.com/WUR-AI/diffWOFOST/blob/main/.github/workflows/build.yml takes about 20 minutes. can we optimize this?

[SarahAlidoost]

in #94 and #92 some solutions are implemented.