Stop duplicating Apple Silicon Spark 3.4 PR builds

### Describe the bug

We run the 3.4 tests twice for each scala version on Apple Silicon:

<img width="570" alt="Image" src="https://github.com/user-attachments/assets/ce5ec6aa-df06-4a73-badb-d4d27a766318" />

### Steps to reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_