Investigate perf improvements using NVidia plugin

[edfreeman]

To utilise the NVidia GPU, use:

	"runArgs": [
		"--gpus",
		"all"
	],

in the devcontainer,

and then:

builder = (
    SparkSession.builder.appName(workload_name)
    .master("local[*]")
    ...
    .config("spark.plugins", "com.nvidia.spark.SQLPlugin")
)

extra_packages = ["com.nvidia:rapids-4-spark_2.12:24.06.0"]

spark = configure_spark_with_delta_pip(builder, extra_packages=extra_packages).getOrCreate()

[edfreeman]

Start-up time is prolonged slightly, since it has to download the package.

Instructions are here: https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/on-premise.html#local-mode

Note: nvidia-smi is automatically available when the docker runArgs commands as shown above are defined in the devcontainer.

[edfreeman]

Tested on mdp-python. Tests took a lot longer. Initial test run which downloaded package took 2 mins to start-up SparkSession. Subsequent tests still took ~15 seconds to start-up SparkSession compared to ~5 seconds without GPU config. Laptop fan also went wild (battery on "best performance").

Hypothesis is that test data is too small to worry about benefits explicit GPU config provides. Propose closing down this issue.