Fix run_tpcds data dir

[gabotechs]

Which issue does this PR close?

• Closes #.

Rationale for this change

Running ./bench.sh run tpcds with a freshly created ./bench.sh data tpcds fails with the following error:

Please prepare TPC-DS data first by following instructions:
  ./bench.sh data tpcds

This PR fixes it

What changes are included in this PR?

Fixes the TPCDS_DIR variable in run_tpcds

Are these changes tested?

just benchmark scripts

Are there any user-facing changes?

no need

[comphead]

Thanks @gabotechs I think it shouldn't be there. by default the script checks for datafusion-benchmarks repo here https://github.com/apache/datafusion-benchmarks/tree/main/tpcds/data/sf1 and there is no tpcds-sf1.

you can specify your own DATA_DIR like

export DATA_DIR=../../datafusion-benchmarks/tpcds/data/sf1/
and then run tpcds benchmarks

[gabotechs]

🤔 Are you sure? I get the impression that this is why the benchmark run commands are failing

#19761 (comment)

Also, note how the data_tpcds() function counterpart actually has this same line:

https://github.com/apache/datafusion/blob/main/benchmarks/bench.sh#L633

# Downloads TPC-DS data
data_tpcds() {
    TPCDS_DIR="${DATA_DIR}/tpcds_sf1"

[comphead]

I just checked for #19635 the TPCDS benchmark the commands provided in
https://github.com/apache/datafusion/blob/main/benchmarks/README.md#comparing-performance-of-main-and-a-pr
and it worked fine

[gabotechs]

Ok, I see what happened here:

• @alamb actually added this line TPCDS_DIR="${DATA_DIR}/tpcds_sf1" already in https://github.com/apache/datafusion/pull/19244/files#diff-1769f5787dc11c8b1f1b48288cdf3c89d25a5b5cbc6be4740bfcc70a6313ba99L694-R688, and then doing ./benchmarks/bench.sh data tpcds && ./benchmarks/bench.sh run tpcds started working out of the box.
• @comphead reverted this change in https://github.com/apache/datafusion/pull/19552/files#diff-1769f5787dc11c8b1f1b48288cdf3c89d25a5b5cbc6be4740bfcc70a6313ba99R686-R687, and doing ./benchmarks/bench.sh data tpcds && ./benchmarks/bench.sh run tpcds no longer works out of the box, requiring users to manually set the DATA_DIR env variable with export DATA_DIR=../../datafusion-benchmarks/tpcds/data/sf1/, and breaking Andrew's benchmarking bot integration here

IMO it would be nicer if ./benchmarks/bench.sh data tpcds && ./benchmarks/bench.sh run tpcds worked out of the box without requiring users to set the DATA_DIR env in the same way it works for the TPC-H benchmark.

In fact, I'd bet the intention behind this code here https://github.com/apache/datafusion/blob/main/benchmarks/bench.sh#L644-L646 is that it works that way, as it's explicitly extracting the contents to "${DATA_DIR}/tpcds_sf1":

        echo "Extracting TPC-DS parquet data to ${TPCDS_DIR}..."
        unzip -o -j -d "${TPCDS_DIR}" "${DATA_DIR}/datafusion-benchmarks.zip" datafusion-benchmarks-main/tpcds/data/sf1/*
        echo "TPC-DS data extracted."

However happy to follow your lead here, I can survive setting up an extra env variable.

[comphead]

IMO it would be nicer if ./benchmarks/bench.sh data tpcds && ./benchmarks/bench.sh run tpcds worked out of the box without requiring users to set the DATA_DIR env in the same way it works for the TPC-H benchmark.

That would be real nice but data generation is tricky for TPCDS so we rely on pregenerated data in datafusion-benchmarks repo

[alamb]

IMO it would be nicer if ./benchmarks/bench.sh data tpcds && ./benchmarks/bench.sh run tpcds worked out of the box without requiring users to set the DATA_DIR env in the same way it works for the TPC-H benchmark.

That would be real nice but data generation is tricky for TPCDS so we rely on pregenerated data in datafusion-benchmarks repo

BTW @clflushopt and I are working on a solution to that -- sneak peek:

• feat: TPC-DS support clflushopt/tpchgen-rs#212