diff --git a/README.md b/README.md index bf6c8612c2..9f7db6eba1 100644 --- a/README.md +++ b/README.md @@ -44,8 +44,8 @@ The following chart shows the time it takes to run the 22 TPC-H queries against using a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html) for details of the environment used for these benchmarks. -When using Comet, the overall run time is reduced from 616 seconds to 379 seconds, a 1.62x speedup, with query 1 -running more than 7x faster than Spark. +When using Comet, the overall run time is reduced from 616 seconds to 374 seconds, a 1.6x speedup, with query 1 +running 9x faster than Spark. Running the same queries with DataFusion standalone (without Spark) using the same number of cores results in a 3.6x speedup compared to Spark. @@ -53,16 +53,21 @@ speedup compared to Spark. Comet is not yet achieving full DataFusion speeds in all cases, but with future work we aim to provide a 2x-4x speedup for a broader set of queries. -![](docs/source/_static/images/benchmark-results/2024-08-23/tpch_allqueries.png) +![](docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png) Here is a breakdown showing relative performance of Spark, Comet, and DataFusion for each TPC-H query. -![](docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_compare.png) +![](docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png) -The following chart shows how much Comet currently accelerates each query from the benchmark. Performance optimization -is an ongoing task, and we welcome contributions from the community to help achieve even greater speedups in the future. +The following charts shows how much Comet currently accelerates each query from the benchmark. -![](docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_speedup_rel.png) +### Relative speedup + +![](docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png) + +### Absolute speedup + +![](docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png) These benchmarks can be reproduced in any environment using the documentation in the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage diff --git a/docs/source/_static/images/benchmark-results/2024-08-23/tpcds_allqueries.png b/docs/source/_static/images/benchmark-results/0.2.0/tpcds_allqueries.png similarity index 100% rename from docs/source/_static/images/benchmark-results/2024-08-23/tpcds_allqueries.png rename to docs/source/_static/images/benchmark-results/0.2.0/tpcds_allqueries.png diff --git a/docs/source/_static/images/benchmark-results/2024-08-23/tpcds_queries_compare.png b/docs/source/_static/images/benchmark-results/0.2.0/tpcds_queries_compare.png similarity index 100% rename from docs/source/_static/images/benchmark-results/2024-08-23/tpcds_queries_compare.png rename to docs/source/_static/images/benchmark-results/0.2.0/tpcds_queries_compare.png diff --git a/docs/source/_static/images/benchmark-results/2024-08-23/tpcds_queries_speedup_abs.png b/docs/source/_static/images/benchmark-results/0.2.0/tpcds_queries_speedup_abs.png similarity index 100% rename from docs/source/_static/images/benchmark-results/2024-08-23/tpcds_queries_speedup_abs.png rename to docs/source/_static/images/benchmark-results/0.2.0/tpcds_queries_speedup_abs.png diff --git a/docs/source/_static/images/benchmark-results/2024-08-23/tpch_allqueries.png b/docs/source/_static/images/benchmark-results/0.2.0/tpch_allqueries.png similarity index 100% rename from docs/source/_static/images/benchmark-results/2024-08-23/tpch_allqueries.png rename to docs/source/_static/images/benchmark-results/0.2.0/tpch_allqueries.png diff --git a/docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_compare.png b/docs/source/_static/images/benchmark-results/0.2.0/tpch_queries_compare.png similarity index 100% rename from docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_compare.png rename to docs/source/_static/images/benchmark-results/0.2.0/tpch_queries_compare.png diff --git a/docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_speedup_rel.png b/docs/source/_static/images/benchmark-results/0.2.0/tpch_queries_speedup_rel.png similarity index 100% rename from docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_speedup_rel.png rename to docs/source/_static/images/benchmark-results/0.2.0/tpch_queries_speedup_rel.png diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_allqueries.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_allqueries.png new file mode 100644 index 0000000000..ed82f83269 Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_allqueries.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_compare.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_compare.png new file mode 100644 index 0000000000..c0ddff31eb Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_compare.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_abs.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_abs.png new file mode 100644 index 0000000000..e2146caf84 Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_abs.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_rel.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_rel.png new file mode 100644 index 0000000000..b51d00b589 Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_rel.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png new file mode 100644 index 0000000000..8fbc3d45c8 Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png new file mode 100644 index 0000000000..42aec60ebd Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png new file mode 100644 index 0000000000..7a60581554 Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png differ diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png new file mode 100644 index 0000000000..68ac562643 Binary files /dev/null and b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png differ diff --git a/docs/source/contributor-guide/benchmark-results/2024-08-23/comet-tpcds.json b/docs/source/contributor-guide/benchmark-results/0.2.0/comet-tpcds.json similarity index 100% rename from docs/source/contributor-guide/benchmark-results/2024-08-23/comet-tpcds.json rename to docs/source/contributor-guide/benchmark-results/0.2.0/comet-tpcds.json diff --git a/docs/source/contributor-guide/benchmark-results/2024-08-23/comet-tpch.json b/docs/source/contributor-guide/benchmark-results/0.2.0/comet-tpch.json similarity index 100% rename from docs/source/contributor-guide/benchmark-results/2024-08-23/comet-tpch.json rename to docs/source/contributor-guide/benchmark-results/0.2.0/comet-tpch.json diff --git a/docs/source/contributor-guide/benchmark-results/2024-08-23/datafusion-python-tpch.json b/docs/source/contributor-guide/benchmark-results/0.2.0/datafusion-python-tpch.json similarity index 100% rename from docs/source/contributor-guide/benchmark-results/2024-08-23/datafusion-python-tpch.json rename to docs/source/contributor-guide/benchmark-results/0.2.0/datafusion-python-tpch.json diff --git a/docs/source/contributor-guide/benchmark-results/2024-08-23/spark-tpcds.json b/docs/source/contributor-guide/benchmark-results/0.2.0/spark-tpcds.json similarity index 100% rename from docs/source/contributor-guide/benchmark-results/2024-08-23/spark-tpcds.json rename to docs/source/contributor-guide/benchmark-results/0.2.0/spark-tpcds.json diff --git a/docs/source/contributor-guide/benchmark-results/2024-08-23/spark-tpch.json b/docs/source/contributor-guide/benchmark-results/0.2.0/spark-tpch.json similarity index 100% rename from docs/source/contributor-guide/benchmark-results/2024-08-23/spark-tpch.json rename to docs/source/contributor-guide/benchmark-results/0.2.0/spark-tpch.json diff --git a/docs/source/contributor-guide/benchmark-results/0.3.0/comet-tpcds.json b/docs/source/contributor-guide/benchmark-results/0.3.0/comet-tpcds.json new file mode 100644 index 0000000000..f1c60c1477 --- /dev/null +++ b/docs/source/contributor-guide/benchmark-results/0.3.0/comet-tpcds.json @@ -0,0 +1,551 @@ +{ + "engine": "datafusion-comet", + "benchmark": "tpcds", + "data_path": "/mnt/bigdata/tpcds/sf100/", + "query_path": "../../tpcds/queries-spark", + "spark_conf": { + "spark.eventLog.enabled": "true", + "spark.app.startTime": "1727280150257", + "spark.comet.cast.allowIncompatible": "true", + "spark.comet.explainFallback.enabled": "false", + "spark.comet.exec.shuffle.enabled": "true", + "spark.app.name": "DataFusion Comet Benchmark derived from TPC-H / TPC-DS", + "spark.memory.offHeap.enabled": "true", + "spark.app.submitTime": "1727280149981", + "spark.jars": "file:///opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.driver.port": "45021", + "spark.repl.local.jars": "file:///opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.serializer.objectStreamReset": "100", + "spark.comet.exec.optimizer.enabled": "true", + "spark.submit.deployMode": "client", + "spark.sql.sources.useV1SourceList": "avro,parquet", + "spark.executor.cores": "8", + "spark.driver.host": "woody.lan", + "spark.driver.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.sql.warehouse.dir": "file:/home/andy/git/apache/datafusion-benchmarks/runners/datafusion-comet/spark-warehouse", + "spark.driver.maxResultSize": "2G", + "spark.shuffle.manager": "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager", + "spark.comet.exec.enabled": "true", + "spark.comet.scan.enabled": "true", + "spark.memory.offHeap.size": "20g", + "spark.comet.exec.stddev.enabled": "false", + "spark.master": "spark://woody:7077", + "spark.executor.id": "driver", + "spark.comet.exec.shuffle.mode": "auto", + "spark.executor.memoryOverhead": "9829M", + "spark.driver.memory": "8G", + "spark.plugins": "org.apache.spark.CometPlugin", + "spark.sql.extensions": "org.apache.comet.CometSparkSessionExtensions", + "spark.comet.scan.preFetch.enabled": "false", + "spark.app.id": "app-20240925100230-0066", + "spark.executor.memory": "32G", + "spark.comet.parquet.read.parallel.io.enabled": "true", + "spark.comet.convert.parquet.enabled": "true", + "spark.executor.extraClassPath": "/opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.rdd.compress": "True", + "spark.executor.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.app.initial.jar.urls": "spark://woody.lan:45021/jars/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.comet.enabled": "true", + "spark.executor.instances": "1", + "spark.submit.pyFiles": "", + "spark.cores.max": "8", + "spark.comet.exec.sortMergeJoin.enabled": "true", + "spark.driver.extraClassPath": "/opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.comet.explain.native.enabled": "false" + }, + "1": [ + 3.816532850265503, + 1.5830879211425781, + 1.453113317489624 + ], + "2": [ + 3.4293885231018066, + 2.565912961959839, + 2.6678359508514404 + ], + "3": [ + 2.155968427658081, + 1.799889087677002, + 1.799043893814087 + ], + "4": [ + 52.68221926689148, + 51.16434931755066, + 51.50085258483887 + ], + "5": [ + 8.645052194595337, + 7.996783256530762, + 7.958329439163208 + ], + "6": [ + 2.7072160243988037, + 2.643002510070801, + 2.6134305000305176 + ], + "7": [ + 4.35659646987915, + 4.112969398498535, + 4.143396615982056 + ], + "8": [ + 2.3853414058685303, + 2.208953619003296, + 2.2727391719818115 + ], + "9": [ + 8.30320119857788, + 8.015457153320312, + 8.220923662185669 + ], + "10": [ + 3.4127700328826904, + 3.2664177417755127, + 3.2959940433502197 + ], + "11": [ + 25.62278699874878, + 28.673637866973877, + 28.277305364608765 + ], + "12": [ + 1.674375057220459, + 1.4291894435882568, + 1.4657387733459473 + ], + "13": [ + 5.160863161087036, + 5.096550703048706, + 5.129783868789673 + ], + "14": [ + 39.00242233276367, + 38.167762756347656, + 38.97830009460449 + ], + "15": [ + 5.700632095336914, + 5.7491090297698975, + 5.677053451538086 + ], + "16": [ + 14.03771185874939, + 13.499528884887695, + 13.686702489852905 + ], + "17": [ + 5.540913343429565, + 5.608847141265869, + 5.417699337005615 + ], + "18": [ + 4.35630989074707, + 4.431116104125977, + 4.370074033737183 + ], + "19": [ + 4.982686758041382, + 5.0598976612091064, + 5.0963966846466064 + ], + "20": [ + 2.2973546981811523, + 2.2430708408355713, + 2.2421908378601074 + ], + "21": [ + 3.1087920665740967, + 2.943782091140747, + 2.9603769779205322 + ], + "22": [ + 13.771265029907227, + 14.794103145599365, + 13.931464195251465 + ], + "23": [ + 69.57482981681824, + 69.43873310089111, + 69.58835792541504 + ], + "24": [ + 23.1614191532135, + 23.163695335388184, + 24.0635769367218 + ], + "25": [ + 5.684634447097778, + 5.67729115486145, + 5.756789445877075 + ], + "26": [ + 1.8555688858032227, + 1.8365309238433838, + 1.9008393287658691 + ], + "27": [ + 4.151207447052002, + 4.1996071338653564, + 4.266855478286743 + ], + "28": [ + 9.944178342819214, + 10.027048826217651, + 10.200180292129517 + ], + "29": [ + 9.452579498291016, + 9.690422773361206, + 9.583150625228882 + ], + "30": [ + 1.6411223411560059, + 1.6098473072052002, + 1.629262924194336 + ], + "31": [ + 10.478539228439331, + 10.830296277999878, + 11.075891017913818 + ], + "32": [ + 2.332636833190918, + 2.3953561782836914, + 2.4027011394500732 + ], + "33": [ + 4.772242546081543, + 4.851176023483276, + 4.87350058555603 + ], + "34": [ + 2.590566873550415, + 2.5666065216064453, + 2.551725149154663 + ], + "35": [ + 4.480301380157471, + 4.39709210395813, + 4.982789754867554 + ], + "36": [ + 4.541256666183472, + 4.603529691696167, + 4.545770168304443 + ], + "37": [ + 4.692156076431274, + 4.8345627784729, + 4.833616256713867 + ], + "38": [ + 9.658130168914795, + 10.484132528305054, + 9.307052850723267 + ], + "39": [ + 15.334113359451294, + 15.607725858688354, + 15.690910816192627 + ], + "40": [ + 8.385895729064941, + 8.604487180709839, + 8.499430656433105 + ], + "41": [ + 0.15598559379577637, + 0.12737298011779785, + 0.12588715553283691 + ], + "42": [ + 1.6363544464111328, + 1.6473345756530762, + 1.660442590713501 + ], + "43": [ + 2.309523582458496, + 2.367438554763794, + 2.3620736598968506 + ], + "44": [ + 2.3491501808166504, + 2.333522319793701, + 2.3469016551971436 + ], + "45": [ + 3.148803949356079, + 3.086167573928833, + 3.144118547439575 + ], + "46": [ + 5.4704461097717285, + 5.293477535247803, + 5.371654510498047 + ], + "47": [ + 8.775780200958252, + 8.780009031295776, + 8.853479623794556 + ], + "48": [ + 3.7787883281707764, + 3.846055746078491, + 3.8031275272369385 + ], + "49": [ + 9.96819806098938, + 10.018867254257202, + 10.208271980285645 + ], + "50": [ + 10.743677139282227, + 10.664198160171509, + 10.71104621887207 + ], + "51": [ + 15.080579042434692, + 14.893556118011475, + 14.929142475128174 + ], + "52": [ + 1.7665414810180664, + 1.7757434844970703, + 1.8361780643463135 + ], + "53": [ + 2.257715940475464, + 2.269807815551758, + 2.3653101921081543 + ], + "54": [ + 6.051342010498047, + 5.924309492111206, + 5.830335855484009 + ], + "55": [ + 1.8112921714782715, + 1.8508126735687256, + 1.8144702911376953 + ], + "56": [ + 3.9956424236297607, + 3.8586716651916504, + 4.068262338638306 + ], + "57": [ + 4.434924364089966, + 4.541691780090332, + 4.480496883392334 + ], + "58": [ + 6.606714725494385, + 6.617114305496216, + 6.7758948802948 + ], + "59": [ + 5.474972724914551, + 5.6025848388671875, + 5.721226692199707 + ], + "60": [ + 4.074477195739746, + 4.170703649520874, + 4.095849275588989 + ], + "61": [ + 8.984313011169434, + 9.283936500549316, + 9.071494340896606 + ], + "62": [ + 1.4348909854888916, + 1.437838077545166, + 1.420447826385498 + ], + "63": [ + 2.210664987564087, + 2.242994785308838, + 2.2700700759887695 + ], + "64": [ + 22.29287362098694, + 22.06548523902893, + 23.052949905395508 + ], + "65": [ + 11.633065223693848, + 11.980775356292725, + 11.981684684753418 + ], + "66": [ + 4.406243801116943, + 4.385590314865112, + 4.462177038192749 + ], + "67": [ + 55.08261775970459, + 55.632354497909546, + 55.194530963897705 + ], + "68": [ + 5.874284744262695, + 5.873559951782227, + 5.65632438659668 + ], + "69": [ + 3.281365394592285, + 3.7473018169403076, + 3.390223979949951 + ], + "70": [ + 5.336367130279541, + 5.260185241699219, + 5.346327304840088 + ], + "71": [ + 4.3159096240997314, + 4.272960901260376, + 4.360578775405884 + ], + "72": [ + 157.58523988723755, + 158.94361400604248, + 158.75202560424805 + ], + "73": [ + 2.263822555541992, + 2.3628854751586914, + 2.268707275390625 + ], + "74": [ + 21.566213130950928, + 21.691570520401, + 21.462543725967407 + ], + "75": [ + 17.495147228240967, + 17.937005281448364, + 17.643712043762207 + ], + "76": [ + 3.911562204360962, + 3.9740471839904785, + 3.9597601890563965 + ], + "77": [ + 4.440341472625732, + 4.403706073760986, + 4.63965106010437 + ], + "78": [ + 33.38267111778259, + 33.61130118370056, + 33.39215278625488 + ], + "79": [ + 4.204308748245239, + 4.2987847328186035, + 4.121588468551636 + ], + "80": [ + 23.4927659034729, + 23.82388687133789, + 23.482262134552002 + ], + "81": [ + 1.9902129173278809, + 2.1421022415161133, + 1.8984606266021729 + ], + "82": [ + 7.600916624069214, + 7.687270641326904, + 7.578652381896973 + ], + "83": [ + 1.035053014755249, + 1.073936939239502, + 0.9711096286773682 + ], + "84": [ + 0.9261374473571777, + 0.970311164855957, + 0.8874790668487549 + ], + "85": [ + 2.3212785720825195, + 2.288992404937744, + 2.2958807945251465 + ], + "86": [ + 1.1350421905517578, + 1.2193758487701416, + 1.0820298194885254 + ], + "87": [ + 10.146758317947388, + 7.282029628753662, + 6.987994194030762 + ], + "88": [ + 10.508327960968018, + 10.807860136032104, + 10.596072435379028 + ], + "89": [ + 2.761861801147461, + 2.8541927337646484, + 2.8903892040252686 + ], + "90": [ + 0.6046450138092041, + 0.640927791595459, + 0.5932576656341553 + ], + "91": [ + 0.6034653186798096, + 0.7081351280212402, + 0.6046607494354248 + ], + "92": [ + 0.9139211177825928, + 0.965613842010498, + 0.8822739124298096 + ], + "93": [ + 15.011253595352173, + 15.154502868652344, + 15.64320969581604 + ], + "94": [ + 7.04274845123291, + 7.1615777015686035, + 7.122234344482422 + ], + "95": [ + 27.565587997436523, + 27.728916883468628, + 28.071967840194702 + ], + "96": [ + 1.3260400295257568, + 1.382220983505249, + 1.3186144828796387 + ], + "97": [ + 18.267298698425293, + 18.40156888961792, + 19.765196800231934 + ], + "98": [ + 4.867635726928711, + 5.010523557662964, + 4.995208978652954 + ], + "99": [ + 2.577802896499634, + 2.631770133972168, + 2.6019437313079834 + ] +} \ No newline at end of file diff --git a/docs/source/contributor-guide/benchmark-results/0.3.0/comet-tpch.json b/docs/source/contributor-guide/benchmark-results/0.3.0/comet-tpch.json new file mode 100644 index 0000000000..a944d634eb --- /dev/null +++ b/docs/source/contributor-guide/benchmark-results/0.3.0/comet-tpch.json @@ -0,0 +1,158 @@ +{ + "engine": "datafusion-comet", + "benchmark": "tpch", + "data_path": "/mnt/bigdata/tpch/sf100/", + "query_path": "../../tpch/queries", + "spark_conf": { + "spark.app.id": "app-20240925073153-0063", + "spark.eventLog.enabled": "true", + "spark.comet.cast.allowIncompatible": "true", + "spark.comet.explainFallback.enabled": "false", + "spark.comet.exec.shuffle.enabled": "true", + "spark.app.startTime": "1727271112517", + "spark.app.name": "DataFusion Comet Benchmark derived from TPC-H / TPC-DS", + "spark.jars": "file:///opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.repl.local.jars": "file:///opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.driver.port": "34811", + "spark.serializer.objectStreamReset": "100", + "spark.sql.shuffle.partitions": "200", + "spark.comet.exec.optimizer.enabled": "true", + "spark.submit.deployMode": "client", + "spark.comet.exec.all.enabled": "true", + "spark.executor.cores": "8", + "spark.driver.host": "woody.lan", + "spark.app.initial.jar.urls": "spark://woody.lan:34811/jars/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.app.submitTime": "1727271112211", + "spark.driver.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.sql.warehouse.dir": "file:/home/andy/git/apache/datafusion-benchmarks/runners/datafusion-comet/spark-warehouse", + "spark.shuffle.manager": "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager", + "spark.comet.exec.enabled": "true", + "spark.executor.id": "driver", + "spark.master": "spark://woody:7077", + "spark.comet.exec.shuffle.mode": "auto", + "spark.executor.memoryOverhead": "9829M", + "spark.sql.extensions": "org.apache.comet.CometSparkSessionExtensions", + "spark.driver.memory": "8G", + "spark.plugins": "org.apache.spark.CometPlugin", + "spark.executor.memory": "32G", + "spark.executor.extraClassPath": "/opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.rdd.compress": "True", + "spark.executor.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.executor.instances": "1", + "spark.cores.max": "8", + "spark.comet.enabled": "true", + "spark.submit.pyFiles": "", + "spark.driver.extraClassPath": "/opt/comet-0.3.0-rc1/comet-spark-spark3.4_2.12-0.3.0.jar", + "spark.comet.explain.native.enabled": "false" + }, + "1": [ + 12.971000671386719, + 11.223021030426025, + 11.336856842041016 + ], + "2": [ + 9.000791788101196, + 8.1630539894104, + 8.247971773147583 + ], + "3": [ + 18.040359020233154, + 17.38596749305725, + 17.590354442596436 + ], + "4": [ + 9.595617294311523, + 9.145000457763672, + 9.266036748886108 + ], + "5": [ + 32.39970541000366, + 31.349050283432007, + 31.327285766601562 + ], + "6": [ + 2.6660282611846924, + 2.538214683532715, + 2.548441171646118 + ], + "7": [ + 19.370503664016724, + 18.865960359573364, + 18.848506212234497 + ], + "8": [ + 35.31184720993042, + 34.60439658164978, + 34.68396043777466 + ], + "9": [ + 52.99108004570007, + 52.737311363220215, + 52.82331967353821 + ], + "10": [ + 16.981639623641968, + 16.52319836616516, + 16.779926300048828 + ], + "11": [ + 7.6612114906311035, + 7.585915565490723, + 7.590675354003906 + ], + "12": [ + 7.741648197174072, + 7.382411003112793, + 7.447863340377808 + ], + "13": [ + 9.488050699234009, + 9.235808610916138, + 9.207865953445435 + ], + "14": [ + 5.5587310791015625, + 5.3419411182403564, + 5.352286338806152 + ], + "15": [ + 9.539324760437012, + 9.04695463180542, + 9.065588235855103 + ], + "16": [ + 8.170030355453491, + 6.349788427352905, + 6.321645021438599 + ], + "17": [ + 33.64475202560425, + 33.39249873161316, + 33.553722858428955 + ], + "18": [ + 31.16910982131958, + 30.762484073638916, + 31.085059881210327 + ], + "19": [ + 5.836177349090576, + 5.904679298400879, + 5.887378692626953 + ], + "20": [ + 8.477156639099121, + 8.255759716033936, + 8.207967281341553 + ], + "21": [ + 42.812713384628296, + 42.157392263412476, + 42.44283938407898 + ], + "22": [ + 4.80202579498291, + 4.737614631652832, + 4.709359645843506 + ] +} \ No newline at end of file diff --git a/docs/source/contributor-guide/benchmark-results/0.3.0/datafusion-python-tpch.json b/docs/source/contributor-guide/benchmark-results/0.3.0/datafusion-python-tpch.json new file mode 100644 index 0000000000..8f6ca23a29 --- /dev/null +++ b/docs/source/contributor-guide/benchmark-results/0.3.0/datafusion-python-tpch.json @@ -0,0 +1,73 @@ +{ + "engine": "datafusion-python", + "datafusion-version": "40.1.0", + "benchmark": "tpch", + "data_path": "/mnt/bigdata/tpch/sf100", + "query_path": "../../tpch/queries/", + "1": [ + 5.120208740234375 + ], + "2": [ + 3.7675418853759766 + ], + "3": [ + 5.07387113571167 + ], + "4": [ + 1.9587907791137695 + ], + "5": [ + 8.816755771636963 + ], + "6": [ + 1.0937256813049316 + ], + "7": [ + 17.575931549072266 + ], + "8": [ + 8.337157964706421 + ], + "9": [ + 17.257150411605835 + ], + "10": [ + 7.945651292800903 + ], + "11": [ + 2.7080790996551514 + ], + "12": [ + 1.7468397617340088 + ], + "13": [ + 5.695569276809692 + ], + "14": [ + 2.0220673084259033 + ], + "15": [ + 4.112257719039917 + ], + "16": [ + 1.9162819385528564 + ], + "17": [ + 21.812944650650024 + ], + "18": [ + 23.510076999664307 + ], + "19": [ + 2.842766761779785 + ], + "20": [ + 7.717615842819214 + ], + "21": [ + 19.96519947052002 + ], + "22": [ + 1.0273211002349854 + ] +} \ No newline at end of file diff --git a/docs/source/contributor-guide/benchmark-results/0.3.0/spark-tpcds.json b/docs/source/contributor-guide/benchmark-results/0.3.0/spark-tpcds.json new file mode 100644 index 0000000000..ee14ebf8b1 --- /dev/null +++ b/docs/source/contributor-guide/benchmark-results/0.3.0/spark-tpcds.json @@ -0,0 +1,524 @@ +{ + "engine": "datafusion-comet", + "benchmark": "tpcds", + "data_path": "/mnt/bigdata/tpcds/sf100/", + "query_path": "../../tpcds/queries-spark", + "spark_conf": { + "spark.driver.port": "44463", + "spark.eventLog.enabled": "true", + "spark.driver.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.sql.warehouse.dir": "file:/home/andy/git/apache/datafusion-benchmarks/runners/datafusion-comet/spark-warehouse", + "spark.app.startTime": "1727283728621", + "spark.master": "spark://woody:7077", + "spark.executor.id": "driver", + "spark.app.name": "DataFusion Comet Benchmark derived from TPC-H / TPC-DS", + "spark.driver.memory": "8G", + "spark.executor.memory": "32G", + "spark.app.id": "app-20240925110209-0067", + "spark.rdd.compress": "True", + "spark.executor.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.serializer.objectStreamReset": "100", + "spark.executor.instances": "1", + "spark.cores.max": "8", + "spark.submit.pyFiles": "", + "spark.submit.deployMode": "client", + "spark.app.submitTime": "1727283728323", + "spark.executor.cores": "8", + "spark.driver.host": "woody.lan" + }, + "1": [ + 5.148470163345337, + 2.36395001411438, + 2.3829591274261475 + ], + "2": [ + 3.7131166458129883, + 3.016281843185425, + 2.998471260070801 + ], + "3": [ + 1.8333020210266113, + 1.6105778217315674, + 1.5797779560089111 + ], + "4": [ + 63.02493453025818, + 61.70905685424805, + 61.739248514175415 + ], + "5": [ + 13.236883401870728, + 12.700476169586182, + 12.588982105255127 + ], + "6": [ + 2.8634281158447266, + 2.5480568408966064, + 2.5376477241516113 + ], + "7": [ + 3.803929328918457, + 3.429166316986084, + 3.450505495071411 + ], + "8": [ + 2.494192600250244, + 2.2105562686920166, + 2.2187435626983643 + ], + "9": [ + 7.078914165496826, + 6.6791112422943115, + 6.821255445480347 + ], + "10": [ + 3.3953888416290283, + 2.905381917953491, + 3.076423406600952 + ], + "11": [ + 21.38438391685486, + 19.796375274658203, + 20.1093692779541 + ], + "12": [ + 1.2897343635559082, + 1.1005258560180664, + 0.9992177486419678 + ], + "13": [ + 5.353672981262207, + 4.162343502044678, + 4.228771448135376 + ], + "14": [ + 52.11452293395996, + 50.927995920181274, + 51.1369264125824 + ], + "15": [ + 4.239675998687744, + 3.907071352005005, + 3.8751213550567627 + ], + "16": [ + 15.561628341674805, + 15.79569387435913, + 15.450873851776123 + ], + "17": [ + 6.133510112762451, + 6.383626461029053, + 6.069354057312012 + ], + "18": [ + 4.332320690155029, + 4.148226976394653, + 4.21068549156189 + ], + "19": [ + 2.850370407104492, + 2.8427679538726807, + 3.0280649662017822 + ], + "20": [ + 1.3475830554962158, + 1.261359691619873, + 1.285959243774414 + ], + "21": [ + 2.9580078125, + 2.6976616382598877, + 2.8327407836914062 + ], + "22": [ + 25.865466356277466, + 26.382410049438477, + 26.69676446914673 + ], + "23": [ + 128.46632599830627, + 126.95195817947388, + 127.14342927932739 + ], + "24": [ + 20.395217657089233, + 19.451897382736206, + 18.934618711471558 + ], + "25": [ + 5.7130351066589355, + 5.470770835876465, + 5.496434688568115 + ], + "26": [ + 2.0510032176971436, + 2.269962787628174, + 2.0385682582855225 + ], + "27": [ + 3.1545770168304443, + 3.2275774478912354, + 3.0304830074310303 + ], + "28": [ + 11.161650657653809, + 10.845736026763916, + 10.703468799591064 + ], + "29": [ + 6.834911823272705, + 6.716768264770508, + 7.089026927947998 + ], + "30": [ + 2.2383787631988525, + 2.344122886657715, + 2.2877557277679443 + ], + "31": [ + 7.08621883392334, + 6.844470500946045, + 6.877709865570068 + ], + "32": [ + 2.0182371139526367, + 1.9883370399475098, + 2.0473082065582275 + ], + "33": [ + 3.645477294921875, + 3.6712701320648193, + 3.3282854557037354 + ], + "34": [ + 2.3055543899536133, + 2.3120596408843994, + 2.2760725021362305 + ], + "35": [ + 4.114598274230957, + 4.314220905303955, + 4.346586227416992 + ], + "36": [ + 2.717860460281372, + 2.626767873764038, + 2.603105306625366 + ], + "37": [ + 6.1655378341674805, + 5.978333473205566, + 6.085343599319458 + ], + "38": [ + 8.834673643112183, + 9.236085414886475, + 8.568750381469727 + ], + "39": [ + 9.26044511795044, + 9.1658616065979, + 9.236857652664185 + ], + "40": [ + 10.207687854766846, + 10.389538526535034, + 11.357187271118164 + ], + "41": [ + 0.2273883819580078, + 0.20333337783813477, + 0.20030641555786133 + ], + "42": [ + 1.4819755554199219, + 1.575575828552246, + 1.5451006889343262 + ], + "43": [ + 1.8859820365905762, + 1.699096441268921, + 1.682723045349121 + ], + "44": [ + 1.0896844863891602, + 1.0091238021850586, + 0.9841156005859375 + ], + "45": [ + 2.0448622703552246, + 2.071485996246338, + 1.8746685981750488 + ], + "46": [ + 3.9426820278167725, + 3.5915961265563965, + 3.412421703338623 + ], + "47": [ + 6.88372540473938, + 6.5370237827301025, + 6.394831895828247 + ], + "48": [ + 9.33659291267395, + 8.765706062316895, + 8.697250366210938 + ], + "49": [ + 9.317560195922852, + 8.850082874298096, + 9.48132848739624 + ], + "50": [ + 14.689724683761597, + 14.642268419265747, + 15.39272665977478 + ], + "51": [ + 20.38764500617981, + 20.185483694076538, + 20.54289960861206 + ], + "52": [ + 1.8236849308013916, + 1.8166379928588867, + 1.7648234367370605 + ], + "53": [ + 2.407712936401367, + 2.203848123550415, + 2.3282134532928467 + ], + "54": [ + 4.095530986785889, + 3.9434800148010254, + 3.975391149520874 + ], + "55": [ + 1.8237614631652832, + 1.722341775894165, + 1.6877577304840088 + ], + "56": [ + 3.6321635246276855, + 3.1052613258361816, + 3.2294297218322754 + ], + "57": [ + 3.6315739154815674, + 3.426823139190674, + 3.525855779647827 + ], + "58": [ + 4.154082298278809, + 3.8242034912109375, + 3.7452712059020996 + ], + "59": [ + 5.0502402782440186, + 4.763699531555176, + 4.827987432479858 + ], + "60": [ + 3.3202879428863525, + 3.0526845455169678, + 3.105377197265625 + ], + "61": [ + 4.481921672821045, + 4.371694087982178, + 4.577329874038696 + ], + "62": [ + 1.3757414817810059, + 1.2972588539123535, + 1.3132882118225098 + ], + "63": [ + 2.2113375663757324, + 2.1810450553894043, + 2.202092409133911 + ], + "64": [ + 31.69989562034607, + 31.285507917404175, + 30.000827074050903 + ], + "65": [ + 15.508227825164795, + 15.570954084396362, + 15.413981914520264 + ], + "66": [ + 3.5777909755706787, + 3.5254597663879395, + 3.4596121311187744 + ], + "67": [ + 103.37566041946411, + 99.12435960769653, + 100.88314938545227 + ], + "68": [ + 3.669990062713623, + 3.9507150650024414, + 3.7124698162078857 + ], + "69": [ + 2.6502199172973633, + 2.953432559967041, + 2.747390031814575 + ], + "70": [ + 4.107975006103516, + 4.202328205108643, + 4.052009582519531 + ], + "71": [ + 3.6340830326080322, + 3.575644016265869, + 3.530362844467163 + ], + "72": [ + 162.64638757705688, + 161.72935438156128, + 164.1088535785675 + ], + "73": [ + 2.101400136947632, + 2.1288535594940186, + 2.0507609844207764 + ], + "74": [ + 25.46847701072693, + 23.589718103408813, + 24.60220503807068 + ], + "75": [ + 16.084983348846436, + 15.937219858169556, + 16.16034722328186 + ], + "76": [ + 3.6872665882110596, + 3.2498128414154053, + 3.1501200199127197 + ], + "77": [ + 4.002533674240112, + 3.9785006046295166, + 4.237426280975342 + ], + "78": [ + 44.70254564285278, + 43.92543983459473, + 44.28680419921875 + ], + "79": [ + 3.171165704727173, + 3.1139674186706543, + 3.2380402088165283 + ], + "80": [ + 39.16297960281372, + 38.643226861953735, + 38.34842777252197 + ], + "81": [ + 2.531498432159424, + 2.4354569911956787, + 2.612056255340576 + ], + "82": [ + 12.046911239624023, + 11.19921350479126, + 11.121217966079712 + ], + "83": [ + 1.6479520797729492, + 1.853271722793579, + 1.8834664821624756 + ], + "84": [ + 1.683358907699585, + 1.5293524265289307, + 1.507997751235962 + ], + "85": [ + 4.9209089279174805, + 5.000877618789673, + 4.839233160018921 + ], + "86": [ + 1.2056818008422852, + 1.1538653373718262, + 1.242598295211792 + ], + "87": [ + 9.39367413520813, + 9.151134014129639, + 9.208250999450684 + ], + "88": [ + 8.107990503311157, + 8.18473196029663, + 8.227517127990723 + ], + "89": [ + 2.739013433456421, + 2.678351879119873, + 2.683817148208618 + ], + "90": [ + 0.8000590801239014, + 0.8856337070465088, + 0.7577085494995117 + ], + "91": [ + 0.9848968982696533, + 0.9668941497802734, + 1.1044719219207764 + ], + "92": [ + 1.4028904438018799, + 1.3235609531402588, + 1.4188199043273926 + ], + "93": [ + 34.74214005470276, + 34.99839782714844, + 34.499898195266724 + ], + "94": [ + 8.9618239402771, + 8.190898656845093, + 8.174163341522217 + ], + "95": [ + 28.088911294937134, + 28.05694031715393, + 27.966723918914795 + ], + "96": [ + 1.152230978012085, + 1.1219878196716309, + 1.1294939517974854 + ], + "97": [ + 13.105412721633911, + 13.00275731086731, + 13.133348226547241 + ], + "98": [ + 2.657196044921875, + 2.6066102981567383, + 2.633361577987671 + ], + "99": [ + 2.0350334644317627, + 1.960425853729248, + 1.8787071704864502 + ] +} \ No newline at end of file diff --git a/docs/source/contributor-guide/benchmark-results/0.3.0/spark-tpch.json b/docs/source/contributor-guide/benchmark-results/0.3.0/spark-tpch.json new file mode 100644 index 0000000000..1837c82e6f --- /dev/null +++ b/docs/source/contributor-guide/benchmark-results/0.3.0/spark-tpch.json @@ -0,0 +1,139 @@ +{ + "engine": "datafusion-comet", + "benchmark": "tpch", + "data_path": "/mnt/bigdata/tpch/sf100/", + "query_path": "../../tpch/queries", + "spark_conf": { + "spark.app.submitTime": "1727275341119", + "spark.eventLog.enabled": "true", + "spark.driver.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.sql.warehouse.dir": "file:/home/andy/git/apache/datafusion-benchmarks/runners/datafusion-comet/spark-warehouse", + "spark.executor.id": "driver", + "spark.master": "spark://woody:7077", + "spark.app.name": "DataFusion Comet Benchmark derived from TPC-H / TPC-DS", + "spark.driver.memory": "8G", + "spark.driver.port": "44751", + "spark.executor.memory": "32G", + "spark.app.id": "app-20240925084221-0064", + "spark.rdd.compress": "True", + "spark.executor.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", + "spark.app.startTime": "1727275341382", + "spark.serializer.objectStreamReset": "100", + "spark.executor.instances": "1", + "spark.cores.max": "8", + "spark.submit.pyFiles": "", + "spark.submit.deployMode": "client", + "spark.executor.cores": "8", + "spark.driver.host": "woody.lan" + }, + "1": [ + 104.32338833808899, + 101.92840933799744, + 102.29649090766907 + ], + "2": [ + 12.577291488647461, + 11.24374008178711, + 11.141963481903076 + ], + "3": [ + 23.692935943603516, + 21.609262943267822, + 21.613842725753784 + ], + "4": [ + 16.685965538024902, + 16.50067663192749, + 16.418885707855225 + ], + "5": [ + 43.853904008865356, + 43.304861545562744, + 43.29374384880066 + ], + "6": [ + 3.5047988891601562, + 3.3367137908935547, + 3.3941538333892822 + ], + "7": [ + 20.229097604751587, + 19.84404230117798, + 19.959458589553833 + ], + "8": [ + 31.940091371536255, + 31.665536165237427, + 32.250579833984375 + ], + "9": [ + 66.0380346775055, + 65.85967254638672, + 66.10620164871216 + ], + "10": [ + 19.16370391845703, + 18.043572902679443, + 18.115468978881836 + ], + "11": [ + 11.898521423339844, + 11.491883993148804, + 11.488958597183228 + ], + "12": [ + 12.325279474258423, + 12.277081489562988, + 12.350794553756714 + ], + "13": [ + 20.350739002227783, + 20.20111894607544, + 20.239644289016724 + ], + "14": [ + 5.517431259155273, + 5.459251880645752, + 5.43597936630249 + ], + "15": [ + 15.64854121208191, + 15.332119703292847, + 15.386141300201416 + ], + "16": [ + 6.227680444717407, + 6.065835237503052, + 6.085387706756592 + ], + "17": [ + 56.808584690093994, + 57.28081512451172, + 56.35479974746704 + ], + "18": [ + 71.03480815887451, + 69.92680954933167, + 70.1363697052002 + ], + "19": [ + 6.895759105682373, + 6.749705791473389, + 7.524952173233032 + ], + "20": [ + 9.70884919166565, + 9.403893232345581, + 9.452422857284546 + ], + "21": [ + 58.89342188835144, + 58.909135580062866, + 58.7743935585022 + ], + "22": [ + 8.485138177871704, + 9.303241729736328, + 8.181260108947754 + ] +} \ No newline at end of file diff --git a/docs/source/contributor-guide/benchmark-results/tpc-ds.md b/docs/source/contributor-guide/benchmark-results/tpc-ds.md index 0aba020e1e..5575fef021 100644 --- a/docs/source/contributor-guide/benchmark-results/tpc-ds.md +++ b/docs/source/contributor-guide/benchmark-results/tpc-ds.md @@ -20,24 +20,26 @@ under the License. # Apache DataFusion Comet: Benchmarks Derived From TPC-DS The following benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and -data stored locally on NVMe storage. Performance characteristics will vary in different environments and we encourage -you to run these benchmarks in your own environments. +data stored locally in Parquet format on NVMe storage. Performance characteristics will vary in different environments +and we encourage you to run these benchmarks in your own environments. -Comet can not yet run all TPC-DS queries natively. The tracking issue is [#858](https://github.com/apache/datafusion-comet/issues/858). +The tracking issue for improving TPC-DS performance is [#858](https://github.com/apache/datafusion-comet/issues/858). -Comet currently provides a 21% speedup for TPC-DS @ SF=100GB. - -![](../../_static/images/benchmark-results/2024-08-23/tpcds_allqueries.png) +![](../../_static/images/benchmark-results/0.3.0/tpcds_allqueries.png) Here is a breakdown showing relative performance of Spark and Comet for each query. -![](../../_static/images/benchmark-results/2024-08-23/tpcds_queries_compare.png) +![](../../_static/images/benchmark-results/0.3.0/tpcds_queries_compare.png) + +The following chart shows how much Comet currently accelerates each query from the benchmark in relative terms. + +![](../../_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_rel.png) -The following chart shows how much Comet currently accelerates each query from the benchmark. +The following chart shows how much Comet currently accelerates each query from the benchmark in absolute terms. -![](../../_static/images/benchmark-results/2024-08-23/tpcds_queries_speedup_abs.png) +![](../../_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_abs.png) The raw results of these benchmarks in JSON format is available here: -- [Spark](./2024-08-23/spark-tpcds.json) -- [Comet](./2024-08-23/comet-tpcds.json) +- [Spark](0.3.0/spark-tpcds.json) +- [Comet](0.3.0/comet-tpcds.json) diff --git a/docs/source/contributor-guide/benchmark-results/tpc-h.md b/docs/source/contributor-guide/benchmark-results/tpc-h.md index 3c78ad778f..22cab0b453 100644 --- a/docs/source/contributor-guide/benchmark-results/tpc-h.md +++ b/docs/source/contributor-guide/benchmark-results/tpc-h.md @@ -20,25 +20,27 @@ under the License. # Apache DataFusion Comet: Benchmarks Derived From TPC-H The following benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and -data stored locally on NVMe storage. Performance characteristics will vary in different environments and we encourage -you to run these benchmarks in your own environments. +data stored locally in Parquet format on NVMe storage. Performance characteristics will vary in different environments +and we encourage you to run these benchmarks in your own environments. -Comet can not yet run all TPC-H queries natively. The tracking issue is [#391](https://github.com/apache/datafusion-comet/issues/391). +The tracking issue for improving TPC-H performance is [#391](https://github.com/apache/datafusion-comet/issues/391). -Comet currently provides a 62% speedup for TPC-H @ SF=100GB. - -![](../../_static/images/benchmark-results/2024-08-23/tpch_allqueries.png) +![](../../_static/images/benchmark-results/0.3.0/tpch_allqueries.png) Here is a breakdown showing relative performance of Spark, Comet, and DataFusion for each query. -![](../../_static/images/benchmark-results/2024-08-23/tpch_queries_compare.png) +![](../../_static/images/benchmark-results/0.3.0/tpch_queries_compare.png) + +The following chart shows how much Comet currently accelerates each query from the benchmark in relative terms. + +![](../../_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png) -The following chart shows how much Comet currently accelerates each query from the benchmark. +The following chart shows how much Comet currently accelerates each query from the benchmark in absolute terms. -![](../../_static/images/benchmark-results/2024-08-23/tpch_queries_speedup_rel.png) +![](../../_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png) The raw results of these benchmarks in JSON format is available here: -- [Spark](./2024-08-23/spark-tpch.json) -- [Comet](./2024-08-23/comet-tpch.json) -- [DataFusion](./2024-08-23/datafusion-python-tpch.json) \ No newline at end of file +- [Spark](0.3.0/spark-tpch.json) +- [Comet](0.3.0/comet-tpch.json) +- [DataFusion](0.3.0/datafusion-python-tpch.json) \ No newline at end of file