Add spark session extension for Hyperspace

[paryoja]

What is the context for this pull request?

• Tracking Issue: Fixes [FEATURE REQUEST]: Enable hyperspace with SparkSessionExtention #503

What changes were proposed in this pull request?

Added a feature to enable Hyperspace with SparkSessionExtention

Does this PR introduce any user-facing change?

Yes. Users now can enable Hyperspace as follows

spark-shell -c spark.sql.extensions=com.microsoft.hyperspace.HyperspaceSparkSessionExtension

or

val spark = SparkSession
       .builder()
       .appName("...")
       .master("...")
       .config("spark.sql.extensions", "com.microsoft.hyperspace.HyperspaceSparkSessionExtension")
       .getOrCreate()

How was this patch tested?

Manually with spark-shell. If an automated test is required, I will add it.

[sezruby]

LGTM, thanks @paryoja!
@clee704 Could you have another look and approve the PR?

[clee704]

According to the new model, enableHyperspace only exists for backward compatibility. I think it's better to factor out the rule insertion code out of this method and invoke the method here and from enableHyperspace.

[paryoja]

@clee704 Do you mean this part?

      if (!sparkSession.sessionState.experimentalMethods.extraOptimizations.contains(
          ApplyHyperspace)) {
        sparkSession.sessionState.experimentalMethods.extraOptimizations ++=
          ApplyHyperspace :: Nil
      }
      if (!sparkSession.sessionState.experimentalMethods.extraStrategies.contains(
          BucketUnionStrategy)) {
        sparkSession.sessionState.experimentalMethods.extraStrategies ++=
          BucketUnionStrategy :: Nil
      }

Where should I put this code because package object hyperspace is quite customer side interfaces, so not sure if it is ok to create a function like

package object hyperspace {

  /**
   * Hyperspace-specific implicit class on SparkSession.
   */
  implicit class Implicits(sparkSession: SparkSession) {

    def enableHyperspace(): SparkSession = {
      HyperspaceConf.setHyperspaceApplyEnabled(sparkSession, true)
      addOptimizationsIfNeeded()
      sparkSession
    }

    private def addOptimizationsIfNeeded(): Unit = {
      if (!sparkSession.sessionState.experimentalMethods.extraOptimizations.contains(
          ApplyHyperspace)) {
        sparkSession.sessionState.experimentalMethods.extraOptimizations ++=
          ApplyHyperspace :: Nil
      }
      if (!sparkSession.sessionState.experimentalMethods.extraStrategies.contains(
          BucketUnionStrategy)) {
        sparkSession.sessionState.experimentalMethods.extraStrategies ++=
          BucketUnionStrategy :: Nil
      }
    }
  }
}

[clee704]

You can put addOptimizationsIfNeeded() in a companion object HyperspaceSparkSessionExtension and call the method from here and enableHyperspace().

[paryoja]

@clee704 can you check whether I implemented as you expected?

[sezruby]

I think it's better to locate the function in package.scala