From 6ced21570e4c19494caef6fc12b4431a192d0b5f Mon Sep 17 00:00:00 2001 From: Ruifeng Zheng Date: Wed, 23 Jul 2025 16:54:06 +0800 Subject: [PATCH] nit --- python/docs/source/migration_guide/pyspark_upgrade.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst b/python/docs/source/migration_guide/pyspark_upgrade.rst index 50a2488bd3ea9..a758b50050635 100644 --- a/python/docs/source/migration_guide/pyspark_upgrade.rst +++ b/python/docs/source/migration_guide/pyspark_upgrade.rst @@ -22,12 +22,10 @@ Upgrading PySpark Upgrading from PySpark 4.0 to 4.1 --------------------------------- +* In Spark 4.1, Python 3.9 support was dropped in PySpark. * In Spark 4.1, ``DataFrame['name']`` on Spark Connect Python Client no longer eagerly validate the column name. To restore the legacy behavior, set ``PYSPARK_VALIDATE_COLUMN_NAME_LEGACY`` environment variable to ``1``. - * In Spark 4.1, Arrow-optimized Python UDF supports UDT input / output instead of falling back to the regular UDF. To restore the legacy behavior, set ``spark.sql.execution.pythonUDF.arrow.legacy.fallbackOnUDT`` to ``true``. - * In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, enable ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled``. - * In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDTF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, enable ``spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled``. Upgrading from PySpark 3.5 to 4.0