Version 18.2

This serverless compute release roughly corresponds to Databricks Runtime 18.2 . New features CREATE OR REPLACE support for temporary tables : CREATE OR REPLACE TEMP TABLE syntax is now supported, allowing you to create or replace temporary tables in a single statement. This eliminates the need to explicitly drop and recreate temporary tables. agg() alias for measure() function : agg() is now available as an alias for the measure() function. This change is fully backward compatible. Existing queries that use measure() continue to work without modification, and agg() produces identical results when used with the same arguments. Snowflake JDBC driver upgrade : The Snowflake JDBC driver is upgraded from 3.22.0 to 3.28.0. pyspark.pipelines.testing namespace alias : pyspark.pipelines.testing is now available as a convenience alias for dlt.testing APIs. Import Lakeflow Spark Declarative Pipelines pipeline testing utilities through either namespace. Delta table history includes write option flags : Delta table history ( DESCRIBE HISTORY ) now includes write option flags in the operationParameters column for WRITE and REPLACE TABLE operations. When the following options are explicitly enabled, they appear as boolean flags in the history (only included when true ): For WRITE and REPLACE TABLE operations: isDynamicPartitionOverwrite : present when dynamic partition overwrite mode was used canOverwriteSchema : present when schema overwrite ( overwriteSchema ) was enabled canMergeSchema : present when schema merge ( mergeSchema ) was enabled For REPLACE TABLE operations: predicate : present when replaceWhere was used isV1WriterSaveAsTableOverwrite : present when the replace was triggered by a .saveAsTable overwrite Selectively replace data with replaceOn and replaceUsing DataFrame APIs : The replaceOn and replaceUsing options in the Scala and Python DataFrame APIs are now generally available. Use these options to replace part of the table with the result of a DataFrame. replaceOn replaces rows that match a user-defined condition. replaceUsing replaces rows where specified columns are equal. These APIs complement the INSERT REPLACE ON and INSERT REPLACE USING SQL statements. See Selectively overwrite data with Delta Lake . Behavior changes NULL struct preservation in INSERT, MERGE, and streaming writes with schema evolution : For INSERT , MERGE , and streaming writes that use schema evolution , a NULL struct in the source is now stored as NULL in the target. Previously, that value was incorrectly materialized as a non-null struct with every field set to NULL, while the same operations without schema evolution preserved NULL structs correctly. If your code relied on receiving a non-null struct whose fields were all NULL, update your code to handle a NULL struct instead. Fix for LEFT OUTER JOIN LATERAL dropping rows : A bug that incorrectly dropped rows from LEFT OUTER JOIN LATERAL queries is now fixed. Queries using this construct now return the correct results. To temporarily revert to the previous behavior, set spark.databricks.sql.optimizer.lateralJoinPreserveOuterSemantic to true . NATURAL JOIN respects case-insensitive column matching : NATURAL JOIN now correctly uses case-insensitive column matching when spark.sql.caseSensitive is set to false (the default). Previously, NATURAL JOIN used case-sensitive comparison to identify common columns, causing columns that differed only in case (for example, ID versus id ) to not be recognized as matching. This caused NATURAL JOIN to silently produce cross-join results. This fix aligns NATURAL JOIN behavior with USING joins , which already handled case-insensitivity correctly. Queries affected by this bug now return correct results with properly joined columns. SQL UDF dependency validation in Unity Catalog : Unity Catalog now enforces dependency validation for SQL user-defined functions (UDFs) to prevent access control bypass. Previously, SQL functions created through the REST API could reference dependencies the user did not have access to. SQL UDFs with invalid dependency configurations are now blocked from execution. AWS SDK v1 dependencies are shaded : AWS SDK v1 dependencies bundled with the Azure Databricks runtime are now shaded and no longer directly available on the classpath. If your code depends on AWS SDK v1 libraries previously provided by the Azure Databricks runtime, add them as explicit dependencies in your project. This change prepares for the migration to AWS SDK v2, following the end of AWS support for SDK v1. Fix incorrect EPSG authority for ESRI-defined SRID 102100 : The Coordinate Reference System (CRS) mapping for SRID 102100 now correctly uses ESRI:102100 instead of the incorrect EPSG:102100 . This fix ensures geospatial data is stored with the correct authority for better interoperability with other systems.