Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Equality test that is safe for null values.
Added in Databricks Runtime 11.0
Changed in Databricks Runtime 13.0: Supports Spark Connect.
Syntax
eqNullSafe(other)
Parameters
| Parameter | Type | Description |
|---|---|---|
other |
Column or value | A value or Column to compare |
Returns
Column
Notes
Unlike Pandas, PySpark doesn't consider NaN values to be NULL. See the NaN Semantics for details.
Examples
from pyspark.sql import Row
df1 = spark.createDataFrame([
Row(id=1, value='foo'),
Row(id=2, value=None)
])
df1.select(
df1['value'] == 'foo',
df1['value'].eqNullSafe('foo'),
df1['value'].eqNullSafe(None)
).show()
# +-------------+---------------+----------------+
# |(value = foo)|(value <=> foo)|(value <=> NULL)|
# +-------------+---------------+----------------+
# | true| true| false|
# | NULL| false| true|
# +-------------+---------------+----------------+