Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Specifies the input schema. Some data sources (for example, JSON) can infer the input schema automatically from data. Specifying the schema here allows the data source to skip schema inference and speed up data loading.
Syntax
schema(schema)
Parameters
| Parameter | Type | Description |
|---|---|---|
schema |
StructType or str | A StructType object or a DDL-formatted string (for example, col0 INT, col1 DOUBLE). |
Returns
DataStreamReader
Examples
from pyspark.sql.types import StructField, StructType, StringType
spark.readStream.schema(StructType([StructField("data", StringType(), True)]))
# <...streaming.readwriter.DataStreamReader object ...>
spark.readStream.schema("col0 INT, col1 DOUBLE")
# <...streaming.readwriter.DataStreamReader object ...>
Specify a different schema for a CSV file:
import tempfile
with tempfile.TemporaryDirectory(prefix="schema") as d:
spark.readStream.schema("col0 INT, col1 STRING").format("csv").load(d).printSchema()
# root
# |-- col0: integer (nullable = true)
# |-- col1: string (nullable = true)