Share via

format (DataStreamWriter)

Specifies the underlying output data source.

Syntax

format(source)

Parameters

Parameter Type Description
source str Name of the data source, for example 'parquet' or 'console'.

Returns

DataStreamWriter

Examples

df = spark.readStream.format("rate").load()
df.writeStream.format("text")
# <...streaming.readwriter.DataStreamWriter object ...>

Write a Rate source stream to CSV:

import tempfile
import time
with tempfile.TemporaryDirectory(prefix="format1") as d:
    with tempfile.TemporaryDirectory(prefix="format2") as cp:
        df = spark.readStream.format("rate").load()
        q = df.writeStream.format("csv").option("checkpointLocation", cp).start(d)
        time.sleep(5)
        q.stop()
        spark.read.schema("timestamp TIMESTAMP, value STRING").csv(d).show()
  • Last updated on