Share via

start (DataStreamWriter)

Streams the contents of the DataFrame to a data source and returns a StreamingQuery object.

Syntax

start(path=None, format=None, outputMode=None, partitionBy=None, queryName=None, **options)

Parameters

Parameter Type Description
path str, optional Path in a Hadoop-supported file system.
format str, optional The format used to save.
outputMode str, optional How data is written to the sink: append, complete, or update.
partitionBy str or list, optional Names of partitioning columns.
queryName str, optional Unique name for the query.
**options
All other string options. Provide checkpointLocation for most streams; not required for a memory stream.

Returns

StreamingQuery

Examples

df = spark.readStream.format("rate").load()

Basic example:

q = df.writeStream.format('memory').queryName('this_query').start()
q.isActive
# True
q.name
# 'this_query'
q.stop()
q.isActive
# False

With a trigger and additional parameters:

q = df.writeStream.trigger(processingTime='5 seconds').start(
    queryName='that_query', outputMode="append", format='memory')
q.name
# 'that_query'
q.isActive
# True
q.stop()
  • Last updated on