Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Manages all active StreamingQuery instances associated with a SparkSession. Use spark.streams to access this.
Syntax
# Access through SparkSession
spark.streams
Properties
| Property | Description |
|---|---|
active |
Returns a list of all active streaming queries associated with this SparkSession. |
Methods
| Method | Description |
|---|---|
get(id) |
Returns an active query by its unique ID. |
awaitAnyTermination(timeout) |
Waits until any active query terminates, or until the timeout expires. |
resetTerminated() |
Forgets past terminated queries so that awaitAnyTermination() can be used again to wait for new terminations. |
addListener(listener) |
Registers a StreamingQueryListener to receive lifecycle event callbacks. |
removeListener(listener) |
Deregisters a StreamingQueryListener. |
Examples
sdf = spark.readStream.format("rate").load()
sq = sdf.writeStream.format('memory').queryName('this_query').start()
sqm = spark.streams
[q.name for q in sqm.active]
# ['this_query']
sqm.awaitAnyTermination(5)
# True
sq.stop()
sqm.resetTerminated()