pyspark.BarrierTaskContext.barrier¶
-
BarrierTaskContext.barrier()[source]¶ Sets a global barrier and waits until all tasks in this stage hit this barrier. Similar to MPI_Barrier function in MPI, this function blocks until all tasks in the same stage have reached this routine.
New in version 2.4.0.
Warning
In a barrier stage, each task much have the same number of barrier() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout.
Notes
This API is experimental