pyspark.SparkContext.setLocalProperty

SparkContext.setLocalProperty(key, value)

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.

To remove/unset property simply set value to None e.g. sc.setLocalProperty(“key”, None)

New in version 1.0.0.

Parameters

keystr

The key of the local property to set.

valuestr

The value of the local property to set. If set to None then the property will be removed

See also

SparkContext.getLocalProperty()

Notes

If you run jobs in parallel, use pyspark.InheritableThread for thread local inheritance.