pyspark.resource.TaskResourceRequest#

class pyspark.resource.TaskResourceRequest(resourceName, amount)[source]#

A task resource request. This is used in conjunction with the pyspark.resource.ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level. The amount is specified as a float to allow for saying you want more than 1 task per resource. For custom resources, valid values are less than or equal to 0.5 or whole numbers, since a task’s amount must map onto discrete resource addresses. CPUs (resource name “cpus”) are a plain quantity drawn from the executor’s core pool rather than an addressable resource, so any positive amount is valid, e.g. 1.5. Use pyspark.resource.TaskResourceRequests class as a convenience API.

Parameters:

resourceNamestr: Name of the resource
amountfloat: Amount requesting as a float to support fractional resource requests. For custom resources, valid values are less than or equal to 0.5 or whole numbers; this essentially lets you configure X number of tasks to run on a single resource, ie amount equals 0.5 translates into 2 tasks per resource address. For cpus, any positive amount is valid, including fractional ones below 1 (e.g. 0.5) or above 1 (e.g. 1.5).
.. versionadded:: 3.1.0

`amount`
`resourceName`