org.apache.spark.scheduler.cluster.mesos

Class MesosSchedulerBackend

Object

org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend

All Implemented Interfaces:

org.apache.mesos.Scheduler, Logging, SchedulerBackend

public class MesosSchedulerBackend
extends Object
implements SchedulerBackend, org.apache.mesos.Scheduler, Logging

A SchedulerBackend for running fine-grained tasks on Mesos. Each Spark task is mapped to a separate Mesos task, allowing multiple applications to share cluster nodes both in space (tasks from multiple apps can run on different cores) and in time (a core can switch ownership).

Constructor Summary

Constructors

Constructor and Description
MesosSchedulerBackend(TaskSchedulerImpl scheduler, SparkContext sc, String master)

Method Summary

Methods

Modifier and Type	Method and Description
String	appId()
String	applicationId() Get an application ID associated with the job.
ClassLoader	classLoader()
org.apache.mesos.Protos.ExecutorInfo	createExecutorInfo(String execId)
org.apache.mesos.Protos.TaskInfo	createMesosTask(TaskDescription task, String slaveId) Turn a Spark TaskDescription into a Mesos task
int	defaultParallelism()
void	disconnected(org.apache.mesos.SchedulerDriver d)
org.apache.mesos.SchedulerDriver	driver()
void	error(org.apache.mesos.SchedulerDriver d, String message)
byte[]	execArgs()
void	executorLost(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.ExecutorID executorId, org.apache.mesos.Protos.SlaveID slaveId, int status)
void	frameworkMessage(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.ExecutorID e, org.apache.mesos.Protos.SlaveID s, byte[] b)
double	getResource(java.util.List<org.apache.mesos.Protos.Resource> res, String name) Helper function to pull out a resource from a Mesos Resources protobuf
boolean	isFinished(org.apache.mesos.Protos.TaskState state) Check whether a Mesos task state represents a finished task
boolean	isRegistered()
void	killTask(long taskId, String executorId, boolean interruptThread)
void	offerRescinded(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.OfferID o)
void	registered(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.FrameworkID frameworkId, org.apache.mesos.Protos.MasterInfo masterInfo)
Object	registeredLock()
void	reregistered(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.MasterInfo masterInfo)
void	resourceOffers(org.apache.mesos.SchedulerDriver d, java.util.List<org.apache.mesos.Protos.Offer> offers) Method called by Mesos to offer resources on slaves.
void	reviveOffers()
scala.collection.mutable.HashSet<String>	slaveIdsWithExecutors()
void	slaveLost(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.SlaveID slaveId)
void	start()
void	statusUpdate(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.TaskStatus status)
void	stop()
scala.collection.mutable.HashMap<Object,String>	taskIdToSlaveId()
void	waitForRegister()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SchedulerBackend

isReady

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

MesosSchedulerBackend

public MesosSchedulerBackend(TaskSchedulerImpl scheduler,
                     SparkContext sc,
                     String master)

Method Detail

isRegistered

public boolean isRegistered()

registeredLock

public Object registeredLock()

driver

public org.apache.mesos.SchedulerDriver driver()

slaveIdsWithExecutors

public scala.collection.mutable.HashSet<String> slaveIdsWithExecutors()

taskIdToSlaveId

public scala.collection.mutable.HashMap<Object,String> taskIdToSlaveId()

execArgs

public byte[] execArgs()

classLoader

public ClassLoader classLoader()

appId

public String appId()

Specified by:

appId in interface SchedulerBackend

start

public void start()

Specified by:

start in interface SchedulerBackend

createExecutorInfo

public org.apache.mesos.Protos.ExecutorInfo createExecutorInfo(String execId)

offerRescinded

public void offerRescinded(org.apache.mesos.SchedulerDriver d,
                  org.apache.mesos.Protos.OfferID o)

Specified by:

offerRescinded in interface org.apache.mesos.Scheduler

registered

public void registered(org.apache.mesos.SchedulerDriver d,
              org.apache.mesos.Protos.FrameworkID frameworkId,
              org.apache.mesos.Protos.MasterInfo masterInfo)

Specified by:

registered in interface org.apache.mesos.Scheduler

waitForRegister

public void waitForRegister()

disconnected

public void disconnected(org.apache.mesos.SchedulerDriver d)

Specified by:

disconnected in interface org.apache.mesos.Scheduler

reregistered

public void reregistered(org.apache.mesos.SchedulerDriver d,
                org.apache.mesos.Protos.MasterInfo masterInfo)

Specified by:

reregistered in interface org.apache.mesos.Scheduler

resourceOffers

public void resourceOffers(org.apache.mesos.SchedulerDriver d,
                  java.util.List<org.apache.mesos.Protos.Offer> offers)

Method called by Mesos to offer resources on slaves. We respond by asking our active task sets for tasks in order of priority. We fill each node with tasks in a round-robin manner so that tasks are balanced across the cluster.

Specified by:

resourceOffers in interface org.apache.mesos.Scheduler

getResource

public double getResource(java.util.List<org.apache.mesos.Protos.Resource> res,
                 String name)

Helper function to pull out a resource from a Mesos Resources protobuf

createMesosTask

public org.apache.mesos.Protos.TaskInfo createMesosTask(TaskDescription task,
                                               String slaveId)

Turn a Spark TaskDescription into a Mesos task

isFinished

public boolean isFinished(org.apache.mesos.Protos.TaskState state)

Check whether a Mesos task state represents a finished task

statusUpdate

public void statusUpdate(org.apache.mesos.SchedulerDriver d,
                org.apache.mesos.Protos.TaskStatus status)

Specified by:

statusUpdate in interface org.apache.mesos.Scheduler

error

public void error(org.apache.mesos.SchedulerDriver d,
         String message)

Specified by:

error in interface org.apache.mesos.Scheduler

stop

public void stop()

Specified by:

stop in interface SchedulerBackend

reviveOffers

public void reviveOffers()

Specified by:

reviveOffers in interface SchedulerBackend

frameworkMessage

public void frameworkMessage(org.apache.mesos.SchedulerDriver d,
                    org.apache.mesos.Protos.ExecutorID e,
                    org.apache.mesos.Protos.SlaveID s,
                    byte[] b)

Specified by:

frameworkMessage in interface org.apache.mesos.Scheduler

slaveLost

public void slaveLost(org.apache.mesos.SchedulerDriver d,
             org.apache.mesos.Protos.SlaveID slaveId)

Specified by:

slaveLost in interface org.apache.mesos.Scheduler

executorLost

public void executorLost(org.apache.mesos.SchedulerDriver d,
                org.apache.mesos.Protos.ExecutorID executorId,
                org.apache.mesos.Protos.SlaveID slaveId,
                int status)

Specified by:

executorLost in interface org.apache.mesos.Scheduler

killTask

public void killTask(long taskId,
            String executorId,
            boolean interruptThread)

Specified by:

killTask in interface SchedulerBackend

defaultParallelism

public int defaultParallelism()

Specified by:

defaultParallelism in interface SchedulerBackend

applicationId

public String applicationId()

Description copied from interface: SchedulerBackend

Get an application ID associated with the job.

Specified by:

applicationId in interface SchedulerBackend

Returns:

An application ID