Package org.apache.spark.scheduler

Interface ExternalClusterManager

public interface ExternalClusterManager
A cluster manager interface to plugin external scheduler.

  • Method Details

    • canCreate

      boolean canCreate(String masterURL)
      Check if this cluster manager instance can create scheduler components for a certain master URL.
      Parameters:
      masterURL - the master URL
      Returns:
      True if the cluster manager can create scheduler backend/

    • createSchedulerBackend

      SchedulerBackend createSchedulerBackend(SparkContext sc, String masterURL, TaskScheduler scheduler)
      Create a scheduler backend for the given SparkContext and scheduler. This is called after task scheduler is created using ExternalClusterManager.createTaskScheduler().
      Parameters:
      sc - SparkContext
      masterURL - the master URL
      scheduler - TaskScheduler that will be used with the scheduler backend.
      Returns:
      SchedulerBackend that works with a TaskScheduler

    • createTaskScheduler

      TaskScheduler createTaskScheduler(SparkContext sc, String masterURL)
      Create a task scheduler instance for the given SparkContext
      Parameters:
      sc - SparkContext
      masterURL - the master URL
      Returns:
      TaskScheduler that will be responsible for task handling

    • initialize

      void initialize(TaskScheduler scheduler, SchedulerBackend backend)
      Initialize task scheduler and backend scheduler. This is called after the scheduler components are created
      Parameters:
      scheduler - TaskScheduler that will be responsible for task handling
      backend - SchedulerBackend that works with a TaskScheduler