org.apache.spark

Class ShuffleDependency<K,V,C>

Object

org.apache.spark.Dependency<scala.Product2<K,V>>

org.apache.spark.ShuffleDependency<K,V,C>

All Implemented Interfaces:

java.io.Serializable

public class ShuffleDependency<K,V,C>
extends Dependency<scala.Product2<K,V>>

:: DeveloperApi :: Represents a dependency on the output of a shuffle stage. Note that in the case of shuffle, the RDD is transient since we don't need it on the executor side.

param: _rdd the parent RDD param: partitioner partitioner used to partition the shuffle output param: serializer Serializer to use. If not set explicitly then the default serializer, as specified by spark.serializer config option, will be used. param: keyOrdering key ordering for RDD's shuffles param: aggregator map/reduce-side aggregator for RDD's shuffle param: mapSideCombine whether to perform partial aggregation (also known as map-side combine) param: shuffleWriterProcessor the processor to control the write behavior in ShuffleMapTask

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description

ShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.ClassTag<C> evidence$3)

Method Summary

Modifier and Type	Method and Description
scala.Option<Aggregator<K,V,C>>	aggregator()
scala.Option<scala.math.Ordering<K>>	keyOrdering()
boolean	mapSideCombine()
Partitioner	partitioner()
RDD<scala.Product2<K,V>>	rdd()
Serializer	serializer()
org.apache.spark.shuffle.ShuffleHandle	shuffleHandle()
int	shuffleId()
org.apache.spark.shuffle.ShuffleWriteProcessor	shuffleWriterProcessor()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ShuffleDependency

public ShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd,
                         Partitioner partitioner,
                         Serializer serializer,
                         scala.Option<scala.math.Ordering<K>> keyOrdering,
                         scala.Option<Aggregator<K,V,C>> aggregator,
                         boolean mapSideCombine,
                         org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor,
                         scala.reflect.ClassTag<K> evidence$1,
                         scala.reflect.ClassTag<V> evidence$2,
                         scala.reflect.ClassTag<C> evidence$3)

Method Detail

partitioner

public Partitioner partitioner()

serializer

public Serializer serializer()

keyOrdering

public scala.Option<scala.math.Ordering<K>> keyOrdering()

aggregator

public scala.Option<Aggregator<K,V,C>> aggregator()

mapSideCombine

public boolean mapSideCombine()

shuffleWriterProcessor

public org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor()

rdd

public RDD<scala.Product2<K,V>> rdd()

Specified by:

rdd in class Dependency<scala.Product2<K,V>>

shuffleId

public int shuffleId()

shuffleHandle

public org.apache.spark.shuffle.ShuffleHandle shuffleHandle()