org.apache.spark.internal.io

Class SparkHadoopWriterUtils

Object

org.apache.spark.internal.io.SparkHadoopWriterUtils

public class SparkHadoopWriterUtils
extends Object

A helper object that provide common utils used during saving an RDD using a Hadoop OutputFormat (both from the old mapred API and the new mapreduce API)

Constructor Summary

Constructors

Constructor and Description
SparkHadoopWriterUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.mapred.JobID	createJobID(java.util.Date time, int id)
static String	createJobTrackerID(java.util.Date time)
static org.apache.hadoop.fs.Path	createPathFromString(String path, org.apache.hadoop.mapred.JobConf conf)
static scala.util.DynamicVariable<Object>	disableOutputSpecValidation() Allows for the spark.hadoop.validateOutputSpecs checks to be disabled on a case-by-case basis; see SPARK-4835 for more details.
static scala.Tuple2<org.apache.spark.executor.OutputMetrics,scala.Function0<Object>>	initHadoopOutputMetrics(TaskContext context)
static boolean	isOutputSpecValidationEnabled(SparkConf conf)
static void	maybeUpdateOutputMetrics(org.apache.spark.executor.OutputMetrics outputMetrics, scala.Function0<Object> callback, long recordsWritten)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkHadoopWriterUtils

public SparkHadoopWriterUtils()

Method Detail

createJobID

public static org.apache.hadoop.mapred.JobID createJobID(java.util.Date time,
                                                         int id)

createJobTrackerID

public static String createJobTrackerID(java.util.Date time)

createPathFromString

public static org.apache.hadoop.fs.Path createPathFromString(String path,
                                                             org.apache.hadoop.mapred.JobConf conf)

isOutputSpecValidationEnabled

public static boolean isOutputSpecValidationEnabled(SparkConf conf)

initHadoopOutputMetrics

public static scala.Tuple2<org.apache.spark.executor.OutputMetrics,scala.Function0<Object>> initHadoopOutputMetrics(TaskContext context)

maybeUpdateOutputMetrics

public static void maybeUpdateOutputMetrics(org.apache.spark.executor.OutputMetrics outputMetrics,
                                            scala.Function0<Object> callback,
                                            long recordsWritten)

disableOutputSpecValidation

public static scala.util.DynamicVariable<Object> disableOutputSpecValidation()

Allows for the spark.hadoop.validateOutputSpecs checks to be disabled on a case-by-case basis; see SPARK-4835 for more details.

Returns:

(undocumented)