Overview  Package   Class  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD

org.apache.spark.streaming.flume
Class FlumeUtils

Object
  extended by org.apache.spark.streaming.flume.FlumeUtils

public class FlumeUtils
extends Object

Constructor Summary
FlumeUtils()
           
 
Method Summary
static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc, java.net.InetSocketAddress[] addresses, StorageLevel storageLevel)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc, java.net.InetSocketAddress[] addresses, StorageLevel storageLevel, int maxBatchSize, int parallelism)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc, String hostname, int port)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc, String hostname, int port, StorageLevel storageLevel)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc, scala.collection.Seq<java.net.InetSocketAddress> addresses, StorageLevel storageLevel)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc, scala.collection.Seq<java.net.InetSocketAddress> addresses, StorageLevel storageLevel, int maxBatchSize, int parallelism)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc, String hostname, int port, StorageLevel storageLevel)
          Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc, String hostname, int port)
          Creates a input stream from a Flume source.
static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc, String hostname, int port, StorageLevel storageLevel)
          Creates a input stream from a Flume source.
static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc, String hostname, int port, StorageLevel storageLevel, boolean enableDecompression)
          Creates a input stream from a Flume source.
static ReceiverInputDStream<SparkFlumeEvent> createStream(StreamingContext ssc, String hostname, int port, StorageLevel storageLevel)
          Create a input stream from a Flume source.
static ReceiverInputDStream<SparkFlumeEvent> createStream(StreamingContext ssc, String hostname, int port, StorageLevel storageLevel, boolean enableDecompression)
          Create a input stream from a Flume source.
 
Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

FlumeUtils

public FlumeUtils()
Method Detail

createStream

public static ReceiverInputDStream<SparkFlumeEvent> createStream(StreamingContext ssc,
                                                                 String hostname,
                                                                 int port,
                                                                 StorageLevel storageLevel)
Create a input stream from a Flume source.

Parameters:
ssc - StreamingContext object
hostname - Hostname of the slave machine to which the flume data will be sent
port - Port of the slave machine to which the flume data will be sent
storageLevel - Storage level to use for storing the received objects
Returns:
(undocumented)

createStream

public static ReceiverInputDStream<SparkFlumeEvent> createStream(StreamingContext ssc,
                                                                 String hostname,
                                                                 int port,
                                                                 StorageLevel storageLevel,
                                                                 boolean enableDecompression)
Create a input stream from a Flume source.

Parameters:
ssc - StreamingContext object
hostname - Hostname of the slave machine to which the flume data will be sent
port - Port of the slave machine to which the flume data will be sent
storageLevel - Storage level to use for storing the received objects
enableDecompression - should netty server decompress input stream
Returns:
(undocumented)

createStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc,
                                                                     String hostname,
                                                                     int port)
Creates a input stream from a Flume source. Storage level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.

Parameters:
hostname - Hostname of the slave machine to which the flume data will be sent
port - Port of the slave machine to which the flume data will be sent
jssc - (undocumented)
Returns:
(undocumented)

createStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc,
                                                                     String hostname,
                                                                     int port,
                                                                     StorageLevel storageLevel)
Creates a input stream from a Flume source.

Parameters:
hostname - Hostname of the slave machine to which the flume data will be sent
port - Port of the slave machine to which the flume data will be sent
storageLevel - Storage level to use for storing the received objects
jssc - (undocumented)
Returns:
(undocumented)

createStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc,
                                                                     String hostname,
                                                                     int port,
                                                                     StorageLevel storageLevel,
                                                                     boolean enableDecompression)
Creates a input stream from a Flume source.

Parameters:
hostname - Hostname of the slave machine to which the flume data will be sent
port - Port of the slave machine to which the flume data will be sent
storageLevel - Storage level to use for storing the received objects
enableDecompression - should netty server decompress input stream
jssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc,
                                                                        String hostname,
                                                                        int port,
                                                                        StorageLevel storageLevel)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:
hostname - Address of the host on which the Spark Sink is running
port - Port of the host at which the Spark Sink is listening
storageLevel - Storage level to use for storing the received objects
ssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc,
                                                                        scala.collection.Seq<java.net.InetSocketAddress> addresses,
                                                                        StorageLevel storageLevel)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:
addresses - List of InetSocketAddresses representing the hosts to connect to.
storageLevel - Storage level to use for storing the received objects
ssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc,
                                                                        scala.collection.Seq<java.net.InetSocketAddress> addresses,
                                                                        StorageLevel storageLevel,
                                                                        int maxBatchSize,
                                                                        int parallelism)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available.

Parameters:
addresses - List of InetSocketAddresses representing the hosts to connect to.
maxBatchSize - Maximum number of events to be pulled from the Spark sink in a single RPC call
parallelism - Number of concurrent requests this stream should send to the sink. Note that having a higher number of requests concurrently being pulled will result in this stream using more threads
storageLevel - Storage level to use for storing the received objects
ssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                                            String hostname,
                                                                            int port)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:
hostname - Hostname of the host on which the Spark Sink is running
port - Port of the host at which the Spark Sink is listening
jssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                                            String hostname,
                                                                            int port,
                                                                            StorageLevel storageLevel)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:
hostname - Hostname of the host on which the Spark Sink is running
port - Port of the host at which the Spark Sink is listening
storageLevel - Storage level to use for storing the received objects
jssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                                            java.net.InetSocketAddress[] addresses,
                                                                            StorageLevel storageLevel)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:
addresses - List of InetSocketAddresses on which the Spark Sink is running.
storageLevel - Storage level to use for storing the received objects
jssc - (undocumented)
Returns:
(undocumented)

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                                            java.net.InetSocketAddress[] addresses,
                                                                            StorageLevel storageLevel,
                                                                            int maxBatchSize,
                                                                            int parallelism)
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available.

Parameters:
addresses - List of InetSocketAddresses on which the Spark Sink is running
maxBatchSize - The maximum number of events to be pulled from the Spark sink in a single RPC call
parallelism - Number of concurrent requests this stream should send to the sink. Note that having a higher number of requests concurrently being pulled will result in this stream using more threads
storageLevel - Storage level to use for storing the received objects
jssc - (undocumented)
Returns:
(undocumented)
Overview  Package   Class  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD