org.apache.spark.sql.hive

Class HiveContext

Object

org.apache.spark.sql.SQLContext

org.apache.spark.sql.hive.HiveContext

All Implemented Interfaces:

java.io.Serializable, Logging

public class HiveContext
extends SQLContext

An instance of the Spark SQL execution engine that integrates with data stored in Hive. Configuration for Hive is read from hive-site.xml on the classpath.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.spark.sql.SQLContext

SQLContext.implicits

Constructor Summary

Constructors

Constructor and Description
HiveContext(SparkContext sc)

Method Summary

Methods

Modifier and Type	Method and Description
void	analyze(String tableName) Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.
void	refreshTable(String tableName) Invalidate and refresh all the cached the metadata of the given table.
void	setConf(String key, String value) Set the given Spark SQL configuration property.
DataFrame	sql(String sqlText) Executes a SQL query using Spark, returning the result as a DataFrame.

Methods inherited from class org.apache.spark.sql.SQLContext

applySchema, applySchema, applySchema, applySchema, baseRelationToDataFrame, cacheTable, clearCache, createDataFrame, createDataFrame, createDataFrame, createDataFrame, createDataFrame, createDataFrame, createDataFrame, createExternalTable, createExternalTable, createExternalTable, createExternalTable, createExternalTable, createExternalTable, dropTempTable, emptyDataFrame, experimental, getAllConfs, getConf, getConf, implicits, isCached, jdbc, jdbc, jdbc, jsonFile, jsonFile, jsonFile, jsonRDD, jsonRDD, jsonRDD, jsonRDD, jsonRDD, jsonRDD, load, load, load, load, load, load, parquetFile, parquetFile, registerDataFrameAsTable, setConf, sparkContext, table, tableNames, tableNames, tables, tables, udf, uncacheTable

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

HiveContext

public HiveContext(SparkContext sc)

Method Detail

sql

public DataFrame sql(String sqlText)

Description copied from class: SQLContext

Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

Overrides:

sql in class SQLContext

refreshTable

public void refreshTable(String tableName)

Invalidate and refresh all the cached the metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.

analyze

public void analyze(String tableName)

Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.

Right now, it only supports Hive tables and it only updates the size of a Hive table in the Hive metastore.

setConf

public void setConf(String key,
           String value)

Description copied from class: SQLContext

Set the given Spark SQL configuration property.

Overrides:

setConf in class SQLContext