public class LDA extends Estimator<LDAModel> implements LDAParams, DefaultParamsWritable
Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data
Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
Input data (featuresCol):
LDA is given a collection of documents as input data, via the featuresCol parameter.
Each document is specified as a Vector of length vocabSize, where each entry is the
count for the corresponding term (word) in the document. Feature transformers such as
Tokenizer and CountVectorizer
can be useful for converting text to word count vectors.
| Modifier and Type | Method and Description |
|---|---|
LDA |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
LDAModel |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
static LDA |
load(String path) |
static MLReader<LDA> |
read() |
LDA |
setCheckpointInterval(int value) |
LDA |
setDocConcentration(double value) |
LDA |
setDocConcentration(double[] value) |
LDA |
setFeaturesCol(String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDA |
setK(int value) |
LDA |
setKeepLastCheckpoint(boolean value) |
LDA |
setLearningDecay(double value) |
LDA |
setLearningOffset(double value) |
LDA |
setMaxIter(int value) |
LDA |
setOptimizeDocConcentration(boolean value) |
LDA |
setOptimizer(String value) |
LDA |
setSeed(long value) |
LDA |
setSubsamplingRate(double value) |
LDA |
setTopicConcentration(double value) |
LDA |
setTopicDistributionCol(String value) |
StructType |
transformSchema(StructType schema)
:: DeveloperApi ::
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitdocConcentration, getDocConcentration, getK, getKeepLastCheckpoint, getLearningDecay, getLearningOffset, getOldDocConcentration, getOldOptimizer, getOldTopicConcentration, getOptimizeDocConcentration, getOptimizer, getSubsamplingRate, getTopicConcentration, getTopicDistributionCol, k, keepLastCheckpoint, learningDecay, learningOffset, optimizeDocConcentration, optimizer, subsamplingRate, supportedOptimizers, topicConcentration, topicDistributionCol, validateAndTransformSchemafeaturesCol, getFeaturesColgetMaxIter, maxItercheckpointInterval, getCheckpointIntervalclear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringwritesaveinitializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarningpublic static LDA load(String path)
public String uid()
Identifiableuid in interface Identifiablepublic LDA setFeaturesCol(String value)
Vector representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).
value - (undocumented)public LDA setMaxIter(int value)
public LDA setSeed(long value)
public LDA setCheckpointInterval(int value)
public LDA setK(int value)
public LDA setDocConcentration(double[] value)
public LDA setDocConcentration(double value)
public LDA setTopicConcentration(double value)
public LDA setOptimizer(String value)
public LDA setTopicDistributionCol(String value)
public LDA setLearningOffset(double value)
public LDA setLearningDecay(double value)
public LDA setSubsamplingRate(double value)
public LDA setOptimizeDocConcentration(boolean value)
public LDA setKeepLastCheckpoint(boolean value)
public LDA copy(ParamMap extra)
ParamsdefaultCopy().public LDAModel fit(Dataset<?> dataset)
Estimatorpublic StructType transformSchema(StructType schema)
PipelineStageCheck transform validity and derive the output schema from the input schema.
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)