YWW Tools

A package of my (Weiwei Yang's) various tools (most for NLP). Feel free to email me at [email protected] with any questions.

• Check Out
• Dependencies
• Use YWW Tools in Command Line
• LDA (Latent Dirichlet Allocation) in Command Line
  • RTM: Relational Topic Model
    • Lex-WSB-RTM: RTM with Lexical Weights and Weighted Stochastic Block Priors
    • Lex-WSB-Med-RTM: Lex-WSB-RTM with Hinge Loss
  • SLDA: Supervised LDA
    • BS-LDA: Binary SLDA
    • Lex-WSB-BS-LDA: BS-LDA with Lexcial Weights and Weighted Stochastic Block Priors
    • Lex-WSB-Med-LDA: Lex-WSB-BS-LDA with Hinge Loss
  • BP-LDA: LDA with Block Priors
  • ST-LDA: Single Topic LDA
  • WSB-TM: Weighted Stochastic Block Topic Model
• tLDA in Command Line
• Other Tools in Command Line
  • WSBM: Weighted Stochastic Block Model
  • SCC: Strongly Connected Components
  • Stoplist
  • Lemmatizer
  • POS Tagger
  • Stemmer
  • Tokenizer
  • Corpus Converter
  • Tree Builder
• Use YWW Tools Source Code
• LDA Code Examples
  • RTM
    • Lex-WSB-RTM
    • Lex-WSB-Med-RTM
  • SLDA
    • BS-LDA
    • Lex-WSB-BS-LDA
    • Lex-WSB-Med-LDA
  • BP-LDA
  • ST-LDA
  • WSB-TM
• tLDA Code Examples
• Other Code Examples
  • WSBM
  • SCC
  • Tree Builder
  • English Corpus Preprocessing
• Citation
• References

Check Out

git clone [email protected]:ywwbill/YWWTools.git

Dependencies

• Java 8.
• Maven 3.3.
• Files in lib/.

Build

In the package root directory, run command

mvn package

You will get YWWTools-1.jar and deps.jar in target/.

For convenience, I assume you

• copy YWWTools-1.jar and deps.jar to package root directory
• rename YWWTools-1.jar to YWWTools.jar

Use YWW Tools in Command Line

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool <tool-name> --arg-1 <arg-1-value> --arg-2 <arg-2-value> ... --arg-n <arg-n-value>

• Windows users please replace YWWTools.jar:deps.jar with YWWTools.jar;deps.jar.
• Supported <tool-name> (case unsensitive) include
  • LDA: Latent Dirichlet allocation. Include a variety of extensions.
  • TLDA: Tree LDA.
  • WSBM: Weighted stochastic block model. Find blocks in a network.
  • SCC: Strongly connected components.
  • Stoplist: Remove stop words. Support English only, but can support other languages given dictionary.
  • Lemmatizer: Lemmatize POS-tagged corpus. Support English only, but can support other languages given dictionary.
  • POS-Tagger: Tag words' POS. Support English only, but can support other languages given trained models.
  • Stemmer: Stem words. Support English only.
  • Tokenizer: Tokenize corpus. Support English only, but can support other languages given trained models.
  • Corpus-Converter: Convert word corpus into indexed corpus (for LDA) and vice versa.
  • Tree Builder: Build tree priors from word associations.
• You can always use --help to see help information of

  • supported tool names if you don't specify a tool name

     java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --help
    

  • a specific tool if you specify it (take LDA as an example)

     java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool LDA --help
    

• In following command examples, arguments in {} (e.g. {cmd-1|cmd-2|cmd-3}) denote that one and only one of them should be declared.

LDA (Latent Dirichlet Allocation) in Command Line

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model lda --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file>

• Implementation of (Blei et al., 2003).
• Required arguments

  • <vocab-file>: Vocabulary file. Each line contains a unique word.

  • <corpus-file>: Corpus file in which documents are represented by word indexes and frequencies. Each line contains a document in the following format

     <doc-len> <word-type-1>:<frequency-1> <word-type-2>:<frequency-2> ... <word-type-n>:<frequency-n>
    

    <doc-len> is the total number of tokens in this document. <word-type-i> denotes the i-th word in <vocab-file>, starting from 0. Words with zero frequency can be omitted.

  • <model-file>: Trained model file in JSON format. Read and written by program.

• Optional arguments
  • --model <model-name>: The topic model you want to use (default: LDA). Supported <model-name> (case unsensitive) are
    • LDA: Vanilla LDA
    • RTM: Relational topic model.
      • Lex-WSB-RTM: RTM with WSB-computed block priors and lexical weights.
      • Lex-WSB-Med-RTM: Lex-WSB-RTM with hinge loss.
    • SLDA: Supervised LDA. Support multi-class classification.
      • BS-LDA: Binary SLDA.
      • Lex-WSB-BS-LDA: BS-LDA with WSB-computed block priors and lexical weights.
      • Lex-WSB-Med-LDA: Lex-WSB-BS-LDA with hinge loss.
    • BP-LDA: LDA with block priors. Blocks are pre-computed.
    • ST-LDA: Single topic LDA. Each document can only be assigned to one topic.
    • WSB-TM: LDA with block priors. Blocks are computed by WSBM.
  • --test: Use the model for test (default: false).
  • --no-verbose: Stop printing log to console.
  • --alpha <alpha-value>: Parameter of Dirichlet prior of document distribution over topics (default: 1.0). Must be a positive real number.
  • --beta <beta-value>: Parameter of Dirichlet prior of topic distribution over words (default: 0.1). Must be a positive real number.
  • --topics <num-topics>: Number of topics (default: 10). Must be a positive integer.
  • --iters <num-iters>: Number of iterations (default: 100). Must be a positive integer.
  • --update: Update alpha while sampling (default: false).
  • --update-int <update-interval>: Interval of updating alpha (default: 10). Must be a positive integer.
  • --theta <theta-file>: File for document distribution over topics. Each line contains a document's topic distribution. Topic weights are separated by space.
  • --output-topic <topic-file>: File for showing topics.
  • --topic-count <topic-count-file>: File for document-topic counts.
  • --top-word <num-top-word>: Number of words to give when showing topics (default: 10). Must be a positive integer.

RTM: Relational Topic Model

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model rtm --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --rtm-train-graph <rtm-train-graph-file>

• Implementation of (Chang and Blei, 2010).
• Jointly models topics and document links.
• Extends LDA.
• Semi-optional arguments
  • --rtm-train-graph <rtm-train-graph-file> [optional in test]: Link file for RTM to train. Each line contains an edge in the format node-1 \t node-2 \t weight. Node number starts from 0. weight must be a non-negative integer. weight is either 0 or 1 and is optional. Its default value is 1 if not specified.
  • --rtm-test-graph <rtm-test-graph-file> [optional in training]: Link file for RTM to evaluate. Can be the same with RTM train graph. Format is the same as <rtm-train-graph-file>.
• Optional arguments
  • --nu <nu-value>: Variance of normal priors for weight vectors/matrices in RTM and its extensions (default: 1.0). Must be a positive real number.
  • --plr-int <compute-PLR-interval>: Interval of computing predictive link rank (default: 20). Must be a positive integer.
  • --neg: Sample negative links (default: false).
  • --neg-ratio <neg-ratio>: The ratio of number of negative links to number of positive links (default 1.0). Must be a positive real number.
  • --pred <pred-file>: Predicted document link probability matrix file.
  • --reg <reg-file>: Doc-doc regression value file.
  • --directed: Set all edges directed (default: false).

Lex-WSB-RTM: RTM with Lexical Weights and Weighted Stochastic Block Priors

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools yang.weiwei.Tools --tool lda --model lex-wsb-rtm --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --rtm-train-graph <rtm-train-graph-file>

• Extends RTM.
• Optional arguments
  • --wsbm-graph <wsbm-graph-file>: Link file for WSBM to find blocks. See WSBM for details.
  • --alpha-prime <alpha-prime-value>: Parameter of Dirichlet prior of block distribution over topics (default: 1.0). Must be a positive real number.
  • -a <a-value>: Parameter of Gamma prior for block link rates (default: 1.0). Must be a positive real number.
  • -b <b-value>: Parameter of Gamma prior for block link rates (default: 1.0). Must be a positive real number.
  • -g <gamma-value>: Parameter of Dirichlet prior for block distribution (default: 1.0). Must be a positive real number.
  • --blocks <num-blocks>: Number of blocks (default: 10). Must be a positive integer.
  • --output-wsbm <wsbm-output-file>: File for WSBM-identified blocks. See WSBM for details.
  • --block-feature: Include block features in link prediction (default: false).

Lex-WSB-Med-RTM: Lex-WSB-RTM with Hinge Loss

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model lex-wsb-med-rtm --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --rtm-train-graph <rtm-train-graph-file>

• Implementation of (Yang et al., 2016)
• See (Zhu et al., 2012) and (Zhu et al., 2014) for hinge loss.
• Extends Lex-WSB-RTM.
• Link weight is either 1 or -1.
• Optional arguments
  • -c <c-value>: Regularization parameter in hinge loss (default: 1.0). Must be a positive real number.

SLDA: Supervised LDA

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model slda --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --label <label-file>

• Implementation of (McAuliffe and Blei, 2008).
• Jointly models topics and document labels. Support multi-class classification.
• Extends LDA.
• Semi-optional arguments
  • --label <label-file> [optional in test]: Label file. Each line contains corresponding document's numeric label. If a document label is not available, leave the corresponding line empty.
• Optional arguments
  • --sigma <sigma-value>: Variance for the Gaussian generation of response variable in SLDA (default: 1.0). Must be a positive real number.
  • --nu <nu-value>: Variance of normal priors for weight vectors in SLDA and its extensions (default: 1.0). Must be a positive real number.
  • --pred <pred-file>: Predicted label file.
  • --reg <reg-file>: Regression value file.

BS-LDA: Binary SLDA

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model bs-lda --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --label <label-file>

• For binary classification only.
• Extends SLDA.
• Label is either 1 or 0.

Lex-WSB-BS-LDA: BS-LDA with Lexcial Weights and Weighted Stochastic Block Priors

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model lex-wsb-bs-lda --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --label <label-file>

• Extends BS-LDA.
• Optional arguments
  • --wsbm-graph <wsbm-graph-file>: Link file for WSBM to find blocks. See WSBM for details.
  • --alpha-prime <alpha-prime-value>: Parameter of Dirichlet prior of block distribution over topics (default: 1.0). Must be a positive real number.
  • -a <a-value>: Parameter of Gamma prior for block link rates (default: 1.0). Must be a positive real number.
  • -b <b-value>: Parameter of Gamma prior for block link rates (default: 1.0). Must be a positive real number.
  • -g <gamma-value>: Parameter of Dirichlet prior for block distribution (default: 1.0). Must be a positive real number.
  • --blocks <num-blocks>: Number of blocks (default: 10). Must be a positive integer.
  • --directed: Set all edges directed (default: false).
  • --output-wsbm <wsbm-output-file>: File for WSBM-identified blocks. See WSBM for details.

Lex-WSB-Med-LDA: Lex-WSB-BS-LDA with Hinge Loss

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model lex-wsb-med-lda --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --label <label-file>

• See (Zhu et al., 2012) and (Zhu et al., 2014) for hinge loss.
• Extends Lex-WSB-BS-LDA.
• Label is either 1 or -1.
• Optional arguments
  • -c <c-value>: Regularization parameter in hinge loss (default: 1.0). Must be a positive real number.

BP-LDA: LDA with Block Priors

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model bp-lda --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --block-graph <block-graph-file>

• Use priors from pre-computed blocks.
• Extends LDA.
• Semi-optional arguments
  • --block-graph <block-graph-file> [optional in test]: Pre-computed block file. Each line contains a block and consists of one or more documents denoted by document numbers. Document numbers are separated by space.
• Optional arguments
  • --alpha-prime <alpha-prime-value>: Parameter of Dirichlet prior of block distribution over topics (default: 1.0). Must be a positive real number.

ST-LDA: Single Topic LDA

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model st-lda --vocab <vocab-file> --corpus <corpus-file> --short-corpus <short-corpus-file> --trained-model <model-file>

• Implementation of (Hong et al., 2016).
• Each document can only be assigned to one topic.
• Extends LDA.
• Semi-optional arguments
  • --short-corpus <short-corpus-file> [at least one of --short-corpus and --corpus should be specified]: Short corpus file.
• Optional arguments
  • --short-theta <short-theta-file>: Short documents' background topic distribution file.
  • --short-topic-assign <short-topic-assign-file>: Short documents' topic assignment file.

WSB-TM: Weighted Stochastic Block Topic Model

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lda --model wsb-tm --vocab <vocab-file> --corpus <corpus-file> --trained-model <model-file> --wsbm-graph <wsbm-graph-file>

• Use priors from WSBM-computed blocks.
• Extends LDA.
• Semi-optional arguments
  • --wsbm-graph <wsbm-graph-file> [optional in test]: Link file for WSBM to find blocks. See WSBM for details.
• Optional arguments
  • --alpha-prime <alpha-prime-value>: Parameter of Dirichlet prior of block distribution over topics (default: 1.0). Must be a positive real number.
  • -a <a-value>: Parameter of Gamma prior for block link rates (default: 1.0). Must be a positive real number.
  • -b <b-value>: Parameter of Gamma prior for block link rates (default: 1.0). Must be a positive real number.
  • -g <gamma-value>: Parameter of Dirichlet prior for block distribution (default: 1.0). Must be a positive real number.
  • --blocks <num-blocks>: Number of blocks (default: 10). Must be a positive integer.
  • --directed: Set all edges directed (default: false).
  • --output-wsbm <wsbm-output-file>: File for WSBM-identified blocks. See WSBM for details.

tLDA in Command Line

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool tlda --vocab <vocab-file> --tree <tree-prior-file> --corpus <corpus-file> --trained-model <model-file>

• Implementation of (Boyd-Graber et al., 2007).
• Required arguments

  • <vocab-file>: Vocabulary file. Each line contains a unique word.

  • <tree-prior-file>: Tree prior file. Generated by Tree Builder

  • <corpus-file>: Corpus file in which documents are represented by word indexes and frequencies. Each line contains a document in the following format

     <doc-len> <word-type-1>:<frequency-1> <word-type-2>:<frequency-2> ... <word-type-n>:<frequency-n>
    

    <doc-len> is the total number of tokens in this document. <word-type-i> denotes the i-th word in <vocab-file>, starting from 0. Words with zero frequency can be omitted.

  • <model-file>: Trained model file. Read and written by program.

• Optional arguments
  • --test: Use the model for test (default: false).
  • --no-verbose: Stop printing log to console.
  • --alpha <alpha-value>: Parameter of Dirichlet prior of document distribution over topics (default: 0.01). Must be a positive real number.
  • --beta <beta-value>: Parameter of Dirichlet prior of topic distribution over words (default: 0.01). Must be a positive real number.
  • --topics <num-topics>: Number of topics (default: 10). Must be a positive integer.
  • --iters <num-iters>: Number of iterations (default: 100). Must be a positive integer.
  • --update: Update alpha while sampling (default: false).
  • --update-int <update-interval>: Interval of updating alpha (default: 10). Must be a positive integer.
  • --theta <theta-file>: File for document distribution over topics. Each line contains a document's topic distribution. Topic weights are separated by space.
  • --output-topic <topic-file>: File for showing topics.
  • --topic-count <topic-count-file>: File for document-topic counts.
  • --top-word <num-top-word>: Number of words to give when showing topics (default: 10). Must be a positive integer.

Other Tools in Command Line

WSBM: Weighted Stochastic Block Model

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool wsbm --nodes <num-nodes> --blocks <num-blocks> --graph <graph-file> --output <output-file>

• Implementation of (Aicher et al., 2014).
• Find latent blocks in a network, such that nodes in the same block are densely connected and nodes in different blocks are sparsely connected.
• Required arguments
  • <num-nodes>: Number of nodes in the graph. Must be a positive integer.
  • <num-blocks>: Number of blocks. Must be a positive integer.
  • <graph-file>: Graph file. Each line contains an edge in the format node-1 \t node-2 \t weight. Node number starts from 0. weight must be a non-negative integer. weight is optional. Its default value is 1 if not specified.
  • <output-file>: Result file. The i-th line contains the block assignment of i-th node.
• Optional arguments
  • --directed: Set the edges as directed (default: undirected).
  • -a <a-value>: Parameter for edge rates' Gamma prior (default: 1.0). Must be a positive real number.
  • -b <b-value>: Parameter for edge rates' Gamma prior (default: 1.0). Must be a positive real number.
  • -g <gamma-value>: Parameter for block distribution's Dirichlet prior (default 1.0). Must be a positive real number.
  • --iters <num-iters>: Number of iterations (default: 100). Must be a positive integer.
  • --no-verbose: Stop printing log to console.

SCC: Strongly Connected Components

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool scc --nodes <num-nodes> --graph <graph-file> --output <output-file>

• New implementation.
• Find strongly connected components in an undirected graph. In each component, every node is reachable from any other nodes in the same component.
• Arguments
  • <num-nodes>: Number of nodes in the graph. Must be a positive integer.
  • <graph-file>: Graph file. Each line contains an edge in the format node-1 \t node-2. Node number starts from 0.
  • <output-file>: Result file. Each line contains a strongly connected component and consists of one or more nodes denoted by node numbers. Node numbers are separated by space.

Stoplist

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool stoplist --corpus <corpus-file> --output <output-file>

• New implementation.
• Only supports English, but can support other languages if dictionary is provided.
• Required arguments
  • <corpus-file>: Corpus file with stop words. Each line contains a document. Words are separated by space.
  • <output-file>: Corpus file without stop words. Each line contains a document. Words are separated by space.
• Optional arguments
  • --dict <dict-file>: Dictionary file name. Each line contains a stop word.

Lemmatizer

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool lemmatizer --corpus <corpus-file> --output <output-file>

• A re-packaging of opennlp.tools.lemmatizer.SimpleLemmatizer.
• Only supports English, but can support other languages if dictionary is provided.
• Required arguments
  • <corpus-file>: Unlemmatized corpus file. Each line contains a unlemmatized, tokenized, and POS-tagged document.
  • <output-file>: Lemmatized corpus file. Each line contains a lemmatized document. Words are separated by space.
• Optional arguments
  • --dict <dict-file>: Dictionary file name. Each line contains a rule in the format unlemmatized-word \t POS \t lemmatized-word.

POS Tagger

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool pos-tagger --corpus <corpus-file> --output <output-file>

• A re-packaing of opennlp.tools.postag.POSTaggerME (https://opennlp.apache.org/documentation/1.6.0/manual/opennlp.html#tools.postagger)
• Only supports English, but can support other languages if model is provided.
• Required arguments
  • <corpus-file>: Untagged corpus file. Each line contains a tokenized untagged document.
  • <output-file>: Tagged corpus file. Each line contains a tagged document. Each word is annotated as word_POS.
• Optional arguments
  • --model <model-file>: Model file name.

Stemmer

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool stemmer --corpus <corpus-file> --output <output-file>

• A re-packaging of PorterStemmer (http://tartarus.org/~martin/PorterStemmer/index.html)
• Only supports English.
• Arguments
  • <corpus-file>: Unstemmed corpus file. Each line contains an unstemmed document. Words are separated by space.
  • <output-file>: Stemmed corpus file. Each line contains a stemmed document. Words are separated by space.

Tokenizer

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool tokenizer --corpus <corpus-file> --output <output-file>

• A re-packaging of opennlp.tools.tokenize.TokenizerME (https://opennlp.apache.org/documentation/1.6.0/manual/opennlp.html#tools.tokenizer)
• Only supports English, but can support other languages if model is provided.
• Required arguments
  • <corpus-file>: Untokenized corpus file. Each line contains a untokenized document.
  • <output-file>: Tokenized corpus file. Each line contains a tokenized document.
• Optional arguments
  • --model <model-file>: Model file name.

Corpus Converter

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool corpus-converter {--get-vocab|--to-index|--to-word} --word-corpus <word-corpus-file> --index-corpus <index-corpus-file> --vocab <vocab-file>

• New implementation
• Arguments

  • --get-vocab, --to-index, --to-word: One of three should be selected.

    • --get-vocab: Collect vocabulary from <word-corpus-file> and write them in <vocab-file>.
    • --to-index: Convert a word corpus file <word-corpus-file> into an indexed corpus file <index-corpus-file> and write the vocabulary in <vocab-file>.
    • --to-word: Convert an indexed corpus file <index-corpus-file> into a word corpus file <word-corpus-file> given vocabulary file <vocab-file>.

  • <word-corpus-file>: Corpus file in which documents are represented by words. Each line contains a document. Words are separated by space.

  • <index-corpus-file>: Corpus file in which documents are represented by word indexes and frequencies. Not required when using --get-vocab. Each line contains a document in the following format

     <doc-len> <word-type-1>:<frequency-1> <word-type-2>:<frequency-2> ... <word-type-n>:<frequency-n>
    

    <doc-len> is the total number of tokens in this document. <word-type-i> denotes the i-th word in <vocab-file>, starting from 0. Words with zero frequency can be omitted.

  • <vocab-file>: Vocabulary file. Each line contains a unique word.

Tree Builder

java -cp YWWTools.jar:deps.jar yang.weiwei.Tools --tool tree-builder --vocab <vocab-file> --score <score-file> --tree <tree-file>

• Implementation of (Yang et al., 2017)
• Arguments
  • <vocab-file>: Vocabulary file. Each line contains a unique word.
  • <score-file>: Word association file. Assume there are V words in <vocab-file>. There are V lines in the <score-file>. Each line corresponds to a word in the vocabulary and contains V float numbers which denote the word's association scores with all other words.
  • <tree-file>: The tree prior file.
• Optional Arguments
  • --type <tree-type>: Tree prior type. 1 for two-level tree; 2 for hierarchical agglomerative clustering (HAC) tree; 3 for HAC tree with leaf duplication (default 1).
  • --child <num-child>: Number of child nodes per internal node for a two-level tree (default 10).
  • --thresh <threshold>: The confidence threshold for HAC (default 0.0).

Use YWWTools Source Code

To integrate my code into your project, please copy lib/ to your project root directory and

• include YWWTools.jar and deps.jar in your project build path, OR
• add the dependencies (in pom.xml) to your own dependency list

Here are examples for running some algorithms in this package. For more information, please look at JavaDoc in doc/.

LDA Code Examples

• Classes: yang.weiwei.lda.LDA and yang.weiwei.lda.LDAParam.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name"); //initialize a parameter object and set parameters as needed
    LDA ldaTrain = new LDA(param); // initialize an LDA object
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.initialize();
    ldaTrain.sample(100); // set number of iterations as needed
    ldaTrain.writeModel("model_file_name"); // optional, see test code example
    ldaTrain.writeDocTopicDist("theta_file_name"); // optional, write document-topic distribution to file
    ldaTrain.writeResult("topic_file_name", 10); // optional, write top 10 words of each topic to file
    ldaTrain.writeDocTopicCounts("topic_count_file_name") // optional, write document-topic counts to file
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    LDA ldaTest = new LDA(ldaTrain, param); // initialize with pre-trained LDA object
    // LDA ldaTest = new LDA("model_file_name", param); // or initialize with an LDA model in a file
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.initialize();
    ldaTest.sample(100); // set number of iterations as needed
    ldaTest.writeDocTopicDist("theta_file_name"); // optional, write document-topic distribution to file
    ldaTest.writeDocTopicCounts("topic_count_file_name"); // optional, write document-topic counts to file
  

RTM

• Class: yang.weiwei.lda.rtm.RTM.

• Extends LDA.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    RTM ldaTrain = new RTM(param);
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.readGraph("train_graph_file_name", RTM.TRAIN_GRAPH); // read train graph
    ldaTrain.readGraph("test_graph_file_name", RTM.TEST_GRAPH); // read test graph
    ldaTrain.initialize();
    ldaTrain.sample(100); 
    ldaTrain.writePred("pred_file_name"); // optional, write predicted document link probabilities to file
    ldaTrain.writeRegValues("reg_value_file_name"); // optional, write doc-doc regression values to file
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    RTM ldaTest = new RTM(ldaTrain, param);
    // RTM ldaTest = new RTM("model_file_name", param); 
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.readGraph("train_graph_file_name", RTM.TRAIN_GRAPH); // optional
    ldaTest.readGraph("test_graph_file_name", RTM.TEST_GRAPH);
    ldaTest.initialize();
    ldaTest.sample(100); 
    ldaTest.writePred("pred_file_name"); // optional, write predicted document link probabilities to file
    ldaTest.writeRegValues("reg_value_file_name"); // optional, write doc-doc regression values to file
  

Lex-WSB-RTM

• Class: yang.weiwei.lda.rtm.lex_wsb_rtm.LexWSBRTM.

• Extends RTM.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    LexWSBRTM ldaTrain = new LexWSBRTM(param);
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.readGraph("train_graph_file_name", RTM.TRAIN_GRAPH); 
    ldaTrain.readGraph("test_graph_file_name", RTM.TEST_GRAPH); 
    ldaTrain.readBlockGraph("wsbm_graph_file_name"); // optional, read graph for WSBM
    ldaTrain.initialize();
    ldaTrain.sample(100); 
    ldaTrain.writeBlocks("block_file_name"); // optional, write WSBM results to file
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    LexWSBRTM ldaTest = new LexWSBRTM(ldaTrain, param);
    // LexWSBRTM ldaTest = new LexWSBRTM("model_file_name", param); 
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.readGraph("train_graph_file_name", RTM.TRAIN_GRAPH); // optional
    ldaTest.readGraph("test_graph_file_name", RTM.TEST_GRAPH);
    ldaTest.readBlockGraph("wsbm_graph_file_name"); // optional
    ldaTest.initialize();
    ldaTest.sample(100); 
    ldaTest.writeBlocks("block_file_name"); // optional
  

Lex-WSB-Med-RTM

• Class: yang.weiwei.lda.rtm.lex_wsb_med_rtm.LexWSBMedRTM.
• Extends Lex-WSB-RTM.
• Code examples are the same with Lex-WSB-RTM.

SLDA

• Class: yang.weiwei.lda.slda.SLDA.

• Extends LDA.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    SLDA ldaTrain = new SLDA(param);
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.readLabels("label_file_name"); // read label file
    ldaTrain.initialize();
    ldaTrain.sample(100);
    ldaTrain.writePredLabels("pred_label_file_name"); // optional, write predicted labels
    ldaTrain.writeRegValues("reg_value_file_name"); // optioanl, write regression values
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    SLDA ldaTest = new SLDA(ldaTrain, param);
    // SLDA ldaTest = new SLDA("model_file_name", param);
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.readLabels("label_file_name"); // optional
    ldaTest.initialize();
    ldaTest.sample(100);
    ldaTest.writePredLabels("pred_label_file_name"); // optional
    ldaTest.writeRegValues("reg_value_file_name"); // optional
  

BS-LDA

• Class: yang.weiwei.lda.slda.bs_lda.BSLDA
• Extends SLDA.
• Code examples are the same with SLDA.

Lex-WSB-BS-LDA

• Class: yang.weiwei.lda.slda.lex_wsb_bs_lda.LexWSBBSLDA.

• Extends BS-LDA.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    LexWSBBSLDA ldaTrain = new LexWSBBSLDA(param);
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.readLabels("label_file_name");
    ldaTrain.readBlockGraph("wsbm_graph_file_name"); // optional, read graph for WSBM
    ldaTrain.initialize();
    ldaTrain.sample(100);
    ldaTrain.writeBlocks("block_file_name"); // optional, write WSBM results to file
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    LexWSBBSLDA ldaTest = new LexWSBBSLDA(ldaTrain, param);
    // LexWSBBSLDA ldaTest = new LexWSBBSLDA("model_file_name", param);
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.readLabels("label_file_name"); // optional
    ldaTest.readBlockGraph("wsbm_graph_file_name"); // optional
    ldaTest.initialize();
    ldaTest.sample(100);
    ldaTest.writePredLabels("pred_label_file_name"); // optional
    ldaTest.writeBlocks("block_file_name"); // optional
  

Lex-WSB-Med-LDA

• Class: yang.weiwei.lda.slda.lex_wsb_med_lda.LexWSBMedLDA.
• Extends Lex-WSB-BS-LDA.
• Code examples are the same with Lex-WSB-BS-LDA.

BP-LDA

• Class: yang.weiwei.lda.bp_lda.BPLDA

• Extends LDA.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    BPLDA ldaTrain = new BPLDA(param); 
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.readBlocks("block_file_name"); // read block file
    ldaTrain.initialize();
    ldaTrain.sample(100);
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    BPLDA ldaTest = new BPLDA(ldaTrain, param);
    // BPLDA ldaTest = new BPLDA("model_file_name", param);
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.readBlocks("block_file_name"); // optional
    ldaTest.initialize();
    ldaTest.sample(100); 
  

ST-LDA

• Class: yang.weiwei.lda.st_lda.STLDA

• Extends LDA.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    STLDA ldaTrain = new STLDA(param);
    ldaTrain.readCorpus("long_corpus_file_name");
    ldaTrain.readShortCorpus("short_corpus_file_name");
    ldaTrain.initialize();
    ldaTrain.sample(100);
    ldaTrain.writeShortDocTopicDist("short_theta_file_name"); // optional, write short documents' topic distribution to file
    ldaTrain.writeShortDocTopicAssign("short_topic_assign_file_name"); // optional, write short documents' topic assignments to file
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    STLDA ldaTest = new STLDA(ldaTrain, param);
    // STLDA ldaTest = new STLDA("model_file_name", param);
    ldaTest.readCorpus("long_corpus_file_name");
    ldaTest.readShortCorpus("short_corpus_file_name");
    ldaTest.initialize();
    ldaTest.sample(100);
    ldaTest.writeShortDocTopicDist("short_theta_file_name"); // optional
    ldaTest.writeShortDocTopicAssign("short_topic_assign_file_name"); // optional
  

WSB-TM

• Class: yang.weiwei.lda.wsb_tm.WSBTM

• Extends LDA.

• Training code example

    LDAParam param = new LDAParam("vocab_file_name");
    WSBTM ldaTrain = new WSBTM(param); 
    ldaTrain.readCorpus("corpus_file_name");
    ldaTrain.readGraph("wsbm_graph_file_name"); // read graph file
    ldaTrain.initialize();
    ldaTrain.sample(100);
  

• Test code example

    LDAParam param = new LDAParam("vocab_file_name");
    WSBTM ldaTest = new WSBTM(ldaTrain, param);
    // WSBTM ldaTest = new WSBTM("model_file_name", param);
    ldaTest.readCorpus("corpus_file_name");
    ldaTest.readGraph("wsbm_graph_file_name"); // optional
    ldaTest.initialize();
    ldaTest.sample(100); 
  

tLDA Code Examples

• Classes: yang.weiwei.tlda.TLDA and yang.weiwei.tlda.TLDAParam.

• Training code example

    TLDAParam param = new LDAParam("vocab_file_name", "tree_prior_file_name"); //initialize a parameter object and set parameters as needed
    TLDA tldaTrain = new TLDA(param); // initialize a tLDA object
    tldaTrain.readCorpus("corpus_file_name");
    tldaTrain.initialize();
    tldaTrain.sample(100); // set number of iterations as needed
    tldaTrain.writeModel("model_file_name"); // optional, see test code example
    tldaTrain.writeDocTopicDist("theta_file_name"); // optional, write document-topic distribution to file
    tldaTrain.writeWordResult("topic_file_name", 10); // optional, write top 10 words of each topic to file
    tldaTrain.writeDocTopicCounts("topic_count_file_name") // optional, write document-topic counts to file
  

• Test code example

    TLDAParam param = new TLDAParam("vocab_file_name", "tree_prior_file_name");
    TLDA tldaTest = new TLDA(tldaTrain, param); // initialize with pre-trained tLDA object
    // TLDA tldaTest = new TLDA("model_file_name", param); // or initialize with a TLDA model in a file
    tldaTest.readCorpus("corpus_file_name");
    tldaTest.initialize();
    tldaTest.sample(100); // set number of iterations as needed
    tldaTest.writeDocTopicDist("theta_file_name"); // optional, write document-topic distribution to file
    tldaTest.writeDocTopicCounts("topic_count_file_name"); // optional, write document-topic counts to file
  

Other Code Examples

WSBM

• Classes: yang.weiwei.wsbm.WSBM and yang.weiwei.wsbm.WSBMParam.

• Code example

    WSBMParam param = new WSBMParam(); // initialize a parameter object and set parameters as needed
    WSBM wsbm = new WSBM(param); // initialize a WSBM object with parameters
    wsbm.readGraph("graph_file_name");
    wsbm.init();
    wsbm.sample(100); // set number of iterations as needed
    wsbm.printResults();
  

SCC

• Class: yang.weiwei.scc.SCC.

• Code example

    SCC scc = new SCC(10); // initialize with number of nodes
    scc.readGraph("graph_file_name");
    scc.cluster();
    scc.writeCluster("result_file_name");
  

Tree Builder

• Class: yang.weiwei.tlda.TreeBuilder.

• Code example

    TreeBuilder tb = new TreeBuilder();
    tb.build2LevelTree("score_file_name", "vocab_file_name", "tree_file_name", num_Child); // Build a two-level tree
    tb.hac("score_file_name", "vocab_file_name", "tree_file_name", threshold); // Build a tree with hierarchical agglomerative clustering (HAC)
    tb.hacWithLeafDup("score_file_name", "vocab_file_name", "tree_file_name", threshold); // Build a tree with HAC and leaf duplication
  

English Corpus Preprocessing

• Basically there are two ways to preprocess an English corpus for topic models as follows.
  • tokenization -> stop words removal -> stemming
  • tokenization -> POS tagging -> lemmatization -> stop words removal
• The first way is quick but with low word readability. The second one takes more time but produce better readability.
• Finally you may want to remove low (document-)frequency words, in order to accelerate topic modeling without hurting the performance.

Citation

• If you use Tree Builder, please cite

    @InProceedings{Yang:Boyd-Graber:Resnik-2017,
    	Title = {Adapting Topic Models using Lexical Associations with Tree Priors},
    	Booktitle = {Empirical Methods in Natural Language Processing},
    	Author = {Weiwei Yang and Jordan Boyd-Graber and Philip Resnik},
    	Year = {2017},
    	Location = {Copenhagen, Denmark},
    }
  

• If you use Lex-WSB-RTM (aka LBS-RTM), Lex-WSB-Med-RTM (aka LBH-RTM), Lex-WSB-BS-LDA, and/or Lex-WSB-Med-LDA, please cite

    @InProceedings{Yang:Boyd-Graber:Resnik-2016,
    	Title = {A Discriminative Topic Model using Document Network Structure},
    	Booktitle = {Association for Computational Linguistics},
    	Author = {Weiwei Yang and Jordan Boyd-Graber and Philip Resnik},
    	Year = {2016},
    	Location = {Berlin, Germany},
    }
  

• If you use ST-LDA, please cite

    @InProceedings{Hong:Yang:Resnik:Frias-Martinez-2016,
    	Title = {Uncovering Topic Dynamics of Social Media and News: The Case of Ferguson},
    	Booktitle = {International Conference on Social Informatics},
    	Author = {Lingzi Hong and Weiwei Yang and Philip Resnik and Vanessa Frias-Martinez},
    	Year = {2016},
    	Location = {Bellevue, WA, USA}
    }
  

References

LDA: Latent Dirichlet Allocation

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research.

SLDA: Supervised LDA

Jon D. McAuliffe and David M. Blei. 2008. Supervised topic models. In Proceedings of Advances in Neural Information Processing Systems.

Med-LDA: Max-margin LDA

Jun Zhu, Amr Ahmed, and Eric P. Xing. 2012. MedLDA: Maximum margin supervised topic models. Journal of Machine Learning Research.

Jun Zhu, Ning Chen, Hugh Perkins, and Bo Zhang. 2014. Gibbs max-margin topic models with data augmentation. Journal of Machine Learning Research.

RTM: Relational Topic Model

Jonathan Chang and David M. Blei. 2010. Hierarchical relational models for document networks. The Annals of Applied Statistics.

Lex-WSB-Med-RTM: RTM with WSB-computed Block Priors, Lexical Weights, and Hinge Loss

Weiwei Yang, Jordan Boyd-Graber, and Philip Resnik. 2016. A discriminative topic model using document network structure. In Proceedings of Association for Computational Linguistics.

ST-LDA: Single Topic LDA

Lingzi Hong, Weiwei Yang, Philip Resnik, and Vanessa Frias-Martinez. 2016. Uncovering topic dynamics of social media and news: The case of Ferguson. In Proceedings of International Conference on Social Informatics.

WSBM: Weighted Stochastic Block Model

Christopher Aicher, Abigail Z. Jacobs, and Aaron Clauset. 2014. Learning latent block structure in weighted networks. Journal of Complex Networks.

tLDA: Tree LDA

Jordan Boyd-Graber, David M. Blei, and Xiaojin Zhu. 2007. A topic model for word sense disambiguation. Empirical Methods in Natural Language Processing.

Tree Builder

Weiwei Yang, Jordan Boyd-Graber, and Philip Resnik. 2017. Adapting topic models using lexical associations with tree priors. Empirical Methods in Natural Language Processing.

Back to Top