Is it possible to calculate TF-IDF using KH Coder? If yes, then can you please tell me how?

[PiyushKyushu]

Dear Professor Higuchi,
First of all, thank you for this software, it is really wonderful. I am using it for more than a year.
For my research I want TF-IDF, I was wondering if it is possible to calculate this using Kh Coder.
There is a past question which talks about TF-DF not TF-IDF.
Can you kindly tell me the procedure to calculate TF-IDF?

Thanks once again.

[ko-ichi-h]

Dear PiyushKyushu,

Thank you for your post.

Do you need TF-IDF matrix? Something like document-word matrix that have TF-IDF in each cells? Or, do you need one value for each word?

Matrix

1. Go to [Project] [Export] [Document-Word Matrix] [CSV] in the menu to export TF matrix. Save it as "C:\khcoder3\tf.csv"
2. Start R by double clicking "Rgui.bat" in the "khcoder3" folder
3. Run the following command in R (paste it to console and hit enter key) and you get tf-idf.csv in the "khcoder3" folder

d <- read.csv("C:/khcoder3/tf.csv", fileEncoding="UTF-8-BOM",check.names=F)

# calculate tf
n_cut <- NULL
for (i in 1:12){
	if ( colnames(d)[i] == "length_w" ){
		n_cut <- i
		break
	}
}
doc_length_mtr <- d[,(n_cut-1):n_cut]
leng <- as.numeric(doc_length_mtr[,2])
leng[leng ==0] <- 1
d <- d / leng

# delete unnecessary part
n_cut <- n_cut * -1
d <- d[,-1:n_cut]

# prepare a function: binary termfrequency
lw_bintf <- function(m) {
    return( (m>0)*1 )
}

# prepare a function: inverse document frequency
# from Dumais (1992), Nakov (2001) uses log not log2
gw_idf <- function(m) {
    df = rowSums(lw_bintf(m), na.rm=TRUE)
    return ( ( log2(ncol(m)/df) + 1 ) )
}

# calculate tf-idf
d <- t(d)
d <- d[rowSums(d) > 0, ]
d <- d * gw_idf(d)
d <- t(d)

# save
fh <- file("C:/khcoder3/tf-idf.csv", "wb")
writeBin(as.raw(c(0xef, 0xbb, 0xbf)), fh)
close(fh)
fh <- file("C:/khcoder3/tf-idf.csv", "at", encoding="UTF-8")
write.csv(d, fh, quote=F, row.names=F)
close(fh)

One value for each word

Go to [Project] [Export] [Word Frequency List (Excel)] in the menu. Here you can select "TF" or "DF". You can get both TF and DF by this command, so you can calculate TF-IDF yourself using Excel.

[PiyushKyushu]

Dear Professor Higuchi,
Thank you for your reply.
I want one value for each word.
I got TF and DF using KH Coder but I don't know how to calculate IDF from DF.
If you don't mind can you tell me how to calculate IDF and then TF-IDF by using excel?

Thank you.

[ko-ichi-h]

=(LOG(N / DF ,2) + 1) * TF would do it, I think. N is total number of documents, that is shown in the main window of KH Coder. About N, DF, and TF, you need to enter actual numbers or cell address to calculate TF-IDF in Excel.

You should consult the literature to see if the above formula is correct.

[PiyushKyushu]

Dear Professor Higuchi,
Thank you for your reply.
One more question, when it KH Coder calculates term frequency it changes some terms such as data into datum, huawei into huaweus.
Further, it also shows two variations of russia - russium and russia.
Can you explain the reason behind this?

Thank you.

[PiyushKyushu]

Suppose the main window is showing H5 = 100, that means there are 100 documents in one excel file, so I want to confirm that when KH Coder count Term Frequency (TF), it gives the term frequency of one particular term in overall excel file (total frequency of a word in all 100 documents)?
If I am not wrong, to count TF-IDF, we need TF of each word in each document.
Can we use TF given by KH Coder in calculating TF-IDF?
I am confused, please clarify my doubt.

[ko-ichi-h]

One more question, when it KH Coder calculates term frequency it changes some terms such as data into datum, huawei into huaweus.
Further, it also shows two variations of russia - russium and russia.

KH Coder tries to count lemma of each word. To detect lemma, KH Coder uses Stanford POS Tagger in default settings. So, if Stanford POS Tagger recognizes "huawei"'s lemma as "huaweus", KH Coder counts "huaweus".

If you want to unite variations, see #96 .

when KH Coder count Term Frequency (TF), it gives the term frequency of one particular term in overall excel file (total frequency of a word in all 100 documents)?

That's correct.

If I am not wrong, to count TF-IDF, we need TF of each word in each document.

I am not sure. About the matrix I wrote above (TF-IDF for each word in each document), I know the formula. But about "one value for each word" style TF-IDF, I am not sure about the formula. If you find and show me the formula, I can tell you how to calculate it.

BTW, you can get TF of each word in each document using [Project] [Export] [Document-Word Matrix] [CSV] function of KH Coder.

Best.

[PiyushKyushu]

Thank you, Professor Higuchi. I got the document-word matrix.
Now I have TF, and with the formula you gave, I can get IDF from DF.
One last thing, in relation to TF-IDF. Though it is not technical it is more kind of methodological question.
I want top words (let's say 30 words) with the highest TF-IDF. What steps should I take to get that?
I require that because I want to compare top words with TF (most frequent words in a complete corpus) and top words with TF-IDF.
I am really sorry to bother you with these questions.

Thank you once again.

[ko-ichi-h]

Please confirm the formula by consulting literatures. I don't recommend that you just use the above formula without any confirmation.

I would use Excel to get top 30 words.

"Single Column" option is probably convenient in this case.

[PiyushKyushu]

Dear Professor Higuchi,
The formula for TF-IDF is:
TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document)

IDF(t) = log_e(Total number of documents / Number of documents with term t in it)

TF-IDF = TF(t) * IDF(t)

Can you guide me about the steps for calculating TF-IDF using KH Coder and Excel?
I am just a beginner in this field. This is the reason for these questions.
I hope you don't mind.
Thank you for your reply.

[ko-ichi-h]

I do not understand.

TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document)

By this definition, each document has a different value of TF(t). So, it is a definition of "Matrix" type TF-IDF I wrote above. It is not a definition of "One value for each word" type TF-IDF.

[PiyushKyushu]

Dear Professor Higuchi,
This is the source of the definition. http://www.tfidf.com/
I couldn't find the formula you mentioned.
Maybe because all are talking about the matrix I think?
I am confused sensei. What should I do?

Thank you for your reply.

[ko-ichi-h]

In the first place, why do you want to get TF-IDF?

[PiyushKyushu]

Dear Professor Higuchi,
My research includes content analysis and sentiment analysis of the data I collected (newspaper articles).
For the content analysis part, I am using KH Coder.
Using KH Coder I got the most frequent words in my corpus.
As I understand, TF-IDF gives the most important words not the most frequent words.
That is why I want top words in TF-IDF so that I can get the most important words in my corpus. Which I think would be good for overall content analysis.

Thank you.

[ko-ichi-h]

As I understand, TF-IDF gives the most important words not the most frequent words.

Where did you find this claim? Anyway, you need to find a literature containing this claim and the TF-IDF formula used for this purpose. I think that would be the first step.

[ko-ichi-h]

Seems abandoned. I close this issue for now. Please reopen if it's necessary.