TF-IDF Analysis

TF-IDF (term frequency-inverse document frequency) is a term weighting scheme that scores each word by how often it appears in a document, discounted by how many documents in the corpus contain it.

TF-IDF(단어빈도-역문서빈도)는 각 단어가 한 문서에 나타나는 빈도를 그 단어가 포함된 문서 수로 할인하여 점수를 매기는 단어 가중치 방식입니다.

How do I find the words that are most distinctive for each document or group of documents?

각 문서나 문서 집단에서 가장 특징적인 단어는 어떻게 찾나요?

TF-IDF weights each term by its frequency in a document and down-weights terms that appear in many documents, so distinctive words receive high scores while common words receive low ones. It fits keyword identification, document comparison, and as a feature representation for text classification or clustering.

When to use TF-IDF Analysis

• Identifying characteristic words in each document or group of texts
• Turning a corpus into numeric features for classification or clustering
• Measuring document similarity with cosine similarity

When to use something else

• Meaning and context matter more than word counts: use word embeddings or BERT-based models
• Discovering latent themes across documents: use topic modeling (LDA)

Assumptions and requirements

• Corpus of multiple documents
• Tokenization suited to the language, such as morphological analysis for Korean
• Preprocessing such as lowercasing and stopword removal
• Bag-of-words representation that ignores word order

What the analysis reports

• Document-term matrix of TF-IDF weights
• Top-weighted terms per document or group
• Bar charts or word clouds of key terms
• Cosine similarity between documents

How to report it

TF-IDF weighting identified [term 1], [term 2], and [term 3] as the most distinctive terms in [group] documents.

Related methods

• Topic Modeling (LDA)
• TextRank Summarization
• Word Embeddings (Word2Vec)

Run TF-IDF Analysis on Quorant