导入
N-元语法简介
了解什么是n-gram，以及如何使用Python实现二元语法（bigrams）。
常见的N-元语法类型
| N-元语法类型 | 说明 | 示例 | 在文本预处理中的用途 |
|---|---|---|---|
| 一元语法（Unigrams，1-grams） | 序列中的单个词或字符 | “The” “dog” “is” “sleeping” | 作为简单任务的基础特征，或用于分析词频。 |
| 二元语法（Bigrams，2-grams） | 序列中相邻的两个词或字符组成的对 | “The dog” “dog is” “is sleeping” | 用于捕捉词语间的直接关系，在情感分析和语言模型等任务中非常有用。 |
| 三元语法（Trigrams，3-grams） | 序列中连续的三个词或字符组成的组 | “The dog is” “dog is sleeping” | 相比二元语法提供更多上下文信息，适用于语言建模和某些机器翻译模型。 |
| 四元语法（Quadgrams，4-grams） | 序列中连续的四个词或字符组成的组 | “The dog is sleeping” “dog is sleeping on” | 能捕捉更长的上下文模式，在某些机器翻译任务等场景中具有帮助作用。 |
1import pandas as pd 2from sklearn.feature_extraction.text import CountVectorizer 3from nltk.tokenize import word_tokenize 4import string 5 6df = pd.read_csv('reviews.csv') 7def remove_punctuation(text): 8 translator = str.maketrans('', '', string.punctuation) 9 text = text.translate(translator) 10 return text 11df['review_text'] = df['review_text'].apply(remove_punctuation) 12vectorizer = CountVectorizer(tokenizer=word_tokenize) 13X = vectorizer.fit_transform(df['review_text']) 14unigrams = vectorizer.get_feature_names() 15for unigram in unigrams: 16 print(f"'{unigram}'")
1import pandas as pd 2from sklearn.feature_extraction.text import CountVectorizer 3from nltk.tokenize import word_tokenize 4import string 5 6df = pd.read_csv('reviews.csv') 7def remove_punctuation(text): 8 translator = str.maketrans('', '', string.punctuation) 9 text = text.translate(translator) 10 return text 11df['review_text'] = df['review_text'].apply(remove_punctuation) 12vectorizer = CountVectorizer(tokenizer=word_tokenize, ngram_range=(2, 2)) 13X = vectorizer.fit_transform(df['review_text']) 14bigrams = vectorizer.get_feature_names() 15for bigram in bigrams: 16 print(f"'{bigram}'")
1import pandas as pd 2from sklearn.feature_extraction.text import CountVectorizer 3from nltk.tokenize import word_tokenize 4import string 5 6df = pd.read_csv('reviews.csv') 7def remove_punctuation(text): 8 translator = str.maketrans('', '', string.punctuation) 9 text = text.translate(translator) 10 return text 11df['review_text'] = df['review_text'].apply(remove_punctuation) 12vectorizer = CountVectorizer(tokenizer=word_tokenize, ngram_range=(3, 3)) 13X = vectorizer.fit_transform(df['review_text']) 14trigrams = vectorizer.get_feature_names() 15for trigram in trigrams: 16 print(f"'{trigram}'")
1import pandas as pd 2from sklearn.feature_extraction.text import CountVectorizer 3from nltk.tokenize import word_tokenize 4import string 5 6df = pd.read_csv('reviews.csv') 7def remove_punctuation(text): 8 translator = str.maketrans('', '', string.punctuation) 9 text = text.translate(translator) 10 return text 11df['review_text'] = df['review_text'].apply(remove_punctuation) 12n = 4 13vectorizer = CountVectorizer(tokenizer=word_tokenize, ngram_range=(n, n)) 14X = vectorizer.fit_transform(df['review_text']) 15ngrams = vectorizer.get_feature_names() 16for ngram in ngrams: 17 print(f"'{ngram}'")