NLP——Task3 基于机器学习的文本分类

机器学习模型

机器学习是对能通过经验自动改进的计算机算法的研究。机器学习通过历史数据训练出模型对应于人类对经验进行归纳的过程，机器学习利用模型对新数据进行预测对应于人类利用总结的规律对新问题进行预测的过程。

1、Count Vectors + RidgeClassifier

import pandas as pdfrom sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_scoretrain_df = pd.read_csv('C://train_set.csv', sep='\t', nrows=15000)vectorizer = CountVectorizer(max_features=3000)
train_test = vectorizer.fit_transform(train_df['text'])clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, average='macro'))

结果得：

0.7416952793751392

2、TF-IDF + RidgeClassifier

import pandas as pdfrom sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_scoretrain_df = pd.read_csv('C://train_set.csv', sep='\t', nrows=15000)tfidf = TfidfVectorizer(ngram_range=(1,3), max_features=3000)
train_test = tfidf.fit_transform(train_df['text'])clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, average='macro'))

结果得：

0.8721598830546126