• nlp: nature language processing, 目的是让计算机理解人类语言
• syntax: knowledge of the structural relationships betweenwords
• semantics: knowledge of meaning
• pragmetics: knowledge of the relationship of meaning to the goals and intentions of the speaker
• NLTK: natural language toolkit

import nltk
nltk.download()

Pre-processing tasks

• common taks in text processing
  • data cleaning: 去除不需要的标记/符号
  • tokenization: 将文本分割为
  • stemming: 词干提取(如computational -> compute)
  • lemmatization: 词形还原(如run, ran -> run)
  • stop words removal: 去除停用词(如a,an,the…)
  • POS Tagging: 标注词性(如noun,verb…)
• tokenization: 可以分为三大类
  • word tokenization
  • character tokeniaztion
  • subword tokenization: 最主流
    • byte pair encoding: 统计语料中出现最多的字符对，将他们合并为一个新的token, 直到到达预设的词表大小
• stop words: 有些高频词 are not informative
• stemming: porter’s stemming用于处理英语的常用stemming algorithm

from nltk.stem import PorterStemmer
pst = Porterstemmer()

• Unser-define Function(UDF): 包含function and output datatype
• lemmatization: 将differenct inflected forms of word聚成group就叫lemma
  • 如am, are, is -> be

from nltk.corpus input wordnet
from nltk.stem inport wordNetLemmatizer

• POS tagging
• syntactic annotation
  • poenn Treeband of english
• name entity recognization(NER): 识别机构/公司

from nltk import ne_chunk
(...)

• NLP Process
  • text processing
  • feature engineering and text representation
  • learning model (…)
• n-gram encoding
• text features: TFIDF用于quantify the word featuress(例如word importance)
• vector space model
  • 假设t个distinct terms in a document collection,
    • dimension: t=|vocabulary|
    • each term i , in a document
• document collection: document的collection可以表示为
• term frequency: 一个词在一个文档中出现的次数，出现越多的terms越重要，权重更大
• document frequency: 整个documen collection中有多少文档出现了这个词，出现越多的terms are less informative相较于出现较少的terms
• IDF: 一个词在越少的文档中出现，其区分度就越大
• TF-IDF: 即最终的权重W=TF*IDF

---

链接