seq2seq+attention=评论摘要

Summarizing Text with Amazon Reviews
数据集：Amazon 500000评论

本节内容:

• 数据预处理
• 构建Seq2Seq模型
• 训练网络
• 测试效果

seq2seq教程: https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq
https://www.jianshu.com/p/c0c5f1bdbb88

浏览数据集

将少量的含有缺失值的记录和其他属性dropout

1、数据预处理

• 全部转换成小写
• 连词转换
• 去停用词（只在描述中去掉）
• 去除特殊字符https?:\/\/.*[\r\n]*\<a href[_"\-;%()|+&=*%.,!?:#$@\[\]/]

re.sub(r'https?:\/\/.*[\r\n]*\<a href[_"\-;%()|+&=*%.,!?:#$@\[\]/]`', ' ', text)

• 连词转换：
  “can’t’ve”: “cannot have”,
  “'cause”: “because”,
  “could’ve”: “could have”,

# 统计不重复的词的个数
def count_words(count_dict, text):'''Count the number of occurrences of each word in a set of text'''for sentence in text:for word in sentence.split():if word not in count_dict:count_dict[word] = 1else:count_dict[word] += 1# Find the number of times each word was used and the size of the vocabulary
word_counts = {
    }count_words(word_counts, clean_summaries)
print("Size of Vocabulary:", len(word_counts))
count_words(word_counts, clean_texts)
print("Size of Vocabulary:", len(word_counts))

OUT:
Size of Vocabulary: 33809
Size of Vocabulary: 132884

我们的任务通过seq2seq完成encode-decod的操作，需要词向量作为输入，这里的词向量也可以用别人训练好的词向量喂入模型。github上有训练好的词向量

格式：[0]为单词 ,[1：]为向量
/c/en/absolute_value -0.0847 -0.1316 -0.0800 -0.0708 -0.2514 -0.1687 -…

现将单词和其向量转换为字典的格式

embeddings_index = {
    }
with open('numberbatch-en-17.04b.txt', encoding='utf-8') as f:for line in f:values = line.split(' ')word = values[0]embedding = np.asarray(values[1:], dtype='float32')embeddings_index[word] = embedding

词库中的词不一样全都在词向量文件中，所以需要统计词库不在词向量文件中的单词，如果是词频大于20，且词向量文件中没有，就直接建立一个相同维度的向量训练即可。

统计miss单词的占比：

Number of words missing from CN: 3866
Percent of words that are missing from vocabulary: 2.91%

现将词频大于20的构建数字和单词的映射，方便计算机理解。
将标识符也加入词汇中，开始、PAD、unknow和结束符号

codes = ["<UNK>","<PAD>","<EOS>","<GO>"]   

int_to_vocab：

vocab_to_int：

做好词语和数字的映射之后，再将每一个语句中的单词转换为数字形式：有些词如词频小于20、还没有过滤掉的则替换为’unknown’

def convert_to_ints(text, word_count, unk_count, eos=False):'''Convert words in text to an integer.If word is not in vocab_to_int, use UNK's integer. ’unknown‘Total the number of words and UNKs.Add EOS token to the end of texts'''ints = []for sentence in text:sentence_ints = []for word in sentence.split():word_count += 1if word in vocab_to_int:sentence_ints.append(vocab_to_int[word])else:sentence_ints.append(vocab_to_int["<UNK>"])unk_count += 1if eos:sentence_ints.append(vocab_to_int["<EOS>"])ints.append(sentence_ints)return ints, word_count, unk_count

得到数字表示的句子：

为了更能清晰的观察数据，将每个句子的长度转换DataFrame，再用describe函数查看具体情况

查看%90、95、99分位的数值

现在再对句子进行一次过滤，

• 1、对summaries和text过滤：
  一句话中unk太多，设定阈值过滤一些句子
• 2、对summaries和text的长度排序，只取%90分位以下的长度值即可

sorted_summaries = []
sorted_texts = []
max_text_length = 84
max_summary_length = 13
min_length = 2
unk_text_limit = 1
unk_summary_limit = 0for length in range(min(lengths_texts.counts), max_text_length): for count, words in enumerate(int_summaries):if (len(int_summaries[count]) >= min_length andlen(int_summaries[count]) <= max_summary_length andlen(int_texts[count]) >= min_length andunk_counter(int_summaries[count]) <= unk_summary_limit andunk_counter(int_texts[count]) <= unk_text_limit andlength == len(int_texts[count])):sorted_summaries.append(int_summaries[count])

到这里，就完成了对数据的筛选，处理部分，接下来构造Seq2Seq模型

2、构造Seq2Seq模型

Bidirectional RNNs(双向网络)的改进之处便是，假设当前的输出(第t步的输出)不仅仅与前面的序列有关，并且还与后面的序列有关。

例如：预测一个语句中缺失的词语那么就需要根据上下文来进行预测。Bidirectional RNNs是一个相对较简单的RNNs，是由两个RNNs上下叠加在一起组成的。输出由这两个RNNs的隐藏层的状态决定的

• 输入xt乘以权值W+记忆乘以权重V+b
  

（1）数据输入

将数据切分，在每个句子前加‘’标识符

def process_encoding_input(target_data, vocab_to_int, batch_size):'''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)return dec_input

strded_slicc(起始，结束，步长)，举例：

（2）encoding_layer

动态双向RNN，输出enc_output, enc_state

def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):'''Create the encoding layer'''for layer in range(num_layers):with tf.variable_scope('encoder_{}'.format(layer)):cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, input_keep_prob = keep_prob)cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, input_keep_prob = keep_prob)enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw, rnn_inputs,sequence_length,# 一个batchSize的大小dtype=tf.float32)# Join outputs since we are using a bidirectional RNN enc_output = tf.concat(enc_output,2)return enc_output, enc_state

(3)训练，辅助层

1、训练：training_decoding_layer：
tf.contrib.seq2seq.    TrainingHelperBasicDecoderdynamic_decode2、测试：inference_decoding_layer
tf.contrib.seq2seq.GreedyEmbeddingHelperBasicDecoderdynamic_decode

(4)decoding_layer

LSTMCell
BahdanauAttention
AttentionWrapperwith tf.variable_scope("decode"):training_logits = training_decoding_layer(dec_embed_input, summary_length, dec_cell, initial_state,output_layer,vocab_size, max_summary_length)with tf.variable_scope("decode", reuse=True):inference_logits = inference_decoding_layer(embeddings,  vocab_to_int['<GO>'], vocab_to_int['<EOS>'],dec_cell, initial_state, output_layer,max_summary_length,batch_size)

(5)seq2seq_model

将上面建立的操作结合

(6)Graph built

get_batches 得到batch
pad_sentence_batch 将长度不够的句子用’PAD‘补齐
tf.contrib.seq2seq.sequence_loss
tf.train.AdamOptimizer