神经网络梯度下降算法（gradient descent）笔记

说明：此代码并非个人原创，是学习其他深度学习视频教程后总结所得。
总体思路：
1. 用手写数字识别作实例进行分析。
2. 具体的思路我不是非常清楚，就是用一个深度神经网络，选定n多参数，然后就可以在一定程度上模拟任何有规律的方程或者其他现象。
3. 求解的过程就是不断让真实值贴近预测的值，这时如果差别较大，可以改变参数的权重，还有偏向值。
4. 也就是cost(w, b) = |y - output|^2，此时的w, b赋值比较随机，所以就相当于在抛物线的两侧，若想下降到O点，也就是cost函数值最小，每次可以把x(也就是w, b)的值减去一个值（斜率的倍数，在左为负，在右为正），使值逼近O点,直接上图cost函数抛物线更新值
5. 此方法美其名曰梯度下降算法结合代码分析：
文章末尾有完整代码
1.初始化权重和偏向，使用numpy.random.randn(m, n)，具体意思我想你们应该懂吧  
self.weights = [np.random.randn(m, n) for m, n in zip(sizes[1:], sizes[:-1])] 
self.biases = [np.random.randn(k, 1) for k in sizes[1:]]2.每一轮epochs后打乱重排
random.shuffle(training_data)3.更新值
self.update_mini_batch(mini_batch, eta)4. backpropagation计算偏导值，也就是下图中减号后的偏导部分，不包括前面的伊塔参数，具体自己看吧
delta = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])
nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())5. 更新w和b值，文章中的代码
self.weights = [w - (eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)] 
self.biases = [b - (eta/len(mini_batch))*nb for b, nb in zip(self.biases, nabla_b)]6.其他部分比较简单，可以看英文注释，就是求偏导和sigmoid函数等，你们自己看吧。# author: dragon
# date: 2018-7-21
# -*- coding: utf-8 -*-import random
import numpy as npclass Network(object):def __init__(self, sizes):"""initialization parameter--weights and biases:param sizes:"""self.num_layers = len(sizes)self.sizes = sizesself.weights = [np.random.randn(m, n) for m, n in zip(sizes[1:], sizes[:-1])]self.biases = [np.random.randn(k, 1) for k in sizes[1:]]def feedforward(self, a):"""function value:param a: the input:return: sigmoid number"""for w, b in zip(self.weights, self.biases):a = sigmoid(np.dot(w, a) + b)return adef SGD(self, training_data, epochs, mini_batch_size, eta, test_data=None):"""realize stochastic gradient descent:param train_data::param epochs: the number of cycling:param mini_batch_size: each cycling sizes:param eta: learning rate:param test_data::return:None"""if test_data:n_test = len(test_data)n = len(training_data)for j in xrange(epochs):random.shuffle(training_data)mini_batches = [training_data[k: k+mini_batch_size] for k in xrange(0, n, mini_batch_size)]for mini_batch in mini_batches:self.update_mini_batch(mini_batch, eta)if test_data:#evaluate(test_data) is using to evaluate the accuracyprint "Epoch {0}:{1} / {2}".format(j, self.evaluate(test_data), n_test)else:print "Epoch {0} complete".format(j)def update_mini_batch(self, mini_batch, eta):"""uodate the weights and biases:param mini_batch: a lisr of tuple (x, y):param eta: learning rate:return: None"""nabla_w = [np.zeros(w.shape) for w in self.weights]nabla_b = [np.zeros(b.shape) for b in self.biases]for x, y in mini_batch:#backpropagation funtion is for the sum of derivatedelta_nabla_b, delta_nabla_w = self.backprop(x, y)nabla_w = [nw + dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]nabla_b = [nb + dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]self.weights = [w - (eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)]self.biases = [b - (eta/len(mini_batch))*nb for b, nb in zip(self.biases, nabla_b)]def backprop(self, x, y):"""return a tuple (nabla_b, nabla_w)"""nabla_b = [np.zeros(b.shape) for b in self.biases]nabla_w = [np.zeros(w.shape) for w in self.weights]# feedforwardactivation = xactivations = [x]  # list to store all the activations, layer by layerzs = []  # list to store all the z vectors, layer by layerfor b, w in zip(self.biases, self.weights):z = np.dot(w, activation) + bzs.append(z)activation = sigmoid(z)activations.append(activation)# backward passdelta = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])nabla_b[-1] = deltanabla_w[-1] = np.dot(delta, activations[-2].transpose())for l in xrange(2, self.num_layers):z = zs[-l]sp = sigmoid_prime(z)delta = np.dot(self.weights[-l+1].transpose(), delta) * spnabla_b[-l] = deltanabla_w[-l] = np.dot(delta, activations[-l-1].transpose())return (nabla_b, nabla_w)def evaluate(self,test_data):"""network's output is assumed to be the index of whicheverneuron in the final layer has the highest activation:param test_data::return: the number of correct"""test_result = [(np.argmax(self.feedforward(x)), y) for (x, y) in test_data]return sum(int(x == y) for (x, y) in test_result)def cost_derivative(self, output_activation, y):"""return the derivative"""return (output_activation - y)def sigmoid(z):"""  the sigmoid function. """return 1.0/(1.0+np.exp(-z))def sigmoid_prime(z):return sigmoid(z)*(1-sigmoid(z))