统计学习（九）：神经网络

神经网络

感知器

$$f(x)=\sum _{j=0}^p{w}_j{x}_j=\sum _{j=1}^p{w}_j{x}_j+w_0={\mathbf{w}}^T\mathbf{x}$$

用感知器分类

$$f(x)=s(\sum _{j=1}^p{w}_j{x}_j+w_0)=s({\mathbf{w}}^T\mathbf{x})$$

$s$ 是一个阈值函数。分类：$f(x)>0$ 或 $f(x)<0$。

如果不是简单的决定正负（+、-），那么可以构造：
$$\begin{gathered} f(x)=\sigma (\sum _{j=1}^p{w}_j{x}_j+w_0)=\sigma ({\mathbf{w}}^T\mathbf{x}) \\ \sigma (u)=\frac{1}{1+\exp (?u)} \end{gathered}$$

一维的总结

回归：
$$f(x)=\sum _{j=1}^p{w}_j{x}_j+w_0$$
分类：
$$f(x)=1/(1+\exp [?(\sum _{j=1}^p{w}_j{x}_j+w_0)])$$

多类分类

选择 C k : f k ( x ) = max ? l ∈ { 1 , ? , K } f l ( x ) C_k:f_k(x)=\max_{l\in\{1,\cdots,K\}}f_l(x) Ck?:fk?(x)=maxl∈{ 1,?,K}?fl?(x)。

为得到概率，使用 softmax：
$$\begin{gathered} \sigma (u)=\frac{1}{1+e^{?u}}=\frac{e^u}{1+e^u} \\ {f}_k(x)=\frac{\exp (o_k)}{{\sum }_{l=1}^K\exp (o_l)},o_k={\mathbf{w}}^{kT}\mathbf{x} \end{gathered}$$
如果一个类的输出足够大于其他类，则其softmax将接近1（否则为0）。

训练感知器

随机梯度下降：从随机权重开始，在每个点，调整权重以最小化误差。

通用的Update rule：
$$\Delta w_j=?\eta \frac{?Error(f(x^i),y^i)}{?w_j}$$
在每个训练实例后，对于每个权重：
$$\begin{gathered} w_i^{(t+1)}=w_j^{(t)}+\Delta w_i^{(t)} \\ {w}_j\leftarrow w_j+\Delta w_j \end{gathered}$$

回归

$$Error(f(x^i),y^i)=\frac{1}{2}(y^i?f(x^i){)}^2=\frac{1}{2}(y^i?{\mathbf{w}}^T{\mathbf{x}}^i{)}^2$$

对于回归的Update rule为：
$$\Delta w_j=\eta (y^i?f(x^i))x_j^i$$
Sigmoid output:
$$f(x^i)=\sigma ({\mathbf{w}}^T{\mathbf{x}}^i),\sigma (u)=\frac{1}{1+e^{?u}},{\sigma }^{\prime }(u)=u^{\prime }\sigma (u)(1?\sigma (u))$$
Cross-entropy error:
$$Error(f(x^i),y^i)=?y^i\log f(x^i)?(1?y^i)\log (1?f(x^i))$$

K类

K>2的 softmax输出：
$$f_k(x^i)=\frac{\exp ({\mathbf{w}}^{kT}{\mathbf{x}}^i)}{{\sum }_{l=1}^K\exp ({\mathbf{w}}^{lT}{\mathbf{x}}^i)}$$
Cross-entropy error:
$$Error(f(x^i),y^i)=?\sum _{k=1}^K{y}_k^i\log f_k(x^i)$$
Update rule
$$\Delta w_j=\eta (y^i?f(x^i))x_j^i$$

布尔函数

AND

$$f(x)=s(w_0+w_1{x}_1+w_2{x}_2)$$
理想的分界线：

XOR

$$\begin{gathered} f(x)=s(w_0+w_1{x}_1+w_2{x}_2) \\ {w}_0\le 0 \\ {w}_0+w_2>0 \\ {w}_0+w_1>0 \\ {w}_0+w_1+w_2\le 0 \end{gathered}$$

多层感知器

隐藏层：

隐藏单元的输出：
$$z_h=\frac{1}{1+e^{?{\mathbf{w}}_h^T\mathbf{x}}}$$
神经网络输出：
$$\begin{gathered} f(x)={\mathbf{v}}^T\mathbf{z}=v_0+\sum _{h=1}^H\frac{v_h}{1+e^{?{\mathbf{w}}_h^T\mathbf{x}}} \\ \frac{?E}{?w_{hj}}=\frac{?E}{?f(x)}\frac{?f(x)}{?z_h}\frac{?z_h}{?w_{hj}} \\ \frac{?f(x)}{?z_h}\to v_h,\frac{?z_h}{?w_{hj}}\to z_h^i(1?z_h^i)x_j^i \end{gathered}$$

回归

$$\begin{gathered} Error(f(x^i),y^i)=\frac{1}{2}(y^i?f(x^i){)}^2 \\ f(x)=v_0+\sum _{h=1}^H{v}_h{z}_h,z_h=\sigma ({\mathbf{w}}_h^T\mathbf{x}) \\ \Delta v_h={\eta }_1(y^i?f(x^i))z_h^i \\ \Delta w_{hj}=?\eta \frac{?E^i}{?w_{hj}}=?\eta \frac{?E^i}{?f(x^i)}\frac{?f(x^i)}{?z_h^i}\frac{?z_h^i}{?w_{hj}} \\ =?\eta ?(y^i?f(x^i))v_h{z}_h^i(1?z_h^i)x_j^i \end{gathered}$$

初始化所有的 $v_h,w_{hj}$ 到$(?0.01,0.01)$的随机数，重复，直到收敛：
$$\begin{gathered} fori=1,?,n \\ forh=1,?,H \\ {z}_h^i=\sigma ({\mathbf{w}}_h^T{\mathbf{x}}^i) \\ f(x^i)={\mathbf{v}}^T{\mathbf{z}}^i \\ forh=1,?,H \\ \Delta v_h={\eta }_1(y^i?f(x^i))z_h^i \\ forh=1,?,H \\ forj=1,?,p \\ \Delta w_{hj}=\eta (y_k^i?f_k(x^i))v_h{z}_h^i(1?z_h^i)x_j^i \\ forh=1,?,H \\ {v}_h\leftarrow v_h+\Delta v_h \\ forj=1,?,p \\ {w}_{hj}\leftarrow w_{hj}+\Delta w_{hj} \end{gathered}$$

分类

$$\begin{gathered} z_h=\sigma ({\mathbf{w}}_h^T\mathbf{x}) \\ f(x)=\sigma (v_0+\sum _{h=1}^H{v}_h{z}_h) \\ error=?\sum _{i=1}^n{y}^i\log (f(x^i))+(1?y^i)\log (1?f(x^i)) \\ \Delta v_h={\eta }_1\sum _{i=1}^n(y^i?f(x^i))z_h^i \\ \Delta w_{hj}=\eta \sum _{i=1}^n(y^i?f(x^i))v_h{z}_h^i(1?z_h^i)x_j^i \end{gathered}$$

K类

$$\begin{gathered} o_k^i=v_{k0}+\sum _{h=1}^H{v}_{kh}{z}_h^i \\ {f}_k(x)=\frac{\exp (o_k^i)}{{\sum }_{l=1}^K\exp (o_l^i)} \\ error=?\sum _{i=1}^n\sum _{k=1}^K{y}_k^i\log (f_k(x^i)) \\ \Delta v_{kh}={\eta }_1\sum _{i=1}^n(y_k^i?f_k(x^i))z_h^i \\ \Delta w_{hj}=\eta \sum _{i=1}^n(\sum _{k=1}^K(y_k^i?f_k(x^i))v_{kh})z_h^i(1?z_h^i)x_j^i \end{gathered}$$

多个隐藏层

$$\begin{gathered} f(x)=v_0+\sum _{l=1}^{H_2}{v}_l{z}_{2l} \\ {z}_{2l}=\sigma ({\mathbf{w}}_{2l}^T{\mathbf{z}}_1)=\sigma (w_{2l0}+\sum _{h=1}^{H_1}{w}_{2lh}{z}_{1h}) \\ {z}_{1h}=\sigma ({\mathbf{w}}_{1h}^T\mathbf{x})=\sigma (w_{1h0}+\sum _{j=1}^p{w}_{1hj}{x}_j) \\ f(x)=v_0+\sum _{l=1}^{H_2}{v}_l?\sigma (w_{2l0}+\sum _{h=1}^{H_1}{w}_{2lh}?\sigma (w_{1h0}+\sum _{j=1}^p{w}_{1hj}{x}_j)) \end{gathered}$$