【数式編】（逆伝播）のまとめ１ - 無限不可能性ドライブ

f:id:celaeno42:20181027233159p:plain

逆伝播についてはそれぞれの重みとバイアスの $\nabla E$ （勾配）の部分を載せます。
重みやバイアスは、 $\nabla E$ （勾配）に学習率を掛け、現在の重みやバイアスの値から引くことで更新していきます。

出力層

出力層についても共通部分を $\delta$ で表すことにしましょう。

ユニットo11

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{11}^4} = \frac{\partial E}{\partial u_1^4} \frac{\partial u_1^4}{\partial w_{11}^4} = (z_1^4 - t_1) \times z_1^3 = \delta_1^4 \times z_1^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{12}^4} = \frac{\partial E}{\partial u_1^4} \frac{\partial u_1^4}{\partial w_{12}^4} = (z_1^4 - t_1) \times z_2^3 = \delta_1^4 \times z_2^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{13}^4} = \frac{\partial E}{\partial u_1^4} \frac{\partial u_1^4}{\partial w_{13}^4} = (z_1^4 - t_1) \times z_3^3 = \delta_1^4 \times z_3^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{14}^4} = \frac{\partial E}{\partial u_1^4} \frac{\partial u_1^4}{\partial w_{14}^4} = (z_1^4 - t_1) \times z_4^3 = \delta_1^4 \times z_4^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial b_1^4} = \frac{\partial E}{\partial u_1^4} \frac{\partial u_1^4}{\partial b_1^4} = (z_1^4 - t_1) \times 1 = \delta_1^4 \times 1 \end{align}$

ユニットo12

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{21}^4} = \frac{\partial E}{\partial u_2^4} \frac{\partial u_2^4}{\partial w_{21}^4} = (z_2^4 - t_2) \times z_1^3 = \delta_2^4 \times z_1^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{22}^4} = \frac{\partial E}{\partial u_2^4} \frac{\partial u_2^4}{\partial w_{22}^4} = (z_2^4 - t_2) \times z_2^3 = \delta_2^4 \times z_2^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{23}^4} = \frac{\partial E}{\partial u_2^4} \frac{\partial u_2^4}{\partial w_{23}^4} = (z_2^4 - t_2) \times z_3^3 = \delta_2^4 \times z_3^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{24}^4} = \frac{\partial E}{\partial u_2^4} \frac{\partial u_2^4}{\partial w_{24}^4} = (z_2^4 - t_2) \times z_4^3 = \delta_2^4 \times z_4^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial b_2^4} = \frac{\partial E}{\partial u_2^4} \frac{\partial u_2^4}{\partial b_2^4} = (z_2^4 - t_2) \times 1 = \delta_2^4 \times 1 \end{align}$

ユニットo13

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{31}^4} = \frac{\partial E}{\partial u_3^4} \frac{\partial u_3^4}{\partial w_{31}^4} = (z_3^4 - t_3) \times z_1^3 = \delta_3^4 \times z_1^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{32}^4} = \frac{\partial E}{\partial u_3^4} \frac{\partial u_3^4}{\partial w_{32}^4} = (z_3^4 - t_3) \times z_2^3 = \delta_3^4 \times z_2^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{33}^4} = \frac{\partial E}{\partial u_3^4} \frac{\partial u_3^4}{\partial w_{33}^4} = (z_3^4 - t_3) \times z_3^3 = \delta_3^4 \times z_3^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{34}^4} = \frac{\partial E}{\partial u_3^4} \frac{\partial u_3^4}{\partial w_{34}^4} = (z_3^4 - t_3) \times z_4^3 = \delta_3^4 \times z_4^3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial b_3^4} = \frac{\partial E}{\partial u_3^4} \frac{\partial u_3^4}{\partial b_3^4} = (z_3^4 - t_3) \times 1 = \delta_3^4 \times 1 \end{align}$

隠れ層２層め

ユニットh21

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{11}^3} = \frac{\partial E}{\partial u_1^3} \frac{\partial u_1^3}{\partial w_{11}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{11}^4 \times ReLU'(u_1^3) \\ \\+ \delta_2^4 \times w_{21}^4 \times ReLU'(u_1^3) \\ \\+ \delta_3^4 \times w_{31}^4 \times ReLU'(u_1^3) \end{pmatrix} \times z_1^2 = \delta_1^3 \times z_1^2 \\ \\ \\ \frac{\partial E}{\partial w_{12}^3} = \frac{\partial E}{\partial u_1^3} \frac{\partial u_1^3}{\partial w_{12}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{11}^4 \times ReLU'(u_1^3) \\ \\+ \delta_2^4 \times w_{21}^4 \times ReLU'(u_1^3) \\ \\+ \delta_3^4 \times w_{31}^4 \times ReLU'(u_1^3) \end{pmatrix} \times z_2^2 = \delta_1^3 \times z_2^2 \\ \\ \\ \frac{\partial E}{\partial w_{13}^3} = \frac{\partial E}{\partial u_1^3} \frac{\partial u_1^3}{\partial w_{13}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{11}^4 \times ReLU'(u_1^3) \\ \\+ \delta_2^4 \times w_{21}^4 \times ReLU'(u_1^3) \\ \\+ \delta_3^4 \times w_{31}^4 \times ReLU'(u_1^3) \end{pmatrix} \times z_3^2 = \delta_1^3 \times z_3^2 \\ \\ \\ \frac{\partial E}{\partial b_1^3} = \frac{\partial E}{\partial u_1^3} \frac{\partial u_1^3}{\partial b_1^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{11}^4 \times ReLU'(u_1^3) \\ \\+ \delta_2^4 \times w_{21}^4 \times ReLU'(u_1^3) \\ \\+ \delta_3^4 \times w_{31}^4 \times ReLU'(u_1^3) \end{pmatrix} \times 1 = \delta_1^3 \times 1 \end{align}$

ユニットh22

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{21}^3} = \frac{\partial E}{\partial u_2^3} \frac{\partial u_2^3}{\partial w_{21}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{12}^4 \times ReLU'(u_2^3) \\ \\+ \delta_2^4 \times w_{22}^4 \times ReLU'(u_2^3) \\ \\+ \delta_3^4 \times w_{32}^4 \times ReLU'(u_2^3) \end{pmatrix} \times z_1^2 = \delta_2^3 \times z_1^2 \\ \\ \\ \frac{\partial E}{\partial w_{22}^3} = \frac{\partial E}{\partial u_2^3} \frac{\partial u_2^3}{\partial w_{22}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{12}^4 \times ReLU'(u_2^3) \\ \\+ \delta_2^4 \times w_{22}^4 \times ReLU'(u_2^3) \\ \\+ \delta_3^4 \times w_{32}^4 \times ReLU'(u_2^3) \end{pmatrix} \times z_2^2 = \delta_2^3 \times z_2^2 \\ \\ \\ \frac{\partial E}{\partial w_{23}^3} = \frac{\partial E}{\partial u_2^3} \frac{\partial u_2^3}{\partial w_{23}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{12}^4 \times ReLU'(u_2^3) \\ \\+ \delta_2^4 \times w_{22}^4 \times ReLU'(u_2^3) \\ \\+ \delta_3^4 \times w_{32}^4 \times ReLU'(u_2^3) \end{pmatrix} \times z_3^2 = \delta_2^3 \times z_3^2 \\ \\ \\ \frac{\partial E}{\partial b_2^3} = \frac{\partial E}{\partial u_2^3} \frac{\partial u_2^3}{\partial b_2^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{12}^4 \times ReLU'(u_2^3) \\ \\+ \delta_2^4 \times w_{22}^4 \times ReLU'(u_2^3) \\ \\+ \delta_3^4 \times w_{32}^4 \times ReLU'(u_2^3) \end{pmatrix} \times 1 = \delta_2^3 \times 1 \end{align}$

ユニットh23

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{31}^3} = \frac{\partial E}{\partial u_3^3} \frac{\partial u_3^3}{\partial w_{31}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{13}^4 \times ReLU'(u_3^3) \\ \\+ \delta_2^4 \times w_{23}^4 \times ReLU'(u_3^3) \\ \\+ \delta_3^4 \times w_{33}^4 \times ReLU'(u_3^3) \end{pmatrix} \times z_1^2 = \delta_3^3 \times z_1^2 \\ \\ \\ \frac{\partial E}{\partial w_{32}^3} = \frac{\partial E}{\partial u_3^3} \frac{\partial u_3^3}{\partial w_{32}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{13}^4 \times ReLU'(u_3^3) \\ \\+ \delta_2^4 \times w_{23}^4 \times ReLU'(u_3^3) \\ \\+ \delta_3^4 \times w_{33}^4 \times ReLU'(u_3^3) \end{pmatrix} \times z_2^2 = \delta_3^3 \times z_2^2 \\ \\ \\ \frac{\partial E}{\partial w_{33}^3} = \frac{\partial E}{\partial u_3^3} \frac{\partial u_3^3}{\partial w_{33}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{13}^4 \times ReLU'(u_3^3) \\ \\+ \delta_2^4 \times w_{23}^4 \times ReLU'(u_3^3) \\ \\+ \delta_3^4 \times w_{33}^4 \times ReLU'(u_3^3) \end{pmatrix} \times z_3^2 = \delta_3^3 \times z_3^2 \\ \\ \\ \frac{\partial E}{\partial b_3^3} = \frac{\partial E}{\partial u_3^3} \frac{\partial u_3^3}{\partial b_3^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{13}^4 \times ReLU'(u_3^3) \\ \\+ \delta_2^4 \times w_{23}^4 \times ReLU'(u_3^3) \\ \\+ \delta_3^4 \times w_{33}^4 \times ReLU'(u_3^3) \end{pmatrix} \times 1 = \delta_3^3 \times 1 \end{align}$

ユニットh24

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{41}^3} = \frac{\partial E}{\partial u_4^3} \frac{\partial u_4^3}{\partial w_{41}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{14}^4 \times ReLU'(u_4^3) \\ \\+ \delta_2^4 \times w_{24}^4 \times ReLU'(u_4^3) \\ \\+ \delta_3^4 \times w_{34}^4 \times ReLU'(u_4^3) \end{pmatrix} \times z_1^2 = \delta_4^3 \times z_1^2 \\ \\ \\ \frac{\partial E}{\partial w_{42}^3} = \frac{\partial E}{\partial u_4^3} \frac{\partial u_4^3}{\partial w_{42}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{14}^4 \times ReLU'(u_4^3) \\ \\+ \delta_2^4 \times w_{24}^4 \times ReLU'(u_4^3) \\ \\+ \delta_3^4 \times w_{34}^4 \times ReLU'(u_4^3) \end{pmatrix} \times z_2^2 = \delta_4^3 \times z_2^2 \\ \\ \\ \frac{\partial E}{\partial w_{43}^3} = \frac{\partial E}{\partial u_4^3} \frac{\partial u_4^3}{\partial w_{43}^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{14}^4 \times ReLU'(u_4^3) \\ \\+ \delta_2^4 \times w_{24}^4 \times ReLU'(u_4^3) \\ \\+ \delta_3^4 \times w_{34}^4 \times ReLU'(u_4^3) \end{pmatrix} \times z_3^2 = \delta_4^3 \times z_3^2 \\ \\ \\ \frac{\partial E}{\partial b_4^3} = \frac{\partial E}{\partial u_4^3} \frac{\partial u_4^3}{\partial b_4^3} &= \begin{pmatrix} 　\delta_1^4 \times w_{14}^4 \times ReLU'(u_4^3) \\ \\+ \delta_2^4 \times w_{24}^4 \times ReLU'(u_4^3) \\ \\+ \delta_3^4 \times w_{34}^4 \times ReLU'(u_4^3) \end{pmatrix} \times 1 = \delta_4^3 \times 1 \end{align}$

隠れ層１層め

ユニットh11

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{11}^2} &= \frac{\partial E}{\partial u_1^2} \frac{\partial u_1^2}{\partial w_{11}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{11}^3 \times ReLU'(u_1^2) \\ \\ + \delta_2^3 \times w_{21}^3 \times ReLU'(u_1^2) \\ \\ + \delta_3^3 \times w_{31}^3 \times ReLU'(u_1^2) \\ \\ + \delta_4^3 \times w_{41}^3 \times ReLU'(u_1^2) \end{pmatrix} \times x_1 = \delta_1^2 \times x_1 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{12}^2} &= \frac{\partial E}{\partial u_1^2} \frac{\partial u_1^2}{\partial w_{12}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{11}^3 \times ReLU'(u_1^2) \\ \\ + \delta_2^3 \times w_{21}^3 \times ReLU'(u_1^2) \\ \\ + \delta_3^3 \times w_{31}^3 \times ReLU'(u_1^2) \\ \\ + \delta_4^3 \times w_{41}^3 \times ReLU'(u_1^2) \end{pmatrix} \times x_2 = \delta_1^2 \times x_2 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{13}^2} &= \frac{\partial E}{\partial u_1^2} \frac{\partial u_1^2}{\partial w_{13}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{11}^3 \times ReLU'(u_1^2) \\ \\ + \delta_2^3 \times w_{21}^3 \times ReLU'(u_1^2) \\ \\ + \delta_3^3 \times w_{31}^3 \times ReLU'(u_1^2) \\ \\ + \delta_4^3 \times w_{41}^3 \times ReLU'(u_1^2) \end{pmatrix} \times x_3 = \delta_1^2 \times x_3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{14}^2} &= \frac{\partial E}{\partial u_1^2} \frac{\partial u_1^2}{\partial w_{14}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{11}^3 \times ReLU'(u_1^2) \\ \\ + \delta_2^3 \times w_{21}^3 \times ReLU'(u_1^2) \\ \\ + \delta_3^3 \times w_{31}^3 \times ReLU'(u_1^2) \\ \\ + \delta_4^3 \times w_{41}^3 \times ReLU'(u_1^2) \end{pmatrix} \times x_4 = \delta_1^2 \times x_4 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial b_1^2} &= \frac{\partial E}{\partial u_1^2} \frac{\partial u_1^2}{\partial b_1^2} = \begin{pmatrix} 　\delta_1^3 \times w_{11}^3 \times ReLU'(u_1^2) \\ \\ + \delta_2^3 \times w_{21}^3 \times ReLU'(u_1^2) \\ \\ + \delta_3^3 \times w_{31}^3 \times ReLU'(u_1^2) \\ \\ + \delta_4^3 \times w_{41}^3 \times ReLU'(u_1^2) \end{pmatrix} \times 1 = \delta_1^2 \times 1 \end{align}$

ユニットh12

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{21}^2} &= \frac{\partial E}{\partial u_2^2} \frac{\partial u_2^2}{\partial w_{21}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{12}^3 \times ReLU'(u_2^2) \\ \\ + \delta_2^3 \times w_{22}^3 \times ReLU'(u_2^2) \\ \\ + \delta_3^3 \times w_{32}^3 \times ReLU'(u_2^2) \\ \\ + \delta_4^3 \times w_{42}^3 \times ReLU'(u_2^2) \end{pmatrix} \times x_1 = \delta_2^2 \times x_1 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{22}^2} &= \frac{\partial E}{\partial u_2^2} \frac{\partial u_2^2}{\partial w_{22}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{12}^3 \times ReLU'(u_2^2) \\ \\ + \delta_2^3 \times w_{22}^3 \times ReLU'(u_2^2) \\ \\ + \delta_3^3 \times w_{32}^3 \times ReLU'(u_2^2) \\ \\ + \delta_4^3 \times w_{42}^3 \times ReLU'(u_2^2) \end{pmatrix} \times x_2 = \delta_2^2 \times x_2 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{23}^2} &= \frac{\partial E}{\partial u_2^2} \frac{\partial u_2^2}{\partial w_{23}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{12}^3 \times ReLU'(u_2^2) \\ \\ + \delta_2^3 \times w_{22}^3 \times ReLU'(u_2^2) \\ \\ + \delta_3^3 \times w_{32}^3 \times ReLU'(u_2^2) \\ \\ + \delta_4^3 \times w_{42}^3 \times ReLU'(u_2^2) \end{pmatrix} \times x_3 = \delta_2^2 \times x_3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{24}^2} &= \frac{\partial E}{\partial u_2^2} \frac{\partial u_2^2}{\partial w_{24}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{12}^3 \times ReLU'(u_2^2) \\ \\ + \delta_2^3 \times w_{22}^3 \times ReLU'(u_2^2) \\ \\ + \delta_3^3 \times w_{32}^3 \times ReLU'(u_2^2) \\ \\ + \delta_4^3 \times w_{42}^3 \times ReLU'(u_2^2) \end{pmatrix} \times x_4 = \delta_2^2 \times x_4 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial b_2^2} &= \frac{\partial E}{\partial u_2^2} \frac{\partial u_2^2}{\partial b_2^2} = \begin{pmatrix} 　\delta_1^3 \times w_{12}^3 \times ReLU'(u_2^2) \\ \\ + \delta_2^3 \times w_{22}^3 \times ReLU'(u_2^2) \\ \\ + \delta_3^3 \times w_{32}^3 \times ReLU'(u_2^2) \\ \\ + \delta_4^3 \times w_{42}^3 \times ReLU'(u_2^2) \end{pmatrix} \times 1 = \delta_2^2 \times 1 \end{align}$

ユニットh13

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{31}^2} &= \frac{\partial E}{\partial u_3^2} \frac{\partial u_3^2}{\partial w_{31}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{13}^3 \times ReLU'(u_3^2) \\ \\ + \delta_2^3 \times w_{23}^3 \times ReLU'(u_3^2) \\ \\ + \delta_3^3 \times w_{33}^3 \times ReLU'(u_3^2) \\ \\ + \delta_4^3 \times w_{43}^3 \times ReLU'(u_3^2) \end{pmatrix} \times x_1 = \delta_3^2 \times x_1 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{32}^2} &= \frac{\partial E}{\partial u_3^2} \frac{\partial u_3^2}{\partial w_{32}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{13}^3 \times ReLU'(u_3^2) \\ \\ + \delta_2^3 \times w_{23}^3 \times ReLU'(u_3^2) \\ \\ + \delta_3^3 \times w_{33}^3 \times ReLU'(u_3^2) \\ \\ + \delta_4^3 \times w_{43}^3 \times ReLU'(u_3^2) \end{pmatrix} \times x_2 = \delta_3^2 \times x_2 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{33}^2} &= \frac{\partial E}{\partial u_3^2} \frac{\partial u_3^2}{\partial w_{33}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{13}^3 \times ReLU'(u_3^2) \\ \\ + \delta_2^3 \times w_{23}^3 \times ReLU'(u_3^2) \\ \\ + \delta_3^3 \times w_{33}^3 \times ReLU'(u_3^2) \\ \\ + \delta_4^3 \times w_{43}^3 \times ReLU'(u_3^2) \end{pmatrix} \times x_3 = \delta_3^2 \times x_3 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial w_{34}^2} &= \frac{\partial E}{\partial u_3^2} \frac{\partial u_3^2}{\partial w_{34}^2} = \begin{pmatrix} 　\delta_1^3 \times w_{13}^3 \times ReLU'(u_3^2) \\ \\ + \delta_2^3 \times w_{23}^3 \times ReLU'(u_3^2) \\ \\ + \delta_3^3 \times w_{33}^3 \times ReLU'(u_3^2) \\ \\ + \delta_4^3 \times w_{43}^3 \times ReLU'(u_3^2) \end{pmatrix} \times x_4 = \delta_3^2 \times x_4 \end{align}$

$\displaystyle \begin{align} \frac{\partial E}{\partial b_3^2} &= \frac{\partial E}{\partial u_3^2} \frac{\partial u_3^2}{\partial b_3^2} = \begin{pmatrix} 　\delta_1^3 \times w_{13}^3 \times ReLU'(u_3^2) \\ \\ + \delta_2^3 \times w_{23}^3 \times ReLU'(u_3^2) \\ \\ + \delta_3^3 \times w_{33}^3 \times ReLU'(u_3^2) \\ \\ + \delta_4^3 \times w_{43}^3 \times ReLU'(u_3^2) \end{pmatrix} \times 1 = \delta_3^2 \times 1 \end{align}$

これらの計算式を実装することで、ニューラルネットワークを構築することができます。