Softmax求导推导

Softmax求导

之前也推导过Softmax求导过程，但是结论没记住，每次都要重新推，很麻烦，现做个记录。

Softmax公式

$$\hat{y_i} =\frac{exp(x_i)}{\sum_{j} exp(x_j)}$$

注意：在实际实现softmax时需将$x$减去$max(x)$，防止溢出。

交叉熵损失

给定标签$\boldsymbol y$和模型输出$\boldsymbol{\hat y}$，交叉熵损失函数定义为：

$$L = -\sum_{i}y_i log(\hat y_i)$$

求导

使用链式法则求导

$$\frac{\partial L}{\partial \hat y_i} = -\frac{y_i}{\hat y_i}$$ $$\begin{align} \frac{\partial L}{\partial x_i} = &\sum_j \frac{\partial L}{\partial \hat y_j} \frac{\partial \hat y_j}{\partial x_i} = \sum_{i=j} -\frac{y_i}{\hat y_i} (\hat y_i-\hat y_i^2) + \sum_{i \not= j} \frac{y_i}{\hat y_j} \hat y_j \hat y_i = \sum_{i = j}(\hat y_i-1)y_i+\sum_{i \not=j}y_j\hat y_i \\ = & \begin{cases} 1(\hat y_i-1)=\hat y_i - 1 & y_i = 1\\ \hat y_i & y_i = 0 \end{cases} \\ =& \hat y_i - \boldsymbol 1_{y_i = 1} \end{align}$$

其中：

$$\frac{\partial \hat y_j}{\partial x_i} = \begin{cases} \frac{0 - exp(x_j)exp(x_i)}{S^2} = -\hat y_j \hat y_i & i \not= j \\ \frac{exp(x_j) S - exp(x_j)^2}{S^2} = \hat y_j - \hat y_j^2 & i = j \end{cases}$$ $$S = \sum_k exp(x_k)$$

用向量表示：

$$\frac{\partial L}{\partial x_i} = \boldsymbol {\hat y_i} - \boldsymbol {y}$$