用 Python 从零开始创建神经网络（六）：优化（Optimization）介绍
优化（Optimization）介绍
- 引言
引言
在随机初始化的模型中，或者即使是采用更复杂方法初始化的模型中，我们的目标是随着时间的推移培训或教育一个模型。为了训练一个模型，我们调整权重和偏差以提高模型的准确性和置信度。为此，我们需要计算模型的错误量。损失函数，也被称为成本函数，是量化模型错误程度的算法。损失是这一指标的衡量。由于损失是模型的错误，我们理想情况下希望它为0。
你可能会想知道为什么我们不根据 argmax 准确度来计算模型的错误。回想我们之前的置信度示例：[0.22, 0.6, 0.18] 对比 [0.32, 0.36, 0.32]。如果正确的类确实是中间的那一个（索引1），那么两个例子之间的模型准确性将是相同的。但是这两个例子真的像彼此那样准确吗？它们不是，因为准确性只是简单地应用一个 argmax 到输出上，以找到最大值的索引。神经网络的输出实际上是置信度，对正确答案的更多置信度是更好的。因此，我们努力增加正确的置信度并减少错误放置的置信度：
import matplotlib.pyplot as plt
import nnfs
from nnfs.datasets import vertical_data
nnfs.init()
X, y = vertical_data(samples=100, classes=3)
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap='brg')
plt.show()
利用之前创建的代码，我们可以将这个新数据集与一个简单的神经网络结合使用：
# Create dataset
X, y = vertical_data(samples=100, classes=3)
# Create model
dense1 = Layer_Dense(2, 3) # first dense layer, 2 inputs
activation1 = Activation_ReLU()
dense2 = Layer_Dense(3, 3) # second dense layer, 3 inputs, 3 outputs
activation2 = Activation_Softmax()
# Create loss function
loss_function = Loss_CategoricalCrossentropy()
然后创建一些变量，以跟踪最佳损耗和相关的权重：
# Helper variables
lowest_loss = 9999999 # some initial value
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
我们将损失初始化为一个较大的值，当发现一个新的、较小的损失时，就会将其减小。我们还复制了权重和偏置值（copy() 可以确保完整复制，而不是引用对象）。现在，我们根据需要进行多次迭代，为权重和偏置值选择随机值，如果权重和偏置值产生的损失最小，则保存权重和偏置值：
for iteration in range(10000):
# Generate a new set of weights for iteration
dense1.weights = 0.05 * np.random.randn(2, 3)
dense1.biases = 0.05 * np.random.randn(1, 3)
dense2.weights = 0.05 * np.random.randn(3, 3)
dense2.biases = 0.05 * np.random.randn(1, 3)
# Perform a forward pass of the training data through this layer
dense1.forward(X)
activation1.forward(dense1.output)
dense2.forward(activation1.output)
activation2.forward(dense2.output)
# Perform a forward pass through activation function
# it takes the output of second dense layer here and returns loss
loss = loss_function.calculate(activation2.output, y)
# Calculate accuracy from output of activation2 and targets
# calculate values along first axis
predictions = np.argmax(activation2.output, axis=1)
accuracy = np.mean(predictions==y)
# If loss is smaller - print and save weights and biases aside
if loss < lowest_loss:
print('New set of weights found, iteration:', iteration, 'loss:', loss, 'acc:', accuracy)
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
lowest_loss = loss
>>>
New set of weights found, iteration: 0 loss: 1.0986564 acc:
0.3333333333333333
New set of weights found, iteration: 3 loss: 1.098138 acc:
0.3333333333333333
New set of weights found, iteration: 117 loss: 1.0980115 acc:
0.3333333333333333
New set of weights found, iteration: 124 loss: 1.0977516 acc: 0.6
New set of weights found, iteration: 165 loss: 1.097571 acc:
0.3333333333333333
New set of weights found, iteration: 552 loss: 1.0974693 acc: 0.34
New set of weights found, iteration: 778 loss: 1.0968257 acc:
0.3333333333333333
New set of weights found, iteration: 4307 loss: 1.0965533 acc:
0.3333333333333333
New set of weights found, iteration: 4615 loss: 1.0964499 acc:
0.3333333333333333
New set of weights found, iteration: 9450 loss: 1.0964295 acc:
0.3333333333333333
完整代码：
import numpy as np
import matplotlib.pyplot as plt
import nnfs
from nnfs.datasets import vertical_data
nnfs.init()
# Dense layer
class Layer_Dense:
# Layer initialization
def __init__(self, n_inputs, n_neurons):
# Initialize weights and biases
self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)
self.biases = np.zeros((1, n_neurons))
# Forward pass
def forward(self, inputs):
# Calculate output values from inputs, weights and biases
self.output = np.dot(inputs, self.weights) + self.biases
# ReLU activation
class Activation_ReLU:
# Forward pass
def forward(self, inputs):
# Calculate output values from input
self.output = np.maximum(0, inputs)
# Softmax activation
class Activation_Softmax:
# Forward pass
def forward(self, inputs):
# Get unnormalized probabilities
exp_values = np.exp(inputs - np.max(inputs, axis=1,
keepdims=True))
# Normalize them for each sample
probabilities = exp_values / np.sum(exp_values, axis=1,
keepdims=True)
self.output = probabilities
# Common loss class
class Loss:
# Calculates the data and regularization losses
# given model output and ground truth values
def calculate(self, output, y):
# Calculate sample losses
sample_losses = self.forward(output, y)
# Calculate mean loss
data_loss = np.mean(sample_losses)
# Return loss
return data_loss
# Cross-entropy loss
class Loss_CategoricalCrossentropy(Loss):
# Forward pass
def forward(self, y_pred, y_true):
# Number of samples in a batch
samples = len(y_pred)
# Clip data to prevent division by 0
# Clip both sides to not drag mean towards any value
y_pred_clipped = np.clip(y_pred, 1e-7, 1 - 1e-7)
# Probabilities for target values -
# only if categorical labels
if len(y_true.shape) == 1:
correct_confidences = y_pred_clipped[range(samples), y_true]
# Mask values - only for one-hot encoded labels
elif len(y_true.shape) == 2:
correct_confidences = np.sum(y_pred_clipped * y_true, axis=1)
# Losses
negative_log_likelihoods = -np.log(correct_confidences)
return negative_log_likelihoods
nnfs.init()
# Create dataset
X, y = vertical_data(samples=100, classes=3)
# Create model
dense1 = Layer_Dense(2, 3) # first dense layer, 2 inputs
activation1 = Activation_ReLU()
dense2 = Layer_Dense(3, 3) # second dense layer, 3 inputs, 3 outputs
activation2 = Activation_Softmax()
# Create loss function
loss_function = Loss_CategoricalCrossentropy()
# Helper variables
lowest_loss = 9999999 # some initial value
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
for iteration in range(100000):
# Generate a new set of weights for iteration
dense1.weights = 0.05 * np.random.randn(2, 3)
dense1.biases = 0.05 * np.random.randn(1, 3)
dense2.weights = 0.05 * np.random.randn(3, 3)
dense2.biases = 0.05 * np.random.randn(1, 3)
# Perform a forward pass of the training data through this layer
dense1.forward(X)
activation1.forward(dense1.output)
dense2.forward(activation1.output)
activation2.forward(dense2.output)
# Perform a forward pass through activation function
# it takes the output of second dense layer here and returns loss
loss = loss_function.calculate(activation2.output, y)
# Calculate accuracy from output of activation2 and targets
# calculate values along first axis
predictions = np.argmax(activation2.output, axis=1)
accuracy = np.mean(predictions==y)
# If loss is smaller - print and save weights and biases aside
if loss < lowest_loss:
print('New set of weights found, iteration:', iteration, 'loss:', loss, 'acc:', accuracy)
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
lowest_loss = loss
损失当然有所下降，但幅度不大。准确率也没有提高，只有一种情况例外，即模型随机找到了一组权重，从而提高了准确率。不过，在损失相当大的情况下，这种状态并不稳定。再运行 90,000 次迭代，总计 100,000 次：
>>>
New set of weights found, iteration: 13361 loss: 1.0963014 acc: 0.3333333333333333
New set of weights found, iteration: 14001 loss: 1.0959858 acc: 0.3333333333333333
New set of weights found, iteration: 24598 loss: 1.0947443 acc: 0.3333333333333333
损耗继续下降，但精确度没有变化。这似乎不是一种可靠的最小化损失的方法。运行 10 亿次迭代后，最佳结果（损失最小）如下：
>>>
New set of weights found, iteration: 229865000 loss: 1.0911305 acc:
0.3333333333333333
即使是使用这种基础数据集，我们也可以看到，随机搜索权重和偏差的组合需要的时间太长，无法成为一个可接受的方法。另一个想法可能是，不是在每次迭代中都用随机选择的值来设置参数，而是应用这些值的一部分到参数上。通过这种方法，权重将根据当前给我们带来最低损失的结果进行更新，而不是无目的地随机更新。如果调整减少了损失，我们将使其成为新的调整起点。如果由于调整而导致损失增加，那么我们将回到之前的点。使用之前类似的代码，我们将首先从随机选择权重和偏差转变为随机调整它们：
# Update weights with some small random values
dense1.weights += 0.05 * np.random.randn(2, 3)
dense1.biases += 0.05 * np.random.randn(1, 3)
dense2.weights += 0.05 * np.random.randn(3, 3)
dense2.biases += 0.05 * np.random.randn(1, 3)
然后，我们将把结尾的 if 语句改为：
# If loss is smaller - print and save weights and biases aside
if loss < lowest_loss:
print('New set of weights found, iteration:', iteration, 'loss:', loss, 'acc:', accuracy)
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
lowest_loss = loss
# Revert weights and biases
else:
dense1.weights = best_dense1_weights.copy()
dense1.biases = best_dense1_biases.copy()
dense2.weights = best_dense2_weights.copy()
dense2.biases = best_dense2_biases.copy()
修改后完整代码：
import numpy as np
import matplotlib.pyplot as plt
import nnfs
from nnfs.datasets import vertical_data
# Dense layer
class Layer_Dense:
# Layer initialization
def __init__(self, n_inputs, n_neurons):
# Initialize weights and biases
self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)
self.biases = np.zeros((1, n_neurons))
# Forward pass
def forward(self, inputs):
# Calculate output values from inputs, weights and biases
self.output = np.dot(inputs, self.weights) + self.biases
# ReLU activation
class Activation_ReLU:
# Forward pass
def forward(self, inputs):
# Calculate output values from input
self.output = np.maximum(0, inputs)
# Softmax activation
class Activation_Softmax:
# Forward pass
def forward(self, inputs):
# Get unnormalized probabilities
exp_values = np.exp(inputs - np.max(inputs, axis=1,
keepdims=True))
# Normalize them for each sample
probabilities = exp_values / np.sum(exp_values, axis=1,
keepdims=True)
self.output = probabilities
# Common loss class
class Loss:
# Calculates the data and regularization losses
# given model output and ground truth values
def calculate(self, output, y):
# Calculate sample losses
sample_losses = self.forward(output, y)
# Calculate mean loss
data_loss = np.mean(sample_losses)
# Return loss
return data_loss
# Cross-entropy loss
class Loss_CategoricalCrossentropy(Loss):
# Forward pass
def forward(self, y_pred, y_true):
# Number of samples in a batch
samples = len(y_pred)
# Clip data to prevent division by 0
# Clip both sides to not drag mean towards any value
y_pred_clipped = np.clip(y_pred, 1e-7, 1 - 1e-7)
# Probabilities for target values -
# only if categorical labels
if len(y_true.shape) == 1:
correct_confidences = y_pred_clipped[range(samples), y_true]
# Mask values - only for one-hot encoded labels
elif len(y_true.shape) == 2:
correct_confidences = np.sum(y_pred_clipped * y_true, axis=1)
# Losses
negative_log_likelihoods = -np.log(correct_confidences)
return negative_log_likelihoods
nnfs.init()
# Create dataset
X, y = vertical_data(samples=100, classes=3)
# Create model
dense1 = Layer_Dense(2, 3) # first dense layer, 2 inputs
activation1 = Activation_ReLU()
dense2 = Layer_Dense(3, 3) # second dense layer, 3 inputs, 3 outputs
activation2 = Activation_Softmax()
# Create loss function
loss_function = Loss_CategoricalCrossentropy()
# Helper variables
lowest_loss = 9999999 # some initial value
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
for iteration in range(10000):
# Update weights with some small random values
dense1.weights += 0.05 * np.random.randn(2, 3)
dense1.biases += 0.05 * np.random.randn(1, 3)
dense2.weights += 0.05 * np.random.randn(3, 3)
dense2.biases += 0.05 * np.random.randn(1, 3)
# Perform a forward pass of the training data through this layer
dense1.forward(X)
activation1.forward(dense1.output)
dense2.forward(activation1.output)
activation2.forward(dense2.output)
# Perform a forward pass through activation function
# it takes the output of second dense layer here and returns loss
loss = loss_function.calculate(activation2.output, y)
# Calculate accuracy from output of activation2 and targets
# calculate values along first axis
predictions = np.argmax(activation2.output, axis=1)
accuracy = np.mean(predictions==y)
# If loss is smaller - print and save weights and biases aside
if loss < lowest_loss:
print('New set of weights found, iteration:', iteration, 'loss:', loss, 'acc:', accuracy)
best_dense1_weights = dense1.weights.copy()
best_dense1_biases = dense1.biases.copy()
best_dense2_weights = dense2.weights.copy()
best_dense2_biases = dense2.biases.copy()
lowest_loss = loss
# Revert weights and biases
else:
dense1.weights = best_dense1_weights.copy()
dense1.biases = best_dense1_biases.copy()
dense2.weights = best_dense2_weights.copy()
dense2.biases = best_dense2_biases.copy()
>>>
New set of weights found, iteration: 0 loss: 1.0987684 acc: 0.3333333333333333
...
New set of weights found, iteration: 29 loss: 1.0725244 acc: 0.5266666666666666
New set of weights found, iteration: 30 loss: 1.0724432 acc: 0.3466666666666667
...
New set of weights found, iteration: 48 loss: 1.0303522 acc: 0.6666666666666666
New set of weights found, iteration: 49 loss: 1.0292586 acc: 0.6666666666666666
...
New set of weights found, iteration: 97 loss: 0.9277446 acc: 0.7333333333333333
...
New set of weights found, iteration: 152 loss: 0.73390484 acc: 0.8433333333333334
New set of weights found, iteration: 156 loss: 0.7235515 acc: 0.87
New set of weights found, iteration: 160 loss: 0.7049076 acc: 0.9066666666666666
...
New set of weights found, iteration: 7446 loss: 0.17280102 acc: 0.9333333333333333
New set of weights found, iteration: 9397 loss: 0.17279711 acc: 0.93
这次的损失下降了不少，准确率也大幅提高。应用一小部分随机值所得到的结果，我们几乎可以称之为解决方案。如果尝试 100,000 次迭代，也不会有太大进展：
>>>
...
New set of weights found, iteration: 14206 loss: 0.1727932 acc:
0.9333333333333333
New set of weights found, iteration: 63704 loss: 0.17278232 acc:
0.9333333333333333
让我们用之前看到的螺旋数据集来试试：
from nnfs.datasets import spiral_data
X, y = spiral_data(samples=100, classes=3)
>>>
New set of weights found, iteration: 0 loss: 1.1008677 acc: 0.3333333333333333
...
New set of weights found, iteration: 31 loss: 1.0982264 acc: 0.37333333333333335
...
New set of weights found, iteration: 65 loss: 1.0954362 acc: 0.38333333333333336
New set of weights found, iteration: 67 loss: 1.093989 acc: 0.4166666666666667
...
New set of weights found, iteration: 129 loss: 1.0874122 acc: 0.42333333333333334
...
New set of weights found, iteration: 5415 loss: 1.0790575 acc: 0.39
这次训练几乎毫无进展。损失略有减少，精确度勉强高于初始值。稍后我们会了解到，造成这种情况的最可能原因叫做损失的局部最小值。数据复杂度在这里也并非无关紧要。事实证明，难题之所以难，是有原因的，我们需要更聪明地处理这个问题。
本章的章节代码、更多资源和勘误表：https://nnfs.io/ch6