pytorch中torch.nn.functional.normalize、nn.norm和LayerNorm计算中的区别
torch.nn.functional.normalize
计算方式
v = v max ( ∥ v ∥ p , ϵ ) \mathbf{v} = \frac{\mathbf{v}}{\max\left(\|\mathbf{v}\|_p, \epsilon\right)} v=max(∥v∥p,ϵ)v
pytorch例子
a = torch.arange(12).reshape(3,4).float()
torch.nn.functional.normalize(a, dim=-1)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],
[0.3563, 0.4454, 0.5345, 0.6236],
[0.4182, 0.4704, 0.5227, 0.5750]])
'''
这样除一下就相等了
a = torch.arange(12).reshape(3,4).float()
torch.nn.functional.normalize(a, dim=-1)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],
[0.3563, 0.4454, 0.5345, 0.6236],
[0.4182, 0.4704, 0.5227, 0.5750]])
'''
a / a.norm(dim=-1,keepdim=True)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],
[0.3563, 0.4454, 0.5345, 0.6236],
[0.4182, 0.4704, 0.5227, 0.5750]])
'''
nn.norm
计算方式
可以通过p参数指定是pro(Frobenium norm,其实就是平方和开根号)还是nuc(叫nuclear norm,其实就是SVD分解的奇异值,也就是Singular Value)
pytorch例子
a = torch.arange(12).reshape(3,4).float()
a.norm(dim=-1, keepdim=True)
'''
tensor([[ 3.7417],
[11.2250],
[19.1311]])
'''
LayerNorm
计算方式
分子减均值,分母对方差开根号,更多细节参考 超细节的BatchNorm/BN/LayerNorm/LN/SyncLN/ShuffleBN/RMSNorm知识点
pytorch例子
a = torch.arange(12).reshape(3,4).float()
(a - a.mean(dim=-1, keepdims=True)) / torch.sqrt(a.var(dim=-1, keepdims=True, correction=0))
'''
tensor([[-1.3416, -0.4472, 0.4472, 1.3416],
[-1.3416, -0.4472, 0.4472, 1.3416],
[-1.3416, -0.4472, 0.4472, 1.3416]])
'''