数学表达式:
$$PE_{pos 2i}=sin(\frac{pos}{10000^{\frac{2i}{dmodel}}})$$
$$PE_{pos 2i+1}=cos(\frac{pos}{10000^{\frac{2i}{dmodel}}})$$
比如句子:"I love playing table card!"
首先分词:[I, love, play, -ing, table, card]
如果要编码词元"play"那么pos=2,pe=[2][], len(pe[2])=dmodel=512
代码:
import math
def positional_encoding(pos, pe):
for i in range(0,512,2):
pe[2][i] = math.sin(pos/(10000**((2*i)/d_model)))
pe[2][i+1] = math.cos(pos/(10000**((2*i)/d_model)))
return pe
打赏作者