What's the reason for adding pos_embed and token_embed?

[shi-yan]

It seems strange that the token embedding is added to the positional embedding because addition can obscure the original values. For example, with 1+4=5, knowing only 5 makes it difficult to determine the original values. Why is addition used?

A more logical approach might be concatenation, which preserves the original values but increases computation. But why must the positional embedding be the same length as the token embedding, we can potentially use a shorter positional embedding?

The GPT-2 implementation video appears to use a trained positional embedding rather than the cos/sin functions described in the attention paper. Why is this approach taken?