-
Notifications
You must be signed in to change notification settings - Fork 5.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Does it need to enhance matmul_op to support 4-D inputs #7319
Comments
A straightforward way is to use a
For self-attention, for a single head
Or, It seems that:
P.S. I use chrome with the GitHub with MathJax plugin to show Latex formula in the GitHub issue. |
When checking the dot product attention in ConvS2S and Transformer. I found in multi-head (self) attention, both inputs of the batched matrix multiplication can potentially be a 4-D tensor.
It seems we can enhance the current matmul_op to support 4-D tensor as its inputs, however, I guess this is determined by how to batch the computation to accelerate the computation speed.
Or the multiple heads can be simply wrapped in a Python API by using a
for
loop.The text was updated successfully, but these errors were encountered: