parameter count of the model

[Tianchen627]

HI Tim,sorry to bother u but I have trouble when I count the parameter of the model.

In the paper the ConvE model on FB15k-237 with embedding size 200 has 5.05M parameters.How can I get the number 5.05M?Here is my counting method according to the code:

Embedding part:

"""
self.emb_e = torch.nn.Embedding(num_entities, args.embedding_dim, padding_idx=0)
self.emb_rel = torch.nn.Embedding(num_relations, args.embedding_dim, padding_idx=0)
"""
Embedding.weight (Tensor) – the learnable weights of the module of shape (num_embeddings, embedding_dim)
(14541+237)*200=2,955,600

Conv2d part:

"""
self.conv1 = torch.nn.Conv2d(1, 32, (3, 3), 1, 0, bias=args.use_bias)
"""
Conv2d.weight (Tensor) – the learnable weights of the module of shape (out_channels, in_channels,kernel_size[0], kernel_size[1])
Conv2d.bias (Tensor) – the learnable bias of the module of shape (out_channels)
32*1*3*3+32=288+32=320

projection to embedding dimension：

"""
self.register_parameter('b', Parameter(torch.zeros(num_entities)))
self.fc = torch.nn.Linear(args.hidden_size,args.embedding_dim)
"""
Linear.weight – the learnable weights of the module of shape (out_features,in_features)
Linear.bias – the learnable bias of the module of shape (out_features)
14541+9728*200+200=14541+1 945 600+200=1,960,341

When I sum all the parameters above I get the number 4,916,261 which is smaller than 5.05M.I know the BatchNorm layers have some parameters but not that much parameters.

plz tell me what did I miss.
thx!

[TimDettmers]

Thank you for the time writing this up. When I look at your calculation everything seems to be alright. There is one thing that you missed though, that is that reverse relationship has a separate embedding. That is you need to double the number of relations from 237 to 474. This adds another 47,400 parameters.

I am not sure if the model included the reverse relationships when I calculated the parameter count or not, but in any case, your estimate seems to be right. I think what I did when I calculated the overall parameters is to just take all pytorch parameters and calculate its parameters with python functions and use that as the parameter count. It might be, that I made a mistake and got the wrong estimate. The following code should work for that purpose of calculating the number of parameters automatically:

total_param = 0
for name, weight in model.named_parameters():
        total_param = weight.numel()
print('Total param: {0}'.format(total_param))

The problem with this code is that it counts some "empty" unused parameters. My codebase adds some empty string relations to each vocabulary and this leads to dead extra parameters. So in any case, I think it might be reasonable to just round to one digit and say that the model uses 5.0M parameters. I think that still would be fair for comparison purposes as it is still quite precise.

I hope that helps you to understand the details behind certain numbers better.

[Tianchen627]

Thanks for your detailed reply and I get the point.
Happy new year :)

[TimDettmers]

Happy new year :)