Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sum(latents) vs mean(hidden) #184

Open
wassname opened this issue Jan 11, 2024 · 0 comments
Open

sum(latents) vs mean(hidden) #184

wassname opened this issue Jan 11, 2024 · 0 comments

Comments

@wassname
Copy link

I noticed that in the loss, by default, you sum the latent dimension in the l1 loss, but take the mean of the features dimension (in the reconstruction loss). This seems like it would make the optimal l1_coeffecient vary with the model size. Might be worth considering taking the mean of both those dimensions, that way the optimal l1_coeff should be more stable between autoencoder configurations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant