Fix FlaxBigBirdEmbeddings

[ydshieh]

What does this PR do?

Current FlaxBigBirdEmbeddings applies layer norm before dropout, while BigBirdEmbeddings and Google's original BigBird
applies dropout first. This PR fixes this inconsistency.

Flax
(layernorm --> dropout)

transformers/src/transformers/models/big_bird/modeling_flax_big_bird.py

Lines 232 to 233 in 6f29029

	hidden_states = self.LayerNorm(hidden_states)
	hidden_states = self.dropout(hidden_states, deterministic=deterministic)

PyTorch
(dropout immediately after embedding)

transformers/src/transformers/models/big_bird/modeling_big_bird.py

Lines 311 to 312 in 6f29029

	embeddings = self.dropout(embeddings)
	embeddings = self.LayerNorm(embeddings)

Google
(dropout immediately after embedding)
https://github.com/google-research/bigbird/blob/5f2a5aa7fbab23e32e0e0b41c5f0192f0c023e05/bigbird/core/utils.py#L565-L566

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Thanks for fixing - also cc @vasudevgupta7 can you confirm?

[patil-suraj]

Thanks for the fix!

[ydshieh]

I will merge this PR today.