React to series/metadata limits with 400 instead of 429 in the ingester.

[stevesg]

Is your feature request related to a problem? Please describe.
Currently Cortex responds from the write path with 429 in the distributor (upon being rate limited) and the ingester (when series or metadata limits are reached). Rate limiting in the distributor is recoverable, and so a client can choose to respond to the 429 by retrying. The ingester limits, are not recoverable, and retrying is wasteful.

Describe the solution you'd like
We should make a change so that only the distributor responds with the 429, as it would be sensible to retry these. The ingester limits are not recoverable, and so should respond with another distinct 4xx code such as 400.

Describe alternatives you've considered
None, though for Prometheus specifically, a workaround is being introduced to phase the introduction of retry behavior:
prometheus/prometheus#8477

Additional context
Context about why this is significant can be found here:
prometheus/prometheus#8237

[pracucci]

👍 on this. You can also find a related discussion on CNCF Slack.