Adding autoscaling policy configuration for Collection.

[HoustonPutman]

Issue number of the reported bug or feature request: Resolves #55, Resolves #52

Describe your changes
Allowing for the setting of collection autoscaling policies via the Collection CRD.

[AceHack]

Will I be able to target Avialiabily Zones in AWS EKS as part of the policy?

[AceHack]

I would really like to set a policy that looks like

{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": "#ANY"}
or

{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": ["us-east-1a", "us-east-1b", "us-east-1c"]}

[AceHack]

More info
https://lucene.472066.n3.nabble.com/Rule-based-replication-or-sharding-td4407125.html#a4407660

[sepulworld]

This sounds great! We deal with really slow scale ups because new nodes don’t land in the right AZ where EBS is.

[HoustonPutman]

Yeah, I've thought about that for a while. Maybe we can add Kubernetes Node labels to our Solr nodes somehow?

How do y'all currently get solr to run across these availability zones? I'm assuming you have Kubernetes Node labels that describe which AZ each one is in, and you have Affinity rules setup to spread the solrcloud across them?

[HoustonPutman]

Looks like the adding Node Labels as Pod environment variables has been discussed for a while and nothing has been done about it yet. kubernetes/kubernetes#40610

That's unfortunate.

[AceHack]

We've currently just created 3 separate stateful sets and each one is tied to one AZ and adds the appropriate zone property to the solr startup. It's possible to read your AZ in an init container or as part of the pod startup, you'll have to call into the K8s APIs to get it.

[sepulworld]

Another option we will be exploring...

https://github.com/kubernetes/autoscaler/blob/d4a6664b389b052293de8f2c0d16839564911e1f/cluster-autoscaler/FAQ.md#i-have-a-couple-of-pending-pods-but-there-was-no-scale-up

basically, we will move over to 3 ASGs, 1 per AZ (what @AceHack mentions above), and let the one statefulset generate by Solr-operator do its thing. Cluster-autoscaler will keep a balance between AZs and should keep us with pods that come online fast without PV toleration issues. Ill follow up with our findings after we test this out.

[HoustonPutman]

Thanks for the link @sepulworld. Let's move this discussion over to #53, unless y'all think that we should block this PR until we can figure out a way to do this availability zone thing.

I think that the work for that is very important, but it could be done in separate PRs from this.

[HoustonPutman]

Ah, so I didn't understand that policies were managed separately from the collection. This will need some more thought before it's merged in.

Whatever we do, it will probably be similar to the solution we provide for configSets (#85). As both are basically uploading config to Solr/ZK through an API, and telling a collection to use a part of the config.

[janhoy]

Please be aware of https://cwiki.apache.org/confluence/display/SOLR/SIP-8+Autoscaling+policy+engine+V2 in this work - autoscaling may be re-built for solr 9

[HoustonPutman]

Thanks for linking this! It's definitely something to keep in mind. Especially if the operator/kubernetes could help handle the external orchestration.

[sigram]

Would be great for you all to visit that SIP and leave comments, especially in the direction of minimizing the scope of Solr autoscaling and maximizing the ability to use external frameworks ;)

[HoustonPutman]

Closing as Autoscaling has been deprecated in recent releases of Solr.