stellar#145: updated helm chart upgrade steps for replica counts

docs/run-platform-server/running.mdx

@@ -40,7 +40,10 @@ If you chose containerized installation, you don't execute the horizon binary di
   helm install my-horizon stellar/horizon \
   --namespace my-horizon-namespace-on-cluster \
   --set ingest.persistence.enabled=true \
+  --set web.replicaCount=1 \
   --set web.enabled=true \
+  --set ingest.enabled=true \
+  --set ingest.replicaCount=1 \
   --set web.existingSecret=my-db-secret \
   --set global.image.horizon.tag=2.26.1 \
   --set global.network=testnet \
@@ -56,7 +59,7 @@ If you chose containerized installation, you don't execute the horizon binary di
 
   - uses the `global.network=[testnet|pubnet]` parameter, this automates generation of all the horizon configuration parameters specific to network such as archive urls, captive core config, etc and other parameters as mentioned on [Configuring](./configuring.mdx).
   - `global.image.horizon.tag` should be set to one of docker hub tags published on [stellar/stellar-horizon](https://hub.docker.com/r/stellar/stellar-horizon)
-  - enables all roles on the deployment instance: ingesting, api, transaction submission.
+  - enables all roles on the deployment instance: ingesting, and web api(includes transaction submission). If you choose to have a multi-instance deployment with each instance performing a single role of just web api or ingestion, then you will do two helm installations, one for each role: `my-horizon-ingestion-installation` and `my-horizon-api-installation`. Each of these helm installations will set `ingest.enabled`, `web.enabled`, `ingest.replicaCount`, `web.replicaCount` respectively for the role they are performing.
   - to customize further, the best approach is to download the [horizon helm chart values.yaml](https://github.com/stellar/helm-charts/blob/main/charts/horizon/values.yaml), update the settings in your local copy of values.yaml, and pass to helm install, rather than have many individual `--set` on helm install:
 
     <CodeExample>

docs/run-platform-server/upgrading.mdx

@@ -54,7 +54,7 @@ Upgrading production deployents should leverage a secondary, hot-backup deployme
 
 A good strategy for upgrading horizon and applicable to single or multi-instance deployments - shut all instances down, install new horizon version on one of the ingesting instances first. The reason being horizon software will only initate `State Rebuild` and `DB Schema Migration` actions related to an upgrade on an instance that it detects ingestion has been enabled with configuration parameter, `INGEST=true`. This lowers complexity for you during the upgrade as you only need to focus on one instance and it avoids potential concurrent horizon ingestion processes attempting the same upgrade on the database.
 
-- bare metal installations, use apt package manager on linux.
+- bare metal installations, stop the horizon process on all instances first, then shell into one instance that is configured for ingestion, and use apt package manager on linux.
 
   <CodeExample>
 
@@ -65,20 +65,34 @@ A good strategy for upgrading horizon and applicable to single or multi-instance
 
   </CodeExample>
 
-- docker daemon deployments, set the new tag for the image based on release published on dockerhub - [stellar/stellar-horizon](https://hub.docker.com/r/stellar/stellar-horizon/tags)
-- For kubernetes deployments, use your helm cli tool to upgrade the helm installation of horizon on kubernetes cluster that was created prior by following the [run steps](./running.mdx). You will set the `global.image.horizon.tag` to the release tag published on [stellar/stellar-horizon](https://hub.docker.com/r/stellar/stellar-horizon/tags)
+- docker daemon deployments, stop all docker containers first, then choose one container that has ingestion enabled, set the new tag for the image based on release published on dockerhub - [stellar/stellar-horizon](https://hub.docker.com/r/stellar/stellar-horizon/tags), and start the container in docker daemon.
+- For helm installations on kubernetes, first use your helm cli tool to stop all horizon instances by scaling all your horizon installations(for ingest and web) down to 0 replicas, which you've created prior on [run steps](./running.mdx).
+
+  <CodeExample>
+
+  ```bash
+  helm upgrade all-my-horizon-installations \
+  --namespace my-horizon-namespace-on-cluster \
+  --set ingest.replicaCount=0 \
+  --set web.replicaCount=0
+  ```
+
+  </CodeExample>
+
+  Now, use helm to start just a single horizon instance from a helm installation that has ingestion enabled on kubernetes cluster, you will set the `global.image.horizon.tag` to the release tag published on [stellar/stellar-horizon](https://hub.docker.com/r/stellar/stellar-horizon/tags)
 
   <CodeExample>
 
   ```bash
   helm upgrade my-horizon \
   --namespace my-horizon-namespace-on-cluster  \
   --set global.image.horizon.tag=new_horizon_release_number
+  --set ingest.replicaCount=1
   ```
 
   </CodeExample>
 
-#### Confirming the upgrade on single ingestion instance first
+### Confirming the upgrade on single ingestion instance first
 
 If you have [monitoring](./monitoring.mdx) infrastructure in place, then you have two options for assessing the upgrade status:
 
@@ -101,8 +115,20 @@ If metrics and/or the horizon 'status' url respones don't indicate healthy statu
 - A delay in horizon achieving healthy status after an upgrade is expected and legitmate for any upgrade cases where `State Rebuild` or `DB Migration` was noted in the release delta as part of prior [Determine the target version for upgrade step](#determine-the-target-version-for-upgrade). Typically the notes will also mention relative timeframe expectations for those to complete which can be factored in to how long to wait on delay.
 - Check the logs from the upgraded instance to confirm what's going on. Any `State Rebuild` or `DB Migration` initiated will be mentioned.
 
-#### Upgrade all remaining instances
+### Upgrade all remaining instances
+
+At this point, you have upgraded one ingesting instance to the new horizon version, it has automatically updated the database if required and the instance is running with healthy status. Now, install the same horizon software version on the remainder of instances, restarting each after the upgrade. For bare-metal and docker daemon installations that will likely be self explanatory on how to accomplish that for remainder of instances, on helm chart installations, run the helm upgrade again, setting the image tag and also restoring original `replicaCount`s:
+
+<CodeExample>
+
+```bash
+helm upgrade all-my-horizon-installations \
+--namespace my-horizon-namespace-on-cluster \
+--set ingest.replicaCount=1 \
+--set web.replicaCount=1 \
+--set global.image.horizon.tag=new_horizon_release_number
+```
 
-At this point, you have upgraded one ingesting instance to the new horizon version, it has automatically updated the database if required and the instance is running with healthy status. Now, install the same horizon software version on the remainder of instances, restarting each after the upgrade.
+</CodeExample>
 
 For production deployments following the hot backup or blue/green model, this is the opportunity to confirm the inactive deployment has taken the upgrade correctly and stable, at which point, switch the load balancers now to forward traffic to the inactive deployment, making it the active deployment. Now, can take time to perform same upgrade on the other deployment which is now inactive.