Replica (re)creation process ordering and completion

[Tristan971]

Hi,

I was following https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-data-migration/add_remove_replica/#using-altinity-operator, with operator v0.23.7, after losing a replica to an upstream bug (details not very relevant to this issue).

Anyway, trialing the process in our dev environment, I was surprised by 2 specific aspects:

1. The new replica does not get non-operator-managed users added to it
2. The new replica is immediately added to the cluster service endpoint

This causes a couple of issues if the initialization process is not extremely fast (in my cases, with 400+GB of data it's far from immediate...), as clients end up routed to an incomplete replica against which they can't even authenticate, while live replicas still exist.

Maybe this is expected behaviour, but it seems like it would be an improvement to address these 2 concerns, or at least document them in some way. Especially if there are other non-schema elements than users which I'm not thinking of but matter.

Thanks

[Slach]

The new replica does not get non-operator-managed users added to it

do you mean RBAC SQL users?

for replicate RBAC SQL created users over Zookeeper use following approach

spec:
  configuration:
    files:
     config.d/replicated_user_directories.xml: |
     <clickhouse>
       <user_directories replace="replace">
         <users_xml>
           <path>users.xml</path>
         </users_xml>
         <replicated>
             <zookeeper_path>/clickhouse/access</zookeeper_path>
         </replicated>
       </user_directories>
     </clickhouse>

[Tristan971]

I thought rbac was just not replicated at all in ClickHouse and you needed ON CLUSTER shenanigans when working with it… huh…

[alex-zaitsev]

Operator does not know passwords, for example. That's why it is hard to replicate users w/o zookeeper

[Tristan971]

That much I figured yeah, but I expected the operator could pull the hashed password and import the users + DB/table RBAC on the new replicas that way? Not that I was able to find at a glance where in ClickHouse they are

I suppose the solution in #1567 (comment) is what makes the most sense... 🤔

Well, if not possible/desirable, what do you think about the part of things about keeping the new replica marked as unhealthy (from k8s' point of view) until it's in sync?

[alex-zaitsev]

The thing is that it may take a long time for replica to get in sync, especially on big clusters. But we may add a switch for that, it is not that difficult to do.

As a workaround, you may use distributed table (even with a single shard) and 'max_replica_delay_for_distributed_queries' .

[Tristan971]

The thing is that it may take a long time for replica to get in sync, especially on big clusters

That is exactly what brought me to ask this yes 🙂

you may use distributed table

Hmmm that does seem a bit cumbersome, but it also seems like the right approach long-term indeed...

Well, if the switch is easy to add, it would be neat, and if not I can live without it. I figured since it's something the operator tracks already anyway it would be trivial-ish (ie: chi_clickhouse_metric_ReplicasMaxRelativeDelay metric)