M3DB Operator Timing Out Waiting for Instances

[cirego]

Thanks for opening an issue for the M3DB Operator! We'd love to help you, but we need the following information included
with any issue:

• What version of the operator are you running? Please include the docker tag. If using master, please include the git
  SHA logged when the operator first starts.

2.3.2

• What version of Kubernetes are you running? Please include the output of kubectl version.

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

• What are you trying to do?

Create a cluster using a M3DBCluster spec. The M3DB operator created 3 pods:

NAME                                                           READY   STATUS    RESTARTS   AGE
main-cluster-rep0-0                                            1/1     Running   0          13m
main-cluster-rep1-0                                            1/1     Running   0          13m
main-cluster-rep2-0                                            1/1     Running   0          13m

I have verified that there is a placement group defined in Etcd that matches these nodes (excluding the output from that command as I cannot get GitHub to format it correctly).

I am using .podIdentifyConfig.sources.NodeName to set the node identity, as I'm trying to configure this on i3en instances providing fast, local SSD storage at a reasonable price point. I'm using the local storage provisioner to mount a file system under the dataDirVolumeClaimTemplate part of the spec.

• What did you expect to happen?

I expected the cluster to come up with a valid placement group.

• What happened?

Each of the nodes are coming up but the m3db operator is complaining that it's timing out waiting for the instances:

{"level":"info","ts":1566406928.8094895,"msg":"processing statefulset","controller":"m3db-cluster-controller","name":"main-cluster-rep0"}                                                                                                                                                                                                                                                                                                                                                     
{"level":"info","ts":1566406934.674485,"msg":"processing statefulset","controller":"m3db-cluster-controller","name":"main-cluster-rep1"}
{"level":"info","ts":1566406940.3252335,"msg":"processing statefulset","controller":"m3db-cluster-controller","name":"main-cluster-rep2"}
{"level":"info","ts":1566406943.2652535,"msg":"found placement","controller":"m3db-cluster-controller","currentPods":3,"placementInsts":3}
{"level":"warn","ts":1566406943.2657042,"msg":"waiting for instances to be available","controller":"m3db-cluster-controller","instances":["{\"name\":\"main-cluster-rep0-0\",\"uid\":\"39c41235-c435-11e9-9191-068590fe2702\"}","{\"name\":\"main-cluster-rep1-0\",\"uid\":\"4313ad2e-c435-11e9-9191-068590fe2702\"}","{\"name\":\"main-cluster-rep2-0\",\"uid\":\"4985e6d8-c435-11e9-9191-068590fe2702\"}"]}

However, if I try to get the placement using the Coordinator, I get:

prompt] curl http://localhost:7201/api/v1/services/m3db/placement  2>/dev/null | jq .
{
  "error": "placement does not exist"
}

Here are the logs from one of the nodes:

2019/08/21 17:01:55 Go Runtime version: go1.12.6
2019/08/21 17:01:55 Build Version:      v0.10.2
2019/08/21 17:01:55 Build Revision:     78156d31
2019/08/21 17:01:55 Build Branch:       HEAD
2019/08/21 17:01:55 Build Date:         2019-06-14-08:19:09
2019/08/21 17:01:55 Build TimeUnix:     1560500349
2019-08-21T17:01:55.667Z        INFO    tracing disabled for m3query; set `tracing.backend` to enable
2019-08-21T17:01:55.684Z        INFO    resolved cluster namespace      {"namespace": "default"}
2019-08-21T17:01:55.791Z        INFO    no m3msg server configured
2019-08-21T17:01:55.791Z        INFO    using registered interrupt handler
2019-08-21T17:01:55.792Z        INFO    starting API server     {"address": "0.0.0.0:7201"}
{"level":"info","ts":1566406916.6688414,"msg":"tracing disabled; set `tracing.backend` to enable"}
{"level":"info","ts":1566406916.668889,"msg":"no seed nodes set, using dedicated etcd cluster"}
{"level":"warn","ts":1566406916.9082365,"msg":"max index query IDs concurrency was not set, falling back to default value"}
{"level":"warn","ts":1566406916.9085758,"msg":"host doesn't support HugeTLB, proceeding without it"}
{"level":"info","ts":1566406916.9258356,"msg":"bytes pool registering bucket capacity=824638727744, size=824638727752, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.9258587,"msg":"bytes pool registering bucket capacity=824638727760, size=824638727768, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.9258678,"msg":"bytes pool registering bucket capacity=824638727776, size=824638727784, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.9258747,"msg":"bytes pool registering bucket capacity=824638727792, size=824638727800, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.9258814,"msg":"bytes pool registering bucket capacity=824638727808, size=824638727816, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.9258852,"msg":"bytes pool registering bucket capacity=824638727824, size=824638727832, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.9258912,"msg":"bytes pool registering bucket capacity=824638727840, size=824638727848, refillLowWatermark=0.700000, refillHighWatermark=1.000000"}
{"level":"info","ts":1566406916.925929,"msg":"bytes pool init"}
{"level":"info","ts":1566406917.43345,"msg":"bytes pool init done"}
{"level":"info","ts":1566406917.8951168,"msg":"creating dynamic config service client with m3cluster"}
{"level":"warn","ts":1566406917.8954778,"msg":"could not load cache from file","file":"/var/lib/m3kv/_kv_m3db_main-cluster_m3db_embedded.json","error":"error opening cache file /var/lib/m3kv/_kv_m3db_main-cluster_m3db_embedded.json: open /var/lib/m3kv/_kv_m3db_main-cluster_m3db_embedded.json: no such file or directory"}                                                                                                                                                             
{"level":"info","ts":1566406918.1941822,"msg":"node tchannelthrift: listening","address":"0.0.0.0:9000"}
{"level":"info","ts":1566406918.194411,"msg":"node httpjson: listening","address":"0.0.0.0:9002"}
{"level":"info","ts":1566406918.1944299,"msg":"waiting for dynamic topology initialization, if this takes a long time, make sure that a topology/placement is configured"}
{"level":"info","ts":1566406918.1944451,"msg":"adding a watch","service":"m3db","env":"m3db/main-cluster","zone":"embedded","includeUnhealthy":true}
{"level":"warn","ts":1566406918.194501,"msg":"could not load cache from file","file":"/var/lib/m3kv/m3db_embedded.json","error":"error opening cache file /var/lib/m3kv/m3db_embedded.json: open /var/lib/m3kv/m3db_embedded.json: no such file or directory"}                                                                                                                                                                                                                                
{"level":"info","ts":1566406918.1993725,"msg":"initial topology / placement value received"}
{"level":"info","ts":1566406928.293682,"msg":"cluster tchannelthrift: listening","address":"0.0.0.0:9001"}
{"level":"info","ts":1566406928.2938051,"msg":"cluster httpjson: listening","address":"0.0.0.0:9003"}
{"level":"info","ts":1566406928.2938952,"msg":"cluster database initializing topology"}
{"level":"info","ts":1566406928.293915,"msg":"cluster database resolving topology"}
{"level":"info","ts":1566406928.2939181,"msg":"cluster database resolved topology"}
{"level":"warn","ts":1566406928.2939236,"msg":"topology has no shard set for host ID","hostID":"{\"name\":\"main-cluster-rep0-0\",\"uid\":\"60bfd803-c435-11e9-9191-068590fe2702\"}"}
{"level":"info","ts":1566406928.2998626,"msg":"creating namespaces watch"}
{"level":"info","ts":1566406928.2998955,"msg":"waiting for dynamic namespace registry initialization, if this takes a long time, make sure that a namespace is configured"}
{"level":"info","ts":1566406928.3020568,"msg":"initial namespace value received"}
{"level":"info","ts":1566406928.302152,"msg":"resolving namespaces with namespace watch"}
{"level":"warn","ts":1566406928.3022096,"msg":"can not update namespace schema to empty","namespace":"default","currentSchema":"none"}
{"level":"info","ts":1566406928.3022318,"msg":"updating database namespaces","adds":"[default]","updates":"[]","removals":"[]"}
{"level":"info","ts":1566406928.313478,"msg":"using registered interrupt handler"}
{"level":"info","ts":1566406928.31684,"msg":"bootstrap finished","namespace":"default","duration":0.000006768}
{"level":"info","ts":1566406928.3168573,"msg":"bootstrapped"}
{"level":"info","ts":1566406938.6979423,"msg":"successfully updated topology","numHosts":3}

[cirego]

It looks like this might be related to the workaround for issue #137. I commented out the lines in my cluster create script that patch the stateful sets and now m3db coordinator is able to find the instances:

{"level":"warn","ts":1566412976.6914983,"msg":"waiting for instances to be available","controller":"m3db-cluster-controller","instances":["{\"name\":\"main-cluster-rep1-0\",\"uid\":\"3ec08517-c443-11e9-9191-068590fe2702\"}"]}
{"level":"info","ts":1566412976.6915412,"msg":"successfully synced item","controller":"m3db-cluster-controller","key":"m3db/main-cluster"}                                                                    
{"level":"info","ts":1566412976.6916354,"msg":"Event(v1.ObjectReference{Kind:\"M3DBCluster\", Namespace:\"m3db\", Name:\"main-cluster\", UID:\"36dd04b6-c443-11e9-9191-068590fe2702\", APIVersion:\"operator.m3db.io/v1alpha1\", ResourceVersion:\"16900161\", FieldPath:\"\"}): type: 'Warning' reason: 'TimeLongerThanUsual' current unavailable instances: 1","controller":"m3db-cluster-controller"}                                                                                      
{"level":"info","ts":1566413036.6904697,"msg":"found placement","controller":"m3db-cluster-controller","currentPods":3,"placementInsts":3}                      
{"level":"info","ts":1566413036.7970896,"msg":"nothing to do","controller":"m3db-cluster-controller","childrensets":3,"zones":3,"generation":3,"rv":"16900786"}
{"level":"info","ts":1566413036.797113,"msg":"successfully synced item","controller":"m3db-cluster-controller","key":"m3db/main-cluster"}                      
{"level":"info","ts":1566413036.9256744,"msg":"found placement","controller":"m3db-cluster-controller","currentPods":3,"placementInsts":3}
{"level":"info","ts":1566413037.0380418,"msg":"nothing to do","controller":"m3db-cluster-controller","childrensets":3,"zones":3,"generation":4,"rv":"16900787"}

The logs from the nodes also support that things are being bootstrapped properly:

{"level":"info","ts":1566412947.4782019,"msg":"successfully parsed placement","version":2}                                           
{"level":"error","ts":1566412947.4787204,"msg":"cluster db failed marking shards available","shards":[117,139,454,681,735,928,1018,161,282,711,162,274,416,540,805,309,546,625,916,404,349,624,1001,30,651,118,133,484,917,112,148,222,334,773,799,368,503,1023,8,431,525,856,359,683,863,920,85,474,56,156,213,470,842,105,106,358,422,839,337,343,551,748,83,239,260,592,714,937,29,203,319,447,647,652,270,322,390,636,717,95,581,793,1021,250,556,676,700,745,847,1016,26,979,107,356,449,5
54,787,800,900,206,493,504,738,819,338,429,868,960,3,622,895,42,108,482,849,867,890,526,643,687,58,552,617,967,136,426,906,871,47,151,644,164,293,443,778,841,46,182,513,766,783,41,325,352,373,396,975,497,823,15,192,321,955,43,387,410,527,978,66,87,101,459,471,597,685,246,344,705,905,61,99,553,669,678,510,667,926,119,160,378,494,970,93,273,303,314,370,866,968,127,405,453,475,601,626,671,80,181,561,400,415,802,123,330,722,772,816,956,271,524,817,128,299,305,336,17,32,132,204,6
53,910,307,457,563,689,981,988,616,925,935,34,280,329,461,844,899,1003,33,224,141,393,612,633,637,661,991,111,121,424,740,859,950,861,16,137,171,249,635,710,834,982,258,306,342,409,752,138,445,417,608,888,45,150,541,638,810,479,754,811,48,686,901,941,40,168,383,464,562,6,210,267,490,609,694,944,27,74,389,891,295,448,750,212,289,491,517,880,100,237,789,79,276,587,600,792,846,73,296,304,539,713,51,158,199,519,715,764,874,53,70,231,442,590,630,677,962,287,514,618,632,131,216,57
5,727,882,983,613,708,135,229,468,850,896,897,4,91,113,189,829,922,215,403,586,655,749,878,67,946,22,104,312,364,450,558,933,297,402,509,768,845,500,695,702,726,876,995,544,767,1009,174,223,244,489,528,536,109,414,730,751,266,548,550,907,1022,183,406,545,580,732,840,957,126,381,777,81,301,360,594,621,13,699,755,10,440,441,523,578,921,407,992,14,283,291,666,261,462,693,721,808,648,964,365,372,657,7,690,753,781,170,392,430,739,1020,818,904,134,277,567,593,734,747,179,398,650,9
84,257,518,177,252,333,646,1005,313,354,385,501,824,924,9,60,146,506,918,889,44,54,191,376,529,570,591,55,350,675,948,883,520,555,809,311,362,532,649,684,784,835,38,233,977,180,184,323,537,698,791,806,163,639,656,153,165,196,634,286,943,993,660,814,902,1012,24,220,564,614,999,221,294,340,574,903,949,125,186,197,535,573,733,757,353,516,579,1011,21,427,568,919,938,958,178,331,351,397,797,974,837,838,929,976,188,423,701,37,723,954,39,477,238,242,472,804,830,502,605,718,884,994,
998,82,226,328,391,420,785,796,371,631,860,953,281,425,680,762,945,20,59,367,435,76,572,302,703,930,892,11,348,388,769,217,326,543,598,932,419,507,756,851,149,256,335,355,228,952,122,538,606,642,720,828,68,98,870,89,366,473,559,620,857,35,795,969,23,492,794,875,1017,380,481,534,569,599,469,583,665,175,194,1002,1004,52,65,202,219,382,971,357,515,854,102,832,176,339,549,719,169,198,332,547,641,1006,262,508,533,843,758,763,288,394,596,697,97,346,62,379,399,384,542,709,716,771,8
94,195,411,736,831,869,1014,292,298,432,437,986,623,765,18,31,50,72,86,268,691,589,728,731,452,585,603,931,951,1007,75,320,341,488,673,744,881,1008,253,662,218,821,813,913,961,235,300,316,584,759,774,942,96,485,530,663,879,934,451,779,985,144,245,308,761,912,460,94,438,446,607,972,200,375,285,480,595,915,966,704,947,2,28,90,129,207,565,746,263,272,275,310,495,640,25,254,433,465,885,234,466,478,798,803,939,264,742,927,936,172,444,455,395,668,688,103,145,496,664,725,965,511,61
9,706,707,743,852,855,120,208,230,347,959,147,659,801,909,1000,413,997,57,820,152,211,255,505,560,815,421,827,963,84,130,173,201,236,456,566,49,114,990,77,418,682,833,898,205,209,807,88,115,214,377,627,741,858,386,12,227,324,92,240,522,611,628,826,886,571,737,232,278,872,140,225,602,987,190,269,776,877,159,512,812,873,940,279,361,436,155,487,760,980,604,78,110,142,241,265,345,499,167,185,428,557,825,124,439,788,836,434,576,63,401,458,588,914,996,1,251,670,908,989,36,187,790,
822,154,911,5,243,483,658,64,116,775,973,315,369,770,786,893,157,247,408,467,531,1010,679,782,69,248,463,476,853,193,577,610,780,615,865,521,582,629,862,923,0,143,166,290,374,692,712,729,317,498,674,696,864,19,259,363,486,645,887,1013,71,412,654,672,1015,318,724,848,284,327,1019],"error":"key is not at the specified version"}
{"level":"info","ts":1566412947.4811554,"msg":"received update for topology"}
{"level":"info","ts":1566412947.481145,"msg":"received update from kv topology watch"}    
{"level":"info","ts":1566412947.4813147,"msg":"successfully updated topology","numHosts":3}
{"level":"info","ts":1566412947.4820154,"msg":"bootstrap finished","namespace":"default","duration":0.000072873}

Unfortunately, it doesn't appear to be working fully yet. I still get an error trying to list out placements:

curl http://localhost:7201/api/v1/services/m3coordinator/placement 2>/dev/null
 | jq .                        
{                          
  "error": "placement does not exist"
}

And while I can create a database:

curl -X POST http://localhost:7201/api/v1/database/create -d '{                                                                                                                                                                                                                                                                                                                                              
  "type": "local",                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  "namespaceName": "default2",                                                                                                                                                                                                                                                                                                                                                                                                                                                                
  "retentionTime": "48h"                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
}'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
{"namespace":{"registry":{"namespaces":{"default2":{"bootstrapEnabled":true,"flushEnabled":true,"writesToCommitLog":true,"cleanupEnabled":true,"repairEnabled":false,"retentionOptions":{"retentionPeriodNanos":"172800000000000","blockSizeNanos":"7200000000000","bufferFutureNanos":"120000000000","bufferPastNanos":"600000000000","blockDataExpiry":true,"blockDataExpiryAfterNotAccessPeriodNanos":"300000000000","futureRetentionPeriodNanos":"0"},"snapshotEnabled":true,"indexOptions"
:{"enabled":true,"blockSizeNanos":"7200000000000"},"schemaOptions":null,"coldWritesEnabled":false}}}},"placement":{"placement":{"instances":{"m3db_local":{"id":"m3db_local","isolationGroup":"local","zone":"embedded","weight":1,"endpoint":"127.0.0.1:9000","shards":[{"id":0,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":1,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":2,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":3,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":4,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":5,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":6,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":7,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":8,"state":"INITIALIZING
","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":9,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":10,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":11,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":12,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":13,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":14,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":15,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":16,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":17,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":18,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":19,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0",
"cutoffNanos":"0"},{"id":20,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":21,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":22,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":23,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":24,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":25,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":26,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":27,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":28,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":29,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":30,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":31,"state
":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":32,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":33,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":34,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":35,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":36,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":37,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":38,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":39,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":40,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":41,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":42,"state":"INITIALIZING","sourceId":"","cu
toverNanos":"0","cutoffNanos":"0"},{"id":43,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":44,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":45,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":46,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":47,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":48,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":49,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":50,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":51,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":52,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":53,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"}
,{"id":54,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":55,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":56,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":57,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":58,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":59,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":60,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":61,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":62,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"},{"id":63,"state":"INITIALIZING","sourceId":"","cutoverNanos":"0","cutoffNanos":"0"}],"shardSetId":0,"hostname":"localhost","port":9000}},"replicaFactor":1,"numShards":64,"isSharded":true,"cutoverTime":
"0","isMirrored":false,"maxShardSetId":0},"version":0}}

I'm unable to see it listed in via the API (tried to create default2 but only default is listed):

curl http://localhost:7201/api/v1/services/m3db/namespace  2>/dev/null | jq .
{
  "registry": {
    "namespaces": {
      "default": {
        "bootstrapEnabled": true,
        "flushEnabled": true,
        "writesToCommitLog": true,
        "cleanupEnabled": true,
        "repairEnabled": false,
        "retentionOptions": {
          "retentionPeriodNanos": "172800000000000",
          "blockSizeNanos": "7200000000000",
          "bufferFutureNanos": "600000000000",
          "bufferPastNanos": "600000000000",
          "blockDataExpiry": true,
          "blockDataExpiryAfterNotAccessPeriodNanos": "300000000000",
          "futureRetentionPeriodNanos": "0"
        },
        "snapshotEnabled": true,
        "indexOptions": {
          "enabled": true,
          "blockSizeNanos": "7200000000000"
        },
        "schemaOptions": null,
        "coldWritesEnabled": false
      }
    }
  }
}