Branches sometimes not replicated on fetch #782

cloudhead · 2021-09-03T10:36:41Z

radicle-link @ e640fed
When connecting two identical peers (the radicle org node) on the localhost (same machine), one of which has 4 projects and the other being fresh (empty monorepo), after replication of the 4 projects, I see that only one of them has had the default branch replicated, while the others only have certain refs replicated. I've included a diff of the trees.

On the left is the "server" that has all the data, and on the right is the "client" that is fetching from the server.

(logs: https://gist.github.com/cloudhead/02c469951f10c2ccc4d55a97d4adf362)

monorepo1/git/refs/namespaces						  |	monorepo2/git/refs/namespaces/
├── hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto					├── hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto
│   └── refs									│   └── refs
│       ├── rad									│       ├── rad
│       │   ├── id								│       │   ├── id
│       │   ├── ids								│       │   ├── ids
│       │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio			│       │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│       │   └── signed_refs							│       │   └── signed_refs
│       └── remotes								│       └── remotes
│           ├── hydgnjorosxetwr3xrqsfazuornz7m8wrx7brdyk5nxc4j9j9jhjxg	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe		│           ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│           │   ├── heads						  <
│           │   │   ├── cloudhead					  <
│           │   │   │   └── master					  <
│           │   │   └── master						  <
│           │   └── rad								│           │   └── rad
│           │       ├── id							│           │       ├── id
│           │       ├── ids							│           │       ├── ids
│           │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio		│           │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│           │       ├── self							│           │       ├── self
│           │       └── signed_refs						│           │       └── signed_refs
│           ├── hynkyndc6w3p8urucakobzna7sxwgcqny7xxtw88dtx3pkf7m3nrzc	  |	│           └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
│           │   └── rad							  <
│           │       └── ids						  <
│           └── hyyui4i75gkh14um3bhc3f67bm6n3hhof61584ukw6zbwnkyzhuj1o	  <
│               └── rad								│               └── rad
│                   └── ids						  |	│                   ├── id
									  >	│                   ├── ids
									  >	│                   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
									  >	│                   └── signed_refs
├── hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy					├── hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy
│   └── refs									│   └── refs
│       ├── rad									│       ├── rad
│       │   ├── id								│       │   ├── id
│       │   ├── ids								│       │   ├── ids
│       │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio			│       │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│       │   └── signed_refs							│       │   └── signed_refs
│       └── remotes								│       └── remotes
│           ├── hybz9gfgtd9d4pd14a6r66j5hz6f77fed4jdu7pana4fxaxbt369kg	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyd4ge6w1ig37cjg31zzi5kdgd91xtwezkzx91ypzwnh4dn4m5qm7h	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyd7cb39wws6jz5wmifzwtjn8u57siw6knipqttk6nhc657r1fozaw	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hydgnjorosxetwr3xrqsfazuornz7m8wrx7brdyk5nxc4j9j9jhjxg	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe		│           ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│           │   ├── heads							│           │   ├── heads
│           │   │   └── master							│           │   │   └── master
│           │   └── rad								│           │   └── rad
│           │       ├── id							│           │       ├── id
│           │       ├── ids							│           │       ├── ids
│           │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio		│           │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│           │       ├── self							│           │       ├── self
│           │       └── signed_refs						│           │       └── signed_refs
│           ├── hynkyndc6w3p8urucakobzna7sxwgcqny7xxtw88dtx3pkf7m3nrzc	  |	│           └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hynpd6jij8ifug9zkxojj9q5k9mcfxu1pak1t71qbc3b1od1yrmp7h	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hynqybwjzwzfcrydm1iar535qog77fst8jyk8w3yny9nz6d94cwq81	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyy4ebh8amk3ypyawgpjnfpbgkxqwdo5n1iif5s17yghwmewrf4d8a	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyyao9irufccjs8pjjkijasgdt1ts4toiynkceg3scg58th95mmo5h	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyycogqwxy5bg9m9oyqydyf5ychz7qt311fossax5qbngt393b1o91	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyyf6mr6awmj7f4ug9fyfibp9zcidqsz5bjsc5a49on4wrm6kxmztq	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyyrxmbfwe33p54cc41je7fhq7mccg56jiy4szeyehhk4pj9uxwhzs	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           ├── hyyui4i75gkh14um3bhc3f67bm6n3hhof61584ukw6zbwnkyzhuj1o	  <
│           │   └── rad							  <
│           │       └── ids						  <
│           └── hyyz8tw1yp134qprtczndhr43ki8rwjqz13etitpxs8qpc69mjjqxh	  <
│               └── rad								│               └── rad
│                   └── ids						  |	│                   ├── id
									  >	│                   ├── ids
									  >	│                   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
									  >	│                   └── signed_refs
├── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio					├── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│   └── refs									│   └── refs
│       └── rad									│       └── rad
│           └── id								│           └── id
├── hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo					├── hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo
│   └── refs									│   └── refs
│       ├── rad									│       ├── rad
│       │   ├── id								│       │   ├── id
│       │   ├── ids								│       │   ├── ids
│       │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio			│       │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│       │   └── signed_refs							│       │   └── signed_refs
│       └── remotes								│       └── remotes
│           └── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe	  |	│           ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│               ├── heads						  |	│           │   └── rad
│               │   ├── cloudhead					  |	│           │       ├── id
│               │   │   └── ci						  |	│           │       ├── ids
│               │   └── master						  |	│           │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
									  >	│           │       ├── self
									  >	│           │       └── signed_refs
									  >	│           └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
│               └── rad								│               └── rad
│                   ├── id							│                   ├── id
│                   ├── ids							│                   ├── ids
│                   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio		│                   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│                   ├── self						  <
│                   └── signed_refs						│                   └── signed_refs
└── hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo					└── hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo
    └── refs									    └── refs
        ├── rad									        ├── rad
        │   ├── id								        │   ├── id
        │   ├── ids								        │   ├── ids
        │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio			        │   │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
        │   └── signed_refs							        │   └── signed_refs
        └── remotes								        └── remotes
            ├── hyddqac7smapegotydiuny3db1i3u16egp4krqaujor6ch5819tadr	  <
            │   └── rad							  <
            │       └── ids						  <
            ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe		            ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
            │   ├── heads						  <
            │   │   └── master						  <
            │   └── rad								            │   └── rad
            │       ├── id							            │       ├── id
            │       ├── ids							            │       ├── ids
            │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio		            │       │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
            │       ├── self							            │       ├── self
            │       └── signed_refs						            │       └── signed_refs
            └── hynkyndc6w3p8urucakobzna7sxwgcqny7xxtw88dtx3pkf7m3nrzc	  |	            └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
                └── rad								                └── rad
                    └── ids						  |	                    ├── id
									  >	                    ├── ids
									  >	                    │   └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
									  >	                    └── signed_refs

98 directories, 35 files						  |	48 directories, 42 files

The text was updated successfully, but these errors were encountered:

cloudhead · 2021-09-03T11:31:41Z

Some additional notes:

It's always the same project that successfully replicates its branches, I've run this 3-4 times with this result
Since the state of the "server" is also replicated (from upstream), I'm not 100% sure that it is in perfect state. But there are no obvious signs that something is wrong with it, and I'm able to read blobs from the repo, through surf.

kim · 2021-09-03T11:38:58Z

Could you maybe compare the contents of the `signed_refs` to the refs which are actually present?

FintanH · 2021-09-05T08:52:17Z

I narrowed some logs down to the first namespace hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto here log.txt.gz.

When I grep'd /hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto.*heads: the heads of the signed refs were always empty. So I think the mystery here is why those heads never get entered into the signed_refs.

FintanH · 2021-09-06T13:06:45Z

When I grep'd /hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto.*heads: the heads of the signed refs were always empty. So I think the mystery here is why those heads never get entered into the signed_refs.

Just realising that this might be the signed_refs of the seed and not the peer with the heads.

cloudhead · 2021-10-06T14:26:27Z

Still seeing this issue with 2df248a FYI

kim · 2021-10-06T15:26:17Z

Unsurprising, that change is unrelated

adaszko · 2021-11-15T11:29:25Z

Could you maybe compare the contents of the signed_refs to the refs which are actually present?

Here's the first error that I'm getting from attempting to replicate the same repo:

Nov 12 15:16:11.771 DEBUG org-node: Replication of rad:git:hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto succeeded: ReplicateResult { updated_tips: {RefLike("refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master"): Oid(acce81a6568ec4342586a71a
Nov 12 15:16:11.773 DEBUG org-node: Setting repository head for rad:git:hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto to "refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master"
Nov 12 15:16:11.773 ERROR org-node: Error setting head in track_peers: failed to set project head: path "/Users/adaszko/tmp/2nd-node/git/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master" does not exist
Nov 12 15:16:11.773 ERROR org-node: Error tracking peers: failed to set project head: path "/Users/adaszko/tmp/2nd-node/git/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master" does not exist

As you're suggesting @kim, I don't see anything wrong with replication up until the contents of signed_refs:

SIGREFS: {
    hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq: Refs {
        heads: {
            OneLevel(
                "master",
            ): Oid(
                acce81a6568ec4342586a71a314485408c264068,
            ),
        },
        rad: {
            OneLevel(
                "id",
            ): Oid(
                c99d99f55cd277bba2187f136051b377b17b56f8,
            ),
        },
        tags: {},
        notes: {},
        remotes: Remotes(
            {
                hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo: Remotes(
                    {
                        hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe: Remotes(
                            {},
                        ),
                    },
                ),
                hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe: Remotes(
                    {},
                ),
            },
        ),
    },
}

When it comes to the file system, it looks like follows after the error (I'm filtering by the relevant identities):

> hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto
  16/184 (0)
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/signed_refs
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/id
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo/rad/signed_refs
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo/rad/id
> refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/signed_refs
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/self
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/id
  logs/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/id
  logs/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio

As can be seen, unsuprisingly, refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master is missing.

OTOH, when we filter by the identity that was present in signed_refs, we find that its heads replicated properly:

> hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq
  20/184 (0)
> refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
  refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
  refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
  refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
  refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
  refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
  refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
  refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
  refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
  refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
  refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
  refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
  refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id

Full logs (with my changes to the logging code unfortunately) are available in a gist. Those are logs from the replicating peer, not the bootstrap peer, BTW.

kim · 2021-11-15T13:26:39Z

Am I reading this correctly that you're expecting heads/master of ...zo645pe to be in the remotes of ...9wgwa1b, but

either ...9wgwa1b doesn't have it in the first place (although it knows about ...zo645pe)
or the fetching node doesn't consider the signed_refs of ...zo645pe?

adaszko · 2021-11-15T14:00:58Z

Sorry if that was a little bit unclear.

Basically, I'm going by the error message: I'm expecting refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master to exist on the file system.

IOW, I'm expecting hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master to be in remotes of hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto.

kim · 2021-11-15T14:08:44Z

Yes yes. What I'm trying to parse is: there seem to be three nodes involved in your setup: 1. the fetching node 2. the remote node 3. the node whose data 1. is interested in So the question is: does 2. have the data of 3.? If not, we need to look into why. If yes, then 1. fails to discover that the data is there.

adaszko · 2021-11-15T14:34:29Z

There are only two nodes :). I'm trying to keep the bug reproduction as small as possible. I'm also using the same repository data as @cloudhead in his examples, so the identities and hashes should mostly match with his comments (except for the replicating node identity, which is different in my setup).

The network topology couldn't be simpler:
1st node: let's call it the bootstrap node
2nd node: let's call it the replicating node

bootstrap node
    |
    |
    |
    v
replicating node

Test scenario:
Everything is happening on a single machine. The bootstrap node serves test data to the replicating node. It is assumed the test data got extracted into ~/.radicle/root.

In a local checkout of the client-services/org-node directory:

Bootstrap node is run like so: cargo run -- --subgraph 'https://api.thegraph.com/subgraphs/name/radicle-dev/radicle-orgs' --rpc-url REDACTED --identity ~/.radicle/identity --root ~/.radicle/root --orgs 0x8152237402e0f194176154c3a6ea1eb99b611482 --listen 127.0.0.1:8776

~/tmp/2nd-node/ is empty except for containing a manually generated identity file:

The replicating node is run like so: cargo run -- --subgraph 'https://api.thegraph.com/subgraphs/name/radicle-dev/radicle-orgs' --rpc-url REDACTED --identity ~/tmp/2nd-node/identity --root ~/tmp/2nd-node --orgs 0x8152237402e0f194176154c3a6ea1eb99b611482 --bootstrap [email protected]:8776 --listen 127.0.0.1:8777

Now, on to answer your question: ~/.radicle/root/git/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master, i.e. the bootstrap repository file does exist on the file system but isn't being replicated to the replicating node repo.

kim · 2021-11-15T14:51:18Z

Aha, thanks.

The signed_refs of the remote you're trying to fetch looks like this:

commit e562431913af9fa8dad3aaa6cea3a57631af100d (refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/signed_refs)
Author: cloudhead <cloudhead@hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe>
Date:   Fri Aug 27 15:30:33 2021 +0200

    Update rad/signed_refs for rad:git:hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto

diff --git a/refs b/refs
index 56e2f01..8a78704 100644
--- a/refs
+++ b/refs
@@ -1 +1 @@
-{"refs":{"heads":{"master":"0bad586e7f0749c70ee33b06eb6882a2523b0b99"},"rad":{"id":"c99d99f55cd277bba2187f136051b377b17b56f8"},"tags":{},"notes":{},"remotes":{}},"signature":"hybrp6ceians1jffsrhaoeu4aayzq4bc1ozh9enqomof3sur5kqh7ke4jamgoqk66rjdtcarww3sgamtq48jpaxxhkgmf84rodsrtx5ob"}
\ No newline at end of file
+{"refs":{"heads":{"cloudhead/master":"0bad586e7f0749c70ee33b06eb6882a2523b0b99","master":"0bad586e7f0749c70ee33b06eb6882a2523b0b99"},"rad":{"id":"c99d99f55cd277bba2187f136051b377b17b56f8"},"tags":{},"notes":{},"remotes":{}},"signature":"hyn37z39mbp3qbs645nxcma5iub4qpk6bozyarw7i31q6ty1ozc8xessaqqnw8qp1sjpxbr1dpqnw66no49j37tqu56fdqiepdoe6oeyd"}

For some reason, it's not heads/master but heads/cloudhead/master. Does the replicating node receive this?

adaszko · 2021-11-15T14:56:03Z

Nope, heads/cloudhead/master is missing:

Filtering the file system tree on the replicating node side by hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe

gives:

  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/signed_refs
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/self
  refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/id

kim · 2021-11-15T15:40:16Z

Hm ok. Something seems a little off in the logs vs. the repository state you posted, so it's hard to make sense of it. Could you share the key that was used for the bootstrap node so I could run this scenario myself?

radicle-dev/radicle-link#782

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko · 2021-11-26T16:10:09Z

@kim I've made the reproduction smaller by removing two repos and I've created an end-to-end test case using Python at radicle-dev/radicle-client-services#71. Instruction on how to use are in the README file. I hope it makes it easier to debug this issue and also serves as a regression suite for the future.

geigerzaehler mentioned this issue Sep 28, 2021

"waiting room error" spam in logs after cloning a project radicle-dev/radicle-upstream#2327

Closed

rudolfs mentioned this issue Oct 1, 2021

Project replication not working radicle-dev/radicle-upstream#2449

Closed

geigerzaehler mentioned this issue Nov 16, 2021

Can't find default branch radicle-dev/radicle-upstream#2585

Closed

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 22, 2021

GitHub issue #782 reproduction

34c3e4d

radicle-dev/radicle-link#782

adaszko mentioned this issue Nov 22, 2021

Reproduce #782 as an end-to-end test case radicle-dev/radicle-client-services#71

Closed

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 22, 2021

GitHub issue #782 reproduction

6b705b9

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 22, 2021

GitHub issue #782 reproduction

f27d630

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

cloudhead mentioned this issue Nov 23, 2021

[org-node] Bug: Failed to send update refs request to client: receive failed radicle-dev/radicle-client-services#70

Closed

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 26, 2021

GitHub issue #782 reproduction

910e1a1

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 26, 2021

GitHub issue #782 reproduction

1cb0632

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 26, 2021

GitHub issue #782 reproduction

4f5dc67

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 26, 2021

GitHub issue #782 reproduction

398e99a

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 26, 2021

GitHub issue #782 reproduction

70bdba7

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

adaszko added a commit to adaszko/radicle-client-services that referenced this issue Nov 26, 2021

GitHub issue #782 reproduction

f37b3a3

radicle-dev/radicle-link#782 Signed-off-by: Adam Szkoda <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Branches sometimes not replicated on fetch #782

Branches sometimes not replicated on fetch #782

cloudhead commented Sep 3, 2021 •

edited

Loading

cloudhead commented Sep 3, 2021

kim commented Sep 3, 2021 via email

FintanH commented Sep 5, 2021

FintanH commented Sep 6, 2021

cloudhead commented Oct 6, 2021 •

edited

Loading

kim commented Oct 6, 2021

adaszko commented Nov 15, 2021 •

edited

Loading

kim commented Nov 15, 2021

adaszko commented Nov 15, 2021

kim commented Nov 15, 2021 via email

adaszko commented Nov 15, 2021

kim commented Nov 15, 2021

adaszko commented Nov 15, 2021

kim commented Nov 15, 2021 via email

adaszko commented Nov 26, 2021

Branches sometimes not replicated on fetch #782

Branches sometimes not replicated on fetch #782

Comments

cloudhead commented Sep 3, 2021 • edited Loading

cloudhead commented Sep 3, 2021

kim commented Sep 3, 2021 via email

FintanH commented Sep 5, 2021

FintanH commented Sep 6, 2021

cloudhead commented Oct 6, 2021 • edited Loading

kim commented Oct 6, 2021

adaszko commented Nov 15, 2021 • edited Loading

kim commented Nov 15, 2021

adaszko commented Nov 15, 2021

kim commented Nov 15, 2021 via email

adaszko commented Nov 15, 2021

kim commented Nov 15, 2021

adaszko commented Nov 15, 2021

kim commented Nov 15, 2021 via email

adaszko commented Nov 26, 2021

cloudhead commented Sep 3, 2021 •

edited

Loading

cloudhead commented Oct 6, 2021 •

edited

Loading

adaszko commented Nov 15, 2021 •

edited

Loading