"ord --index-sats index" is incredibly slow

[so7ow]

Running on macOS 13.2. M1 Max CPU, ord data on fast internal SSD. Running ord 0.5.0 with --index-sats seems to start fast but then slows exponentially; seems like it is on track to take multiple days or more, hard to know how slow it will be toward the end. Any way to make this any faster?

bitcoin.conf:

txindex=1
rest=1
server=1

[raphjaph]

--index-sats can take up to 2 days, even on an M1 Max. The underlying reason is that the fragmentation of the sat ranges increases over time. We're still looking into what exactly the bottleneck is (memory or rpc calls) but at the moment there isn't anything you can do about it.

There are some open PR we haven't merged yet (#1516 and #1636), that people are claiming helps so you could pull those and try indexing with them.

[so7ow]

There are some open PR we haven't merged yet (#1516 and #1636), that people are claiming helps so you could pull those and try indexing with them.

Yeah, those don't seem to apply to --index-sats.

[andrewtoth]

We're still looking into what exactly the bottleneck is (memory or rpc calls)

With #1516 --index-sats uses zero RPC calls. The bottleneck I think is disk IO and memory. @raphjaph Any thoughts on https://github.com/casey/ord/issues/1630?

[raphjaph]

We're still looking into what exactly the bottleneck is (memory or rpc calls)

With #1516 --index-sats uses zero RPC calls. The bottleneck I think is disk IO and memory. @raphjaph Any thoughts on #1630?

Answered on that issue

[so7ow]

The bottleneck I think is disk IO and memory.

From an end-user perspective, and based on a cursory examination of Activity Monitor in macOS I think it's disk IO. I'm 30-some hours into a --index-sats indexing and I see the ord process using:

CPU: 50-70%
RAM: 2 GB of 32 GB available
Disk writes: 1.73 TB (!!)
Disk reads: 450 GB

[so7ow]

So after hearing of a Windows user with a substantially similarly specced machine who finished the whole --index-sats process in less than a day, I decided to do some rough math. I'm clocking about 750 blocks/hour in the 475,000 block range. So I have about 300,000 blocks left to index. My napkin math says I have over 400 hours to go, and that's assuming the speed stays constant from here, which doesn't seem to be the case. So that's over two weeks.... I don't think this is working quite right!

[andrewtoth]

What kind of hard drive are you using?

[so7ow]

ord data is on the built-in SSD. bitcoin data on USB-C SSD.

[Prestonsr]

Just searched the repo for tb to see if anyone else had noticed this very same thing. I restarted the indexing process (not the --index-sats index, the regular Ord index.) at around 450k blocks and its now at 763k. Just in that block range I have 1 TB written.

[andrewtoth]

@Prestonsr for a normal sync can you please help test/benchmark my improvement branch? https://github.com/casey/ord/issues/1648#issuecomment-1426565987

Myself and some testers have managed to sync with it under an hour.

[veryordinally]

I observed the slowdown on Mac back in November, after I had sped up the indexing. There does seem to be a significant factor on Macs making indexing a lot slower than on somewhat comparable (in terms of RAM, disk, CPU speed) Linux (or even Windows) machines. May be related to unsolved #819

[veryordinally]

Trying to pin down regression. PR https://github.com/casey/ord/pull/703 is my last confirmed reference point before this performance regression on Mac.

[so7ow]

If there's anything I can do to help with testing I'm happy to. I've aborted my long-running --index-sats attempt for now.

[DaWe35]

It takes weeks for me on a VPS.

[andrewtoth]

@so7ow I managed to sync with --index-sats in just under 24 hours. Check out my branch https://github.com/andrewtoth/ord/tree/skip-values.

[so7ow]

@so7ow I managed to sync with --index-sats in just under 24 hours. Check out my branch https://github.com/andrewtoth/ord/tree/skip-values.

So, I'm a little git-illiterate. Would this be the correct process to build?

git clone https://github.com/andrewtoth/ord
cd ord
git checkout skip-values
cargo build --release

[andrewtoth]

Looks right to me.

[so7ow]

Ok, I have it running now. Went pretty quickly up to block 250,000 or so. Now appears to have slowed radically and still chugging toward 300,000. I'll let it run for now but if it gets to more than say ~2 days I'm going to kill it again!

[andrewtoth]

Lol it's been 3 hours. Sounds like it's making good progress.

[so7ow]

Sorry, I'm just jaded from my previous run! 🫠

[andrewtoth]

@so7ow how is it looking today?

[so7ow]

I'm approaching block 500,000 after ~21 hours.

[andrewtoth]

Ahh dang. I was well into 600ks after 12 hours. You sure you are on the right commit?

[so7ow]

Pretty sure....I shared my steps above and it compiled cleanly. Is there a git command I can run to show you where I am?

ord % git branch
master

• skip-values

[andrewtoth]

ok... well was this further than you got after 21 hours last time at least?

Also, thank you for testing!

[so7ow]

Oh yes, it's definitely faster. I didn't really track very carefully so I can't really quantify but up above I had been running for at least a full day and I was only up to around 475000. Progress! I'm just afraid it's going to slow exponentially toward the end and still take a week. But I'll try to be patient. :)

[so7ow]

@andrewtoth I'm at ~33 hours and working through the 535,000s now.

[so7ow]

@andrewtoth ~44 hours and we're in the 585,000s.

[so7ow]

~56 hours - 620,000 blocks and counting

[andrewtoth]

Dang. Now's the hard part 😬

[so7ow]

Are you invested in seeing this finish? I'm not--but I'll leave it running if you want me to. We've pretty well established that whatever you fixed to dramatically speed this up on your platform didn't really carry over to Mac, right?

[andrewtoth]

Yeah well we've determined it is faster somewhat, but still probably will take close to a week. If you don't need the index for yourself I don't really need to see it finished. But thanks again.

[so7ow]

Ok, I'm stopping it!

[so7ow]

Just a quick update that excessively slow sats indexing on Mac continues for me in ord 0.8.0, even as default indexing continues to be very fast on the same machine/environment.