Rapiddisk does not seem to be improving performance?

[8465231]

So I got it all setup and working. I am curious about performance though, When do you plan to release the performance video on youtube? I would love to see the expected results.

I tried the writeback mode which would seem to give me ramdisk performance until I read/write more data then would fit on the ramdisk (at risk of data loss of course)? The ramdisk is 256gb in this test and I am using 10gb tests.

The first test gives only slightly improved write speeds and access times, and stock read speeds.

The 2nd test gives full ramdisk write speeds and ~50% ramdisk read speeds and access times are still only mildly improved?

I don't understand why the 2nd test would have faster write speeds? Also why are the read speeds and access times so slow in comparison?

Next I tied the write-through. The first test was stock read speeds with slightly improved write speeds, stock access times. 2nd test was marginally improved read speed, same write speed but massively improved access times. Not sure how the access times are so much better but the read speeds are only marginally improved?

Am I doing something wrong? Is there something I can tweak?

My goal is to get ramdisk performance that is automatically flushed to disk in the background, I am not worried about data loss in the amount of time that it would take to write to disk.

[pkoutoupis]

First of all, thank you very much for your interest.

I hope to have something created/uploaded in the near future. I do not have a schedule ATM.

Re: writeback...
Can you please describe your first and second tests in more detail? And maybe even provide original disk numbers with the comparison to the rapiddisk enabled volumes.

Re: write-through...
The first test makes sense because the data needs to be staged in RAM for future reads. The improved writes may just be coincidental and within a margin of variation, so they may just seem improved but really aren't. Anyway, both write-through and write-around will ONLY improve read speeds (and not write) only if there is a cache hit. Meaning that if the data is already staged in RAM, you should see an improvement with reading that dataset. If it isn't staged, it will hit the backend target and your reads will not be improved.

Re: access times...
These are typically meta-data read operations and if the meta data is staged in memory, then yes, you should see an improvement.

All of this depends on your workload. Can you please share these details so we can figure things out and if RapidDisk is the best tool for your environment? Thank you.

[8465231]

Thanks for responding!

I have done a bit more testing since yesterday, figure out the write improvements were due to a config issue with the raid array, it is solved now so no longer seeing the write improvements.

Here are the test results with the write-through setup, first is the bare array:

and here is with the rapiddisk WT attached, this is the 2nd time running the test:

This gives basically the performance I would expect from write-through given what you said above except for that odd reduction in the 2nd write speed test. I ran the test a few times and the results were consistent.

It is odd that the speed are still only about ~40-50% of the raw ramdisk though?

My use case is basically disposable datasets used for deep learning and that type of thing, it is replaceable so write-back is the ideal solution I am looking for. Get ramdrive performance until the ramdrive is full, in my case ~80% of datasets should fit within the ramdrive but need the flexibility of it overflowing to the drives should the dataset grow beyond that.

Although I ran into an issue when trying to run the tests on the WB rapiddisk setup, it refused to allow me to mount the device, I could use the raw block device but could not mount it. I ran out of time so I was not able to troubleshoot it further. Could just need a reboot after all the stuff I tried, plan to mess with it later today again.

[pkoutoupis]

OK. I understand now. A lot of the performance impact (for both reads and writes, using WT mode) when utilizing the RapidDisk-Cache module is table look ups and determining what is already in cache and what is not. This can come at a cost of multiple read requests (ironically, in memory). And let us not forget that the underlying I/O operation still needs to process through the device-mapper framework which consists of additional block I/O requests to the underlying devices (which in turn, process their own). The caching component will never give you native volume speed because of how the subsystems and modules work. Anyway, does that explain, 50%? I do not know without profiling the workload and the subsystems.

As for the WB issue, I will need additional information but for now, I am sure that the reboot will address it.

[8465231]

Ok, that makes some sense then.

I was trying the WB mode again after a reboot and was still not able to mount it, I then tried formatting the drive while the WB was enabled and then it allowed me to mount it after that. No idea what is going on there. I will mess around with it some more and see if it happens again.

So the first test went basically as I hoped:

I then tried testing the raw block device as I had been up until this point and that is when I got the strange results I had seen before, it seems to react differently to the block device vs the file system.

Gonna run some FIO tests next to see how those pan out, although if these numbers translate into improves iops as well I think we have a winner!

Is there anyway to adjust the writeback settings? How long does it hold data in the ram cache?

[pkoutoupis]

Hmmm. Not sure. Note that I am relying on dm_writecache from the Linux kernel for writeback support. Tuning this out-of-box would require some feature additions in the rapiddisk userspace code, which is planned but I do not have an immediate date or timeframe for it. The data should remain in memory unless the following conditions are met:

1. The data in a specific location is updated. The updated data is staged in the cache.
2. The data is flushed by the user.
3. The mapping is removed.
4. The cache volume is running out of space and room needs to be made for newer data.

OH, dm-writecache ALSO does not cache data that it detects as sequential and possibly block transfer sizes larger than a certain size (I need to double check the latter). So, you will not notice a performance boost if you are not writing small random writes.

And for setting adjustments, the LVM2 suite does support dm-writecache and the ability to tune it. Just be sure to not confuse it with dm-cache.

[8465231]

Thanks for the information. Ok, so back messing with this again.

I looked into LVM and actually ended up deciding that combining it with rapiddisk might be the best option. Using LVM to handle the tiered storage from the slower spinning drives and the faster SSD's but it does not appear to support ram caching.

Tried stacking rapiddisk on top of the LVM and at first glance it seemed to work great! but then I noticed odd behavior.

If I run kdiskmark on the mounted filesystem the results are as expected but if I run FIO on the mounted filesystem it seems to completely ignore the rapiddisk? Is there a way to force the FIO to use the rapiddisk? Same with dd, it seems to ignore the rapiddisk cache?

Although before if I did either test directly to the rapiddisk block device they seem to work?

[8465231]

hmm, I think I might of found part of the issue, I had the LVM cache setup in writeback mode as well, and that was causing issues it seems, if I switched the LVM cache to writethrough and then rapiddisk to writeback it seems to be working with FIO.

I need to do more testing but out of time for today.

[pkoutoupis]

Thank you for:

1. Your interest in this project.
2. For working through your issues.
3. And for keeping me posted on your progress.

All of which are much appreciated.

[8465231]

Back playing with rapiddisk again lol.

I am curious, why is the performance of rapiddisk so slow? Even on a high end 64 core epyc system with 8 channel memory the most I can seem to get is around 5-7GB/sec and half that with it actually caching another drive.

Whereas with just NVME drives I can get upwards of 7-9GB/s speeds WITH a filesystem in place? The un-cached NVME drives are actually faster except for 4k random performance.

Is this normal?

[pkoutoupis]

Are you using writeback, writethrough or writearound? And how is the whole thing configured and with which tools?

[8465231]

Have tried both WB and WT,

It is configured several ways during testing but all had the same results. If I raided the NVME drives be it LVM or mdadm the performance basically added up minus overhead. so 3-4gb/s for 1, 6-8gb/s for 2 etc.

Testing was done with a bunch of tools, once again they basically all agreed, FIO, DD, kdiskmark.

Rapiddisk on the other hand seems to have a hard limit at around 2-4gb/s, nothing I tried could make it go faster then this.

I was reading and someone mentioned that having a dynamic size ramdisk causes a lot of overhead and pre-alocating the space can improve performance but not sure that is possible with rapiddisk.

[pkoutoupis]

While yes, there is a bit of overhead because the RAM drive is allocating page memory on demand (as it is being requested), I am really curious about this comment:

Rapiddisk on the other hand seems to have a hard limit at around 2-4gb/s, nothing I tried could make it go faster then this.

Is this a standalone RAM drive or a mapped volume?

Also, where did you read this?

I was reading and someone mentioned that having a dynamic size ramdisk causes a lot of overhead and pre-alocating the space can improve performance but not sure that is possible with rapiddisk.

[8465231]

Is this a standalone RAM drive or a mapped volume?

I have seen the same results in all my testing, on my old test system with dual xeons I figured it was just the DDR3 limiting performance, I tested both a cached volume and direct to the block device there for sure. The block device was only a bit faster IIRC.

On the new epyc system I figure the bottleneck would be removed and while it is faster (roughly 50% faster IIRC) it is still much slower then I expected a ramdrive to be.

Although on the new epyc system I had trouble testing the raw block device IIRC but I didn't try very hard either.

Overall I was just super surprised that the ramdrive was slower then a gen 4 NVME drive, much less a raid of them (except 4k, the ramdrive is faster on those of course).

Sad part is that I think I have the setup pretty much working exactly as I hoped now, just limited to these speeds:

don't have a shot of the NVME but they get around 4-6gb/s each and add up good when raided.

Only possible issue I am running into with the ram drive (still using it for the time being, experimenting with removing the NVME from the system and just using some spinners as the backing device) is that it is possibly kicking blocks out of the ram cache long before it is full but need to do more testing.

Also, where did you read this?

I read it somewhere in my hours of searching google about this subject, sorry but I have no idea where exactly but the source seemed fairly credible, like he knew what he was talking about. He was saying that some program that is used to test memory bandwidth allocates all the memory prior to running any tests as it leads to significantly improved results.

He had some other details but it went over my head. Long and short that I got out of it was that the the request for the space in a dynamic memory space is one of the most time consuming parts of the operation and pre-allocating the space can significantly improve performance.

There is something at play for sure though, this test system should be capable of somewhere north of 150GB/s bandwidth to the memory, some overhead is perfectly understandable but topping out at ~3GB/s? Seems like there is something more significant going on.

What kind of performance do you see when testing rapiddisk? Do you get better results?

[pkoutoupis]

Thank you for this information. Yes, a general file system adds a ton of overhead. It is a shame that you had a difficult time rerunning the test on the raw block device.

Also, this thread may provide some more information: #118

[8465231]

Thanks for that link, looks like my numbers are somewhat inline with others except that one guy getting 8gb/s+.

Just find it interesting that even ramdrives can not match NVME drives in bandwidth, I expected it to put the NVME to shame. lol

[pkoutoupis]

@8465231 It just dawned on me that you should be seeing performance improvement if you modify variables from the default like below during module insertion:

# modprobe rapiddisk max_sectors=2048 nr_requests=1024

[8465231]

Interesting, I will check that out when the machine comes down for maintenance then, it was put into service yesterday lol.

I did not see those variables mentioned anywhere previously.

[pkoutoupis]

Another tunable to improve write performance:

# echo "write through" > /sys/devices/virtual/block/rd0/queue/write_cache

I may end up making this default with the module because with my raw device testing using fio, it shows a significant improvement with random DIO writes.

[8465231]

ah, ok that makes more sense

[pkoutoupis]

Let me fix that statement: inherits values are just passed certain functions through to the underlying devices.

[8465231]

ok, I will give these options a try when we do some maintenance on this machine and see what the results are!

[pkoutoupis]

I also noticed that initial page allocation (initial writes) does sometimes perform a bit slower than rewrites (after the memory pages have been allocated). It would make sense (and is related to the note you found on the Internet, posted above). I wonder if you yourself see that difference in your setup. That is, do an initial write run after RAM drive creation and then repeat the same test.

[8465231]

I also noticed that initial page allocation (initial writes) does sometimes perform a bit slower than rewrites (after the memory pages have been allocated). It would make sense (and is related to the note you found on the Internet, posted above). I wonder if you yourself see that difference in your setup. That is, do an initial write run after RAM drive creation and then repeat the same test.

Yes, I noticed the same thing, could not figure out exactly why but that makes sense now that I think about it since the memory would already be allocated. Also explain the inconsistent results as sometimes the re-write would not be any faster but that was generally a different test that would of needed new memory space allocated.

[pkoutoupis]

@8465231
Hello. I know it has been a while and I am not entirely sure if you are still interested but back when you were first looking into this, it was not clear on whether you were building from master or a tagged release. If you were clone and building from master, then it is likely that you were affected by a performance regression that recently discovered. If you clone from master today or download release tag 9.1.0, this regression has been fixed.