[leo_storage][data-compaction] Avoids write operation before leo_storage is not able to execute data-compaction

[yosukehara]

We already delivered auto-compaction feature, and recommend using this feature but in case of its disabled, LeoFS' storage node crashes when disk-full situation happen.

We need to support a LeoFS' storage node reject write-operation before its node reached not able to compaction situation as below:

After stoping WRITE operation, an admin needs to maintain a stoped leo_storage:

• Administration Options:

  • Expands disk space
  • Moves data-files to somewhere, needs to reconfigure obj_containers.path

• Related issue

  • [Check before Compaction] [leo_storage] Consider checking/reserving disk space for compaction #465
  • [Operation Manual] [docs] Manual for node recovery when disk is full #464

[mocchira]

WIP

[yosukehara]

@mocchira We've benchmarked v1.3.7-dev today. Its result is not good, Benchmark LeoFS v1.3.7. We need to reconsider this implementation. So I would like to propose this implementation as below:

Proposal

• One monitor process, leo_object_storage_diskspace_mon (tentative name) checks disk-space every one minute which is configurable
• In case of over the threshold, leo_object_storage_diskspace_mon sets a flag of is_able_to_write to leo_object_storage_server's 'State'
• leo_object_storage_server's put and store always checks is_able_to_write to reject storing an object.

[mocchira]

@yosukehara Confirmed. It seems disksup gen_server process might be bottleneck as multiple leo_object_storage_server (default: 8) call that gen_server each time and also each call cause callers to be locked exclusively (waiting for other calls to finish). so your proposal loos reasonable.

[yosukehara]

@mocchira

Confirmed. It seems disksup gen_server process might be the bottleneck as multiple leo_object_storage_server (default: 8) call that gen_server each time and also each call cause callers to be locked exclusively (waiting for other calls to finish). so your proposal looks reasonable.

Thanks for your confirmation. I'm going to send a PR soon.

[yosukehara]

We've recognized this issue was fixed:

• Benchmark LeoFS v1.3.7