Log filled with Caused by: java.io.IOException: No space left on device errors on data drive free space consumption
#20354
Comments
b1naryth1ef
changed the title
Caused by: java.io.IOException: No space left on device errorsCaused by: java.io.IOException: No space left on device errors on data drive free space consumption
clintongormley
added
feedback_needed
:Core/Infra/Logging
|
@b1naryth1ef What do you think we should do instead? |
|
Generally I would expect any stable, reliable production database would trigger a few log lines indicating whats happening, and what state the database is entering. |
|
I expect it to be able to come up enough for me to be able to post a delete command Or is that too much to ask of a database? How am I supposed to be able to remove the "bad" indices if a filled up the drive causes a DATABASE to refuse to start itself? In what world is this sane behaviour? Can you describe a production database which touts around a major version of 5.x that refuses to start with the following beauty of a log, and is not considered as a total joke? I'm done with this crap. |
|
No useful suggestions presented - closing |
|
@clintongormley what exactly are you looking for in the form of suggestions? Might I offer the following solutions:
|
|
Wait what. In what world of production databases do people do this shit |
|
I think the underlying issue is that, on an exception that clear is saying "stop trying to use disk space"..... You're still doing something that is attempting to use Disk Space. While I would assume that that is a pretty obvious evaluation, I would further more assume that the course of action is also pretty obvious: this is not an exception that might go away the next time the function is ran, something is critically wrong in the environment, and this occurrence should be treated as such. I won't pretend to be an expert in enterprise-grade software, but I would assume that critically failures in needed parts of the runtime environment would warrant a crash. Can't exactly keep hopping when the ground has been taken from under you, so why bother? |
|
The disk that logs are written to is not necessarily the same that data is written to (which is where the exception above comes from). Are you suggesting we should not try to log/output anything when there is a critical exception causing the system to exit? If you have a concrete suggestion for cleaner behavior, please open a new issue detailing the feature request. |
|
You could just make the logs not spammy. Kill the process, and just simply say that it crashed because of no disk space |
If the log writing was the source of the OOS Exception, b1nzy wouldn't have a 28GB log file filled with the exception spam. I would do as NullPixel said, and I said in my second paragraph. If the program is encountering an issue that clearly means it's failing to operate properly, meaning, cannot be recovered while the process is live, you don't continue to operate. I understand that enterprise-grade software needs to be reliable, but this is the result of a problem outside the scope of the software, that cannot be recovered from while the program is running. What else is there to do? Besides spamming the logs? |
|
Got sick of this being left, opened #24299 |
Elasticsearch version: 2.3.2
Plugins installed: []
JVM version: Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
OS version: Ubuntu 14.04.5 LTS (3.19.0-59-generic)
Description of the problem including expected versus actual behavior:
Just had our data drive run out of space on a 1-node elastic instance which caused the logging partition to be filled by a 28GB log file (~300 million lines), filled solely with this repeating section:
We've had this instances data drive fill up before without this issue being raised.
Steps to reproduce:
The text was updated successfully, but these errors were encountered: