Error when uploading file observables #1089
Comments
|
There is a problem with filenames having unicode-characters in it. The problem with these is, they can be uploaded, but the download link is broken. You're talking about upload, right? What are the access rights on /tmp? Can the user thehive write on /tmp? And would you please post the code you use to upload your file? |
|
Hi, thanks for replying :) Yes i am talking about upload. I upload files using TheHive´s webinterface and (for testing) the Sample Code from TheHive4py: https://github.com/TheHive-Project/TheHive4py/blob/master/samples/test-case-observable.py (with my API Key) /tmp Permissions are 777 and Thehive, Cortex and elasticsearch all own files in /tmp so i assumed writing there is not the problem. When using the webinterface for observable uploading it shows this error after some time: |
|
This is what it looks for my installation: Can you stop TheHive, remove thehive's play directory (not the one owned by cortex), and restart thehive? If you have SELinux active, disable it (at least for troubleshooting). One potential source of problems less. Do you have The interesting part is the result of each system call. If there is an error, you will see it. |
|
After typing service thehive stop it took a ridiculous amount of time for thehive to stop (like 1min), in case that is interesting. Here is the content of the strace.out after a failed upload using the webinterface: |
|
Oh, wow. There is really a problem - but not with the file upload, but with a crashing process: Which javaVM do you use ( |
|
openjdk full version "11.0.4+11-post-Debian-1deb10u1" |
|
I run Do you have the chance to change the JVM, for example against Oracle`s? Ah, and I did not explain what happens: SIGSEGV is a segmentation violation error - a process tried to access memory out out it's range. |
|
Yes i have root access on the machine and i dont know of any service running, that would care when i change the JVM, so i could change it. Do you have the necessary command ready? Else i ll just look it up really quick. Do you know how this JVM got on my system? Because i did not knowingly install it, so it must have been part of the Debian ISO or (wich would be weird because it seems to break TheHive) of TheHives installation process. |
|
Oh and take your time i am very greatfull that you are helping me out and i am in no hurry :) |
|
No, I don't have the commands right at hand. Basically, download Oracle's JVM Standard Edition, install it, make You could also downgrade your openJDK to 11.0.3. I'm very confident the JVM is guilty ... PS, I'm curious. Where are you located? |
|
Ok, i ll try that and will let you know if it helped :) |
|
Mainz. :-) |
|
Ok i am having a hard time downgrading the openjdk (older packages are nowhere to be found). I ll take a quick break. By the way: I checked the other Debian 10 VM that i have (the one everything is working on) and it is using the same java-version. Does that mean, that the javavm is not guilty after all? (The failing one is running on different hardware and was set up with a different installer file, could those differences cause the JVM to fail on the one and work fine on the other?) |
|
I think the problem comes from the JVM - but I'm not sure. The point is, the JVM executes the Java Programs running on the system. If you encounter Segmentation Faults, I assume it's the JVM. Another possibility would be that TheHive calls an external program, and that crashes. I'll look how to trace this with Strange enough you have a running system on another host. So what's the difference? do you know if you have corefiles/crashdumps of the dying processes? |
|
I managed to install the oracle JVM but the is still coming up. |
|
Hm. Strange. Sorry I could'nt help with a successful result. If I have another idea, I'll let you know. |
|
Thanks a lot for your time! :) At least i learned quite a lote while trying to fix this^^ |
|
So i gave it another shot and set up a completely new machine on the same esxi server and another one on a older esxi server and both were again facing the same problem. I reopened this issue, so someone with an esxi server can maybe reproduce the problem and look into it. also added this information to the description: I will collect some more information and post it here soon. |
|
I do have a private ESXi (6.5.0) here, so I could verify your results. Can you please summarize your machine settings (#CPUs, Memory, OS-Version, Hive-Version, ElasticSearch-Version)? If I find the time the next few evenings, I could setup a similar host. PS: I'm glad I seem to have been right blaming the JVM :-) |
|
@saadkadhi : this thread here might be interesting for you to follow. |
|
Oh thats great i will give you the interesting settings i see in the vsphere client:
The OS Version ist Debian 10 ISO downloaded from here: TheHive, Elasticsearch, Cortex etc. versions are the latest or what you get when you follow this installation guides: Right now this results in: Note that i also installed Cortex on every attempt i made even though it probably does not take part in the scenario. |
|
Awesome. And a great collection of information on how to setup the confirmation equipment. I'll do my best to setup a similar setup. |
|
Great! Let me know if you need anything else. P.S. I am also glad about you finding the JVM quickfix^^ this way i can keep working with TheHive for now. |
|
Oh maybe i should mention this little script again in order to quickly test fieluploads multiple times: |
|
I have set up a ESXi 6.5 VM with Debian 10 x64, openjdk full version "11.0.4+11-post-Debian-1deb10u1", and successfully uploaded a file in a new case. Setup: TheHive 3.3.1-1 Unfortunatly, I dont have ESXi 6.7 available. Did you problem happen always or sometimes? Did you upload that many files to one single case? What kind of files? |
|
OK, my questions were already answered. I will throw files at my installation using the test script you mentioned, and report back. Give me some time, I have to do this at home. |
|
Problem confirmed My test setup had trouble creating an observable and dropped the error message: Investigation goes on. |
|
Hi, sorry for replying so late i was kinda busy the last days. The errormessage you mention in your last post is the exact same that i get! So at least we now can reproduce the problem :) |
|
I was able to trigger the same problem on Ubuntu 18.04.2 LTS with TheHive 3.4.0-RC2 on a KVM virtualized host. It took 108 runs of your above script. My log message: The system calls from the JVM say: To summarize what I know:
@saadkadhi, @nadouani : if you want access to a system which quite fast and reliably triggers the problem, I can arrange this. |
|
Hello, we probably have the same problem. Importing a set of files into TheHive leads to an error at some point. Version: 3.4.0-RC1 and 3.4.0 hive/application.log: |
|
Hello, |
|
Hello @thofm and @cyberpescadito, as mentioned above the current workaround is to switch from openjdk to oracle. |
Error when uploading file observables
Request Type
Bug
Work Environment
What seems to also be important is:
The Debian 10 Machine is a VM on an ESXi version 6.7.0.30000
When setting up another machine on my local computer using VMWare-Workstation 15 Pro the exact same way it works fine.
Installing java full version "1.8.0_221-b11" (oracle Java 8) seems to fix the issue (it did not appear again since ~300 fileuploads)
Details on the VM-Settings: #1089 (comment)
From TheHives about page:
TheHive 3.3.1-1
Elastic4Play 1.10.0
Play 2.6.21
Elastic4s 5.6.6
ElasticSearch 5.6.9
Problem Description
I have set up a new TheHive instance, following the installation guide on a Debian 10 machine. I have done this multiple times before (a few times on Debian 10 even) and it always worked.
Now yesterday i set up another Debian 10 VM and and noticed, that adding file observables (via the Webinterface and via the API) sometimes results in an Error. (errorlogs below)
Whats confusing me the most is that it will sometimes work, so i guess (i also doublechecked) my configs are all fine.
Steps to Reproduce
This is kinda hard to say since i have no clue what is suddenly causing this. I have another Debian 10 machine, set up the exact same way, where its working properly. Maybe someone has had this issue before or understands the errorlogs better than i do.
Complementary information
This is the /var/log/thehive/application.log entry:
Error during preStart in [FileSource(/tmp/playtemp13354541987563218898/multipartBody15791114881836088669asTemporaryFile, 8192)]: /tmp/playtemp13354541987563218898/multipartBody15791114881836088669asTemporaryFile
java.nio.file.NoSuchFileException: /tmp/playtemp13354541987563218898/multipartBody15791114881836088669asTemporaryFile
at akka.stream.impl.io.FileSource$$anon$2.preStart(IOSources.scala:74)
at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:295)
at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:557)
at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:679)
at akka.stream.impl.fusing.ActorGraphInterpreter.preStart(ActorGraphInterpreter.scala:727)
at akka.actor.Actor.aroundPreStart(Actor.scala:528)
at akka.actor.Actor.aroundPreStart$(Actor.scala:528)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart(ActorGraphInterpreter.scala:670)
at akka.actor.ActorCell.create(ActorCell.scala:652)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:523)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:545)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:283)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2019-08-01 13:55:44,828 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-5 - POST /api/case/AWxNCS0IA69JQvBt7AXt/artifact returned 500
java.nio.file.NoSuchFileException: /tmp/playtemp13354541987563218898/multipartBody15791114881836088669asTemporaryFile
at akka.stream.impl.io.FileSource$$anon$2.preStart(IOSources.scala:74)
at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:295)
at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:557)
at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:679)
at akka.stream.impl.fusing.ActorGraphInterpreter.preStart(ActorGraphInterpreter.scala:727)
at akka.actor.Actor.aroundPreStart(Actor.scala:528)
at akka.actor.Actor.aroundPreStart$(Actor.scala:528)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart(ActorGraphInterpreter.scala:670)
at akka.actor.ActorCell.create(ActorCell.scala:652)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:523)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:545)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:283)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
I hope someone has an idea, whats causing this.
The text was updated successfully, but these errors were encountered: