replication-offset-checkpoint.tmp (No such file or directory) #194
Comments
|
Doesn't that mean that you have a low |
|
No, that's not it. I tried to increase it, but still got same problem. |
|
Well, share, please, as much info then as possible: test-case to play from our side, how much concurrent tests you have, aren't there some process on that server which clean the Looks like it is more Apache Kafka issue than Spring Kafka... |
|
I've noticed that this issue occures when Jenkins builds two microservices simultaneously (and both are using Kafka Test). I think first build cleans up and second one cannot access it. Question - is there a way to customize /tmp dir name? We need to have possibility building services simultaneously. |
|
As I said before: it is the question directly to Apache Kafka. I'm not familiar with Scala, so can't debug and interpret what they do there... |
|
It seems unlikely to be related to concurrent builds; in my experience, the Yeah - it uses |
|
Isn't ReplicaManager.checkpointHighWatermarks() method clearing all available replicas? I think there's possibility to remove /tmp/kafka-n by other thread. @see core/src/main/scala/kafka/server/ReplicaManager.scala |
|
I still don't understand why do you ask that question to us, Spring community when you see the problem in the Apache Kafka directly? If you insist it is Sorry, but it looks like we (at least me) are useless for your on the topic and I don't understand why do you spend time with us not Apache Kafka community? Would be glad to see some cross-link from there to widen knowledge in this Kafka topic. Thanks |
|
Right now I just mocked ReplicaManager.shutdown() method to return false. Thanks for you comments. |
|
I have found this thread via google looking for solution. Did not find it here, but this is the only place which refers to problem I encountered.
If kafka broker runs at the end of unit test it will attempt to write/read data in a dir which is deleted and produces different FileNotFound exceptions. Solution: shutdown embedded kafka broker at the end of the test before System.exit is called. If KafkaEmbedded rule is used properly it will call KafkaEmbedded#after method, which destroys broker before System#exit is called. I use KafkaEmbedded class in Spring integration test and create it as bean. Unfortunately spring context is destroyed in shutdown hook as well and it happens concurrently with other shutdown hooks, so kafka log directory is destroyed before embedded kafka broker is down. I did not find proper solution yet for this usage scenario. |
|
@hoaz , thank you for your report.
Well, it would be better to stop writing in the end of test, e.g. call |
|
Because I have multiple integration tests which share the same Spring test context. I do not want to bring up and shut down kafka broker in each test but rather delegate it to Spring, which caches spring context between tests. |
|
Would you mind to share the code how do you do that? So, placing the |
|
OK, this is a valid solution, there is only one thing I do not like about it. It would be really good if spring-kafka-test could support embedded kafka as a bean, especially taking into account project name :) Here is how we run embedded kafka broker inside of spring container: |
|
Good, thank you for sharing that and for your ideas! So, doesn't it mean that that And what we get there is only some ERROR in logs? |
|
Yes, Spring context is being closed in parallel with other shutdown hooks, they are executed in separate threads (see java.lang.ApplicationShutdownHooks#runHooks). Spring shutdown hook is rather slow, so in most cases And yes again, you are right, ERRORs in logs do not fail build, they just sitting there making me nervous each time I analyze logs. |
|
OK. Great! |
|
I have two errors actually. This one originates from And this one from Kafka LogCleaner, I guess it will be hard to get rid of it: |
|
@hoaz , you don't need The in the end of |
|
Right, long story short, test For test only purposes SPRING_EMBEDDED_KAFKA_BROKERS is sufficient. |
Fixes: spring-projects#194 To avoid concurrent calls of the shutdown hooks make an `KafkaEmbedded` as a `Lifecycle` to let ApplicationContext to call `stop()` before destroying itself That way the Kafka servers and zookeeper and also working directory are destroyed and removed before JMV shutdown hooks and `FileNotFoundException /tmp/kafka-3266968904825284552/replication-offset-checkpoint.tmp` should disappear
|
This create another error: Anyway you dont ned to specify a log dir, kafka already uses a random one with a seed in its name. |
|
I solved this problem. As in #194 (comment), the tmp folder is removed in JVM shutdown, and when using Kafka in another hook, Embedded Kafka calls The following code can prevent Exit.setHaltProcedure((statusCode, message) -> {
if (statusCode != 1) {
Runtime.getRuntime().halt(statusCode);
}
}); |
|
Interesting; thanks. |
|
Where is the As your code test for an int i guess the latter? Anyway, it creates another error. But at least the tests are not failing. |
Hey, I am with the same issue so I am trying to understand your comment, where are you adding this, and how are you using the security manager? Thanks in advance. |
|
@Rouche When JVM shutdown Hook is running, kafka log file is deleted and Exit.halt (1) is called when other shutdown hook accesses kafka log file at the same time. Since halt is called here and status is 1, i only defend against 1. If you encounter a situation where the test fails with a different status value, you can add defense code. An error log may occur, but the test will not fail because the command is not propagated to Runtime.halt. |
|
@gustavomaia class SomeTest {
static {
Exit.setHaltProcedure((statusCode, message) -> {
if (statusCode != 1) {
Runtime.getRuntime().halt(statusCode);
}
});
}
@Test
void test1() {
}
@Test
void test2() {
}
}Even using the SecurityManager, the test can fail. |
See spring-projects#194 See spring-projects#345 See gradle/gradle#11195 **cherry-pick to 2.4.x, 2.3.x, 2.2.x, 1.3.x**
See #194 See #345 See gradle/gradle#11195 **cherry-pick to 2.4.x, 2.3.x, 2.2.x, 1.3.x**
See #194 See #345 See gradle/gradle#11195 **cherry-pick to 2.4.x, 2.3.x, 2.2.x, 1.3.x**
See #194 See #345 See gradle/gradle#11195 **cherry-pick to 2.4.x, 2.3.x, 2.2.x, 1.3.x**
See #194 See #345 See gradle/gradle#11195 **cherry-pick to 2.4.x, 2.3.x, 2.2.x, 1.3.x** # Conflicts: # spring-kafka-test/src/main/java/org/springframework/kafka/test/EmbeddedKafkaBroker.java
See #194 See #345 See gradle/gradle#11195 **cherry-pick to 2.4.x, 2.3.x, 2.2.x, 1.3.x** # Conflicts: # spring-kafka-test/src/main/java/org/springframework/kafka/test/EmbeddedKafkaBroker.java
Resolves spring-projects#194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves spring-projects#194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves spring-projects#194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves spring-projects#194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves spring-projects#194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves #194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves #194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves #194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x**
Resolves #194 Create the temporary directory in EKB instead of the broker to avoid `NoSuchFileException`s during shutdown. **cherry-pick to 2.4.x, 2.3.x, 2.2.x** # Conflicts: # spring-kafka-test/src/main/java/org/springframework/kafka/test/EmbeddedKafkaBroker.java
We are using Embedded Kafka for Integration testing.
What we're getting is
FileNotFoundException(sometimes) on Jenkins.Question - is there a possibility to turn off
ReplicaManageror so? Maybe there's another solution.The text was updated successfully, but these errors were encountered: