Can JQF be used in Spring Application

[flyingjohn]

I use jqf to fuzz generic functions, and the result is good. But when i try to fuzz web application functions written in Spring format, some problems occur. How can i use autowire and mock in JQF, is it possible to extend jqf to fuzz wen application?

[rohanpadhye]

Hi @flyingjohn. I have not tried fuzzing Spring applications (in fact, I don't know much about Spring). Assuming the application is in pure Java, JQF should not have a problem with instrumentation and running tests. However, if the framework is heavily multi-threaded, that could pose some problems with fuzzing.

Could you please describe the problems that you are facing in more detail?

[flyingjohn]

Hi @rohanpadhye, thanks for your answer. JQF performs great when I try to fuzz most java programs. But the thing is that,，Spring is a most-used web framework，it uses a mechanism called "Dependency injection" to create objects dynamically (see @AutoWire)，for example:

@RunWith(SpringRunner.class)

public class OrderServiceTest {

       @Autowired
       private OrderService orderService;

       @Test
       public void testOrderService() {
          // test the orderService
      }
}

SpringJunit4 will load the private object orderService dymanically. But when I use JQF to test the same function like this:

@RunWith(JQF.class)
public class OrderServiceTest {

       @Autowired
       private OrderService orderService;

       @Fuzz
       public void testOrderService() {
          // test the orderService
      }
}

It won't work and throw NullPointerException for orderService. And it's reasonable because JQF doesn't implement @autowire notation.
So, I tried this way:

@RunWith(JQF.class)
public class OrderServiceTest {

       @Fuzz
       public void testOrderService() {
           OrderService orderService = new OrderService();
          // test the orderService
      }
}

However, this still won't work and orderService still be null. So I think objects must be loaded through Dependency injection in Spring applications. Is it true that JQF can't work in this situation（Spring applications）?

[rohanpadhye]

The JQF test runner (JQF.class) simply extends the JUnitQuickCheck test runner. I see that Spring uses its own test runner. In your code, that's @RunWith(SpringRunner.class). What you want to do is somehow combine these.

I have never tried something like this before, but StackOverflow seems to have some posts about people trying to combine MockitoRunner with other runners: https://stackoverflow.com/questions/24431427/multiple-runwith-statements-in-junit. Maybe there is a solution hidden in there somewhere?

[flyingjohn]

Thank you for your help!

[yevgenypats]

@flyingjohn @rohanpadhye I stumbled upon the same problem and as far as I understand this is due to this behaviour of springboot. Any idea of a more elegant solution?

[tech-bee-10]

@flyingjohn @yevgenypats I was trying to explore the same option. Have you found any working solution for this? If yes, can you please share, it would be really helpful :)

[rohanpadhye]

In order to use JQF + Spring, I think you can avoid using @RunWith(SpringRunner.class) (since you will need a @RunWith(JQF.class)) and instead use a combination of SpringClassRule and SpringMethodRule to get dependency injection. See this article: https://eddumelendez.github.io/blog/2015/08/01/spring-4-2-0-springclassrule-and-springmethodrule/. I've not tested it because I don't work with Spring, but as per the article it should solve the problem described by the OP. @rand-guy Let me know if this works for you.

[tech-bee-10]

@rohanpadhye Thanks for sharing this. This helped in moving forward from this issue.

But after going forward, facing another issue where I think it is not able to find classes from the dependencies.

For example, when I try to run the jqf mvn plugin command, where it also tries to initialise some beans, it gives below error but it is already part of dependencies:
nested exception is java.io.FileNotFoundException: class path resource [org/springframework/security/config/annotation/web/configuration/WebMvcSecurityConfiguration.class] cannot be opened because it does not exist

While trying something different it gives:
java.lang.IllegalArgumentException: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct.

Any thoughts on how to resolve those?

[rohanpadhye]

@rand-guy I'm glad the annotations helped. Unfortunately, I don't know how to resolve that specific error. I don't use Spring so I don't know about the API very much.

One thing I can tell you is that JQF is just running the test as a regular JUnit test, using the classpath for scope test as defined in the pom.xml explicitly or implicitly. So, if you can run the following command:

mvn test -Dtest=MyTestClass

Where MyTestClass is the class annotated with @RunWith(JQF.class), then ideally you should see the test execute with a 1000 randomly sampled inputs (without coverage feedback). If this works but running the JQF maven plugin (mvn jqf:fuzz) does not work, then let me know; I can help you debug it. However, if the regular mvn test command also fails with the same exceptions, then the issue is not JQF specific. You just need to figure out how to prepare a JUnit test that executes a Spring application without the SpringRunner, and that's going to be Spring-specific. You may get better answers by asking this question to the Spring developer community: "How to write a JUnit test without SpringRunner?"

[tech-bee-10]

I have tried your suggestion. I tried running below command but for some reason it is not able to find the test and outputs " No tests were executed!".

mvn test -Dtest=MathControllerSpringTest

This is the file:
https://github.com/rand-guy/sampleproject/blob/master/src/test/java/com/example/sampleproject/controllers/MathControllerSpringTest.java

But when I try running the test directly via IDE's Run Test option, it is able to run it successfully and generated 100 tests.

Will you be able to help with these details? Let me know if you need any details which could help ease the debugging.

[rohanpadhye]

Thanks for the sample repo @rand-guy ! I was able to reproduce your issue. Turns out that spring-boot-starter-test pulls in JUnit5 by default so it won't let you run JUnit4 tests out of the box. I did some searching around and apparently have to add a dependency on junit-vintage-engine to be able to run JUnit4 tests. I've filed a PR on your sample repo to add this (tech-bee-10/sampleproject#1).

For me, this makes mvn test -Dtest=MathControllerSpringTest run just fine. I can also use JQF to fuzz class MathControllerTest successfully, but there is still an issue with fuzzing MathControllerSpringTest. I get an error message from Spring saying "No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct". Do you know what that's about or how JUnit is able to perform auto configuration?