Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --fuzzing-ratio to CLI #2496

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+23 −13
Open

Add --fuzzing-ratio to CLI #2496

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2a79a7d
Improve CLI:
Markoutte Aug 10, 2023
57555e4
We can ignore --source option
Markoutte Aug 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions utbot-cli/src/main/kotlin/org/utbot/cli/GenerateTestsAbstractCommand.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import com.github.ajalt.clikt.parameters.options.multiple
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.options.unique
import com.github.ajalt.clikt.parameters.types.choice
import com.github.ajalt.clikt.parameters.types.double
import com.github.ajalt.clikt.parameters.types.long
import mu.KotlinLogging
import org.utbot.common.PathUtil.classFqnToPath
Expand All @@ -25,14 +26,7 @@ import org.utbot.framework.codegen.domain.testFrameworkByName
import org.utbot.framework.codegen.generator.CodeGenerator
import org.utbot.framework.codegen.generator.CodeGeneratorParams
import org.utbot.framework.codegen.services.language.CgLanguageAssistant
import org.utbot.framework.plugin.api.ClassId
import org.utbot.framework.plugin.api.CodegenLanguage
import org.utbot.framework.plugin.api.ExecutableId
import org.utbot.framework.plugin.api.MethodId
import org.utbot.framework.plugin.api.MockStrategyApi
import org.utbot.framework.plugin.api.TestCaseGenerator
import org.utbot.framework.plugin.api.TreatOverflowAsError
import org.utbot.framework.plugin.api.UtMethodTestSet
import org.utbot.framework.plugin.api.*
import org.utbot.framework.plugin.services.JdkInfoDefaultProvider
import org.utbot.summary.summarizeAll
import java.io.File
Expand Down Expand Up @@ -131,14 +125,20 @@ abstract class GenerateTestsAbstractCommand(name: String, help: String) :
.long()
.default(LONG_GENERATION_TIMEOUT)

private val fuzzingRation by option(
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "ration" seems misused here. Consider calling this property fuzzingRatio instead of fuzzingRation.

"--fuzzing-ratio",
help = "Specify the ratio between symbolic engine and fuzzing"
)
.double()
Comment on lines +128 to +132
Copy link

@liblit-at-amazon liblit-at-amazon Feb 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to this comment from @tyuldashev, the default is 0.1 fuzzing, 0.9 symbolic. If we take the help text literally, then the default "ratio between symbolic engine and fuzzing" would be 0.9 ÷ 0.1 = 9. But from my understanding of the code, 9 is not a valid value here, and this argument should actually be 0.1 to mimic the default behavior.

Consider revising this option's name and help text to reflect its actual semantics and to document its default behavior. For example:

private val fuzzingFraction by option(
        "-z",
        "--fuzzing-fraction",
        help = "Specifies the fraction of time spent using fuzzing instead of the symbolic engine (default: $DEFAULT_FUZZING_VALUE)"
)
        .double()
        .default(DEFAULT_FUZZING_VALUE)
        .validate { require(it in 0.0 .. 1.0) { "should be between 0.0 and 1.0." } }

where DEFAULT_FUZZING_VALUE is the default value of org.utbot.framework.plugin.api.TestFlow.fuzzingValue: currently 0.1.


protected open val classLoader: URLClassLoader by lazy {
val urls = classPath!!
.split(File.pathSeparator)
.map { uri ->
uri.toPath().toURL()
}
.toTypedArray()
URLClassLoader(urls)
URLClassLoader(urls, null)
}

abstract override fun run()
Expand All @@ -162,7 +162,15 @@ abstract class GenerateTestsAbstractCommand(name: String, help: String) :
targetMethods,
mockStrategy,
chosenClassesToMockAlways,
generationTimeout
generationTimeout,
generate = fuzzingRation?.let { rat ->
testFlow {
generationTimeout = [email protected]
isSymbolicEngineEnabled = rat < 1.0
isFuzzingEnabled = rat > 0.0
fuzzingValue = rat
}
} ?: defaultTestFlow(generationTimeout)
).let {
if (sourceCodeFile != null) it.summarizeAll(searchDirectory, sourceCodeFile.toFile()) else it
}
Expand Down
8 changes: 5 additions & 3 deletions utbot-cli/src/main/kotlin/org/utbot/cli/GenerateTestsCommand.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ class GenerateTestsCommand :
"-s", "--source",
help = "Specifies source code file for a generated test"
)
.required()
.check("Must exist and end with .java or .kt suffix") {
(it.endsWith(".java") || it.endsWith(".kt")) && Files.exists(Paths.get(it))
}
Expand Down Expand Up @@ -113,7 +112,7 @@ class GenerateTestsCommand :
val testSets = generateTestSets(
testCaseGenerator,
targetMethods,
Paths.get(sourceCodeFile),
sourceCodeFile?.let(Paths::get),
searchDirectory = workingDirectory,
chosenClassesToMockAlways = (Mocker.defaultSuperClassesToMockAlwaysNames + classesToMockAlways)
.mapTo(mutableSetOf()) { ClassId(it) }
Expand Down Expand Up @@ -149,9 +148,12 @@ class GenerateTestsCommand :
projectRootPath == null -> {
println("The path to the project root is required to generate a report. Please, specify \"--project-root\" option.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be we should not ask specifying but suggest specifying the source code. Otherwise the user may think that it is mandatory for proper work of utility.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --project-root is mandatory only when --sarif is set. In others cases it is not mandatory

}
sourceCodeFile == null -> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if we need curly braces here )

println("The source file is not found. Please, specify \"--source\" option.")
}
else -> {
val sourceFinding =
SourceFindingStrategyDefault(classFqn, sourceCodeFile, testsFilePath, projectRootPath)
SourceFindingStrategyDefault(classFqn, sourceCodeFile!!, testsFilePath, projectRootPath)
val report = SarifReport(testSets, testClassBody, sourceFinding).createReport().toJson()
saveToFile(report, sarifReport)
println("The report was saved to \"$sarifReport\".")
Expand Down