Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Detect fast-check PBT library for fuzz section #3073

Merged
merged 3 commits into from
Jun 4, 2023
Merged

✨ Detect fast-check PBT library for fuzz section #3073

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
667da8a
✨ Detect fast-check PBT library for fuzz section
dubzzz May 26, 2023
0d7bf2e
Typo
dubzzz May 26, 2023
f908151
Update missing md files
dubzzz May 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -457,7 +457,7 @@ Name | Description | Risk Level | Token Req
[Contributors](docs/checks.md#contributors) | Does the project have contributors from at least two different organizations? | Low | PAT, GITHUB_TOKEN | Validating |
[Dangerous-Workflow](docs/checks.md#dangerous-workflow) | Does the project avoid dangerous coding patterns in GitHub Action workflows? | Critical | PAT, GITHUB_TOKEN | Unsupported |
[Dependency-Update-Tool](docs/checks.md#dependency-update-tool) | Does the project use tools to help update its dependencies? | High | PAT, GITHUB_TOKEN | Unsupported |
[Fuzzing](docs/checks.md#fuzzing) | Does the project use fuzzing tools, e.g. [OSS-Fuzz](https://github.com/google/oss-fuzz)? | Medium | PAT, GITHUB_TOKEN | Validating
[Fuzzing](docs/checks.md#fuzzing) | Does the project use fuzzing tools, e.g. [OSS-Fuzz](https://github.com/google/oss-fuzz), [QuickCheck](https://hackage.haskell.org/package/QuickCheck) or [fast-check](https://fast-check.dev/)? | Medium | PAT, GITHUB_TOKEN | Validating
[License](docs/checks.md#license) | Does the project declare a license? | Low | PAT, GITHUB_TOKEN | Validating |
[Maintained](docs/checks.md#maintained) | Is the project at least 90 days old, and maintained? | High | PAT, GITHUB_TOKEN | Validating |
[Pinned-Dependencies](docs/checks.md#pinned-dependencies) | Does the project declare and pin [dependencies](https://docs.github.com/en/free-pro-team@latest/github/visualizing-repository-data-with-graphs/about-the-dependency-graph#supported-package-ecosystems)? | Medium | PAT, GITHUB_TOKEN | Validating |
Expand Down
36 changes: 31 additions & 5 deletions checks/raw/fuzzing.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,13 @@ import (
)

const (
fuzzerOSSFuzz = "OSSFuzz"
fuzzerClusterFuzzLite = "ClusterFuzzLite"
oneFuzz = "OneFuzz"
fuzzerBuiltInGo = "GoBuiltInFuzzer"
fuzzerPropertyBasedHaskell = "HaskellPropertyBasedTesting"
fuzzerOSSFuzz = "OSSFuzz"
fuzzerClusterFuzzLite = "ClusterFuzzLite"
oneFuzz = "OneFuzz"
fuzzerBuiltInGo = "GoBuiltInFuzzer"
fuzzerPropertyBasedHaskell = "HaskellPropertyBasedTesting"
fuzzerPropertyBasedJavaScript = "JavaScriptPropertyBasedTesting"
fuzzerPropertyBasedTypeScript = "TypeScriptPropertyBasedTesting"
// TODO: add more fuzzing check supports.
)

Expand Down Expand Up @@ -87,6 +89,30 @@ var languageFuzzSpecs = map[clients.LanguageName]languageFuzzConfig{
"Property-based testing in Haskell generates test instances randomly or exhaustively " +
"and test that specific properties are satisfied."),
},
// Fuzz patterns for JavaScript and TypeScript based on property-based testing.
//
// Based on the import of one of these packages:
// * https://fast-check.dev/
//
// This is not an exhaustive list.
clients.JavaScript: {
filePattern: "*.js",
dubzzz marked this conversation as resolved.
Show resolved Hide resolved
// Look for direct imports of fast-check.
funcPattern: `(from\s+['"]fast-check['"]|require\(\s*['"]fast-check['"]\s*\))`,
Name: fuzzerPropertyBasedJavaScript,
Desc: asPointer(
"Property-based testing in JavaScript generates test instances randomly or exhaustively " +
"and test that specific properties are satisfied."),
},
clients.TypeScript: {
filePattern: "*.ts",
// Look for direct imports of fast-check.
funcPattern: `(from\s+['"]fast-check['"]|require\(\s*['"]fast-check['"]\s*\))`,
Name: fuzzerPropertyBasedTypeScript,
Desc: asPointer(
"Property-based testing in TypeScript generates test instances randomly or exhaustively " +
"and test that specific properties are satisfied."),
},
// TODO: add more language-specific fuzz patterns & configs.
}

Expand Down
74 changes: 74 additions & 0 deletions checks/raw/fuzzing_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,80 @@ func Test_checkFuzzFunc(t *testing.T) {
},
fileContent: "import Test.Hspec",
},
{
name: "JavaScript fast-check via require",
want: true,
fileName: []string{"main.spec.js"},
langs: []clients.Language{
{
Name: clients.JavaScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-check');",
},
{
name: "JavaScript fast-check via import",
want: true,
fileName: []string{"main.spec.js"},
langs: []clients.Language{
{
Name: clients.JavaScript,
NumLines: 50,
},
},
fileContent: "import fc from \"fast-check\";",
},
{
name: "JavaScript with no property-based testing",
want: false,
fileName: []string{"main.spec.js"},
wantErr: true,
langs: []clients.Language{
{
Name: clients.JavaScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-other');",
},
{
name: "TypeScript fast-check via require",
want: true,
fileName: []string{"main.spec.ts"},
langs: []clients.Language{
{
Name: clients.TypeScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-check');",
},
{
name: "TypeScript fast-check via import",
want: true,
fileName: []string{"main.spec.ts"},
langs: []clients.Language{
{
Name: clients.TypeScript,
NumLines: 50,
},
},
fileContent: "import fc from \"fast-check\";",
},
{
name: "TypeScript with no property-based testing",
want: false,
fileName: []string{"main.spec.ts"},
wantErr: true,
langs: []clients.Language{
{
Name: clients.TypeScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-other');",
},
}
for _, tt := range tests {
tt := tt
Expand Down
4 changes: 3 additions & 1 deletion docs/checks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,9 @@ This check tries to determine if the project uses
1. if the repository name is included in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project list;
2. if [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) is deployed in the repository;
3. if there are user-defined language-specified fuzzing functions in the repository.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/) and a limited set of property-based testing libraries for Haskell.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/),
- a limited set of property-based testing libraries for Haskell including [QuickCheck](https://hackage.haskell.org/package/QuickCheck), [Hedgehog](https://hedgehog.qa/), [validity](https://hackage.haskell.org/package/validity) or [SmallCheck](https://hackage.haskell.org/package/smallcheck),
- a limited set of property-based testing libraries for JavaScript and TypeScript including [fast-check](https://fast-check.dev/).
4. if it contains a [OneFuzz](https://github.com/microsoft/onefuzz) integration [detection file](https://github.com/microsoft/onefuzz/blob/main/docs/getting-started.md#detecting-the-use-of-onefuzz);

Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data
Expand Down
4 changes: 3 additions & 1 deletion docs/checks/internal/checks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,9 @@ checks:
1. if the repository name is included in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project list;
2. if [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) is deployed in the repository;
3. if there are user-defined language-specified fuzzing functions in the repository.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/) and a limited set of property-based testing libraries for Haskell.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/),
- a limited set of property-based testing libraries for Haskell including [QuickCheck](https://hackage.haskell.org/package/QuickCheck), [Hedgehog](https://hedgehog.qa/), [validity](https://hackage.haskell.org/package/validity) or [SmallCheck](https://hackage.haskell.org/package/smallcheck),
- a limited set of property-based testing libraries for JavaScript and TypeScript including [fast-check](https://fast-check.dev/).
4. if it contains a [OneFuzz](https://github.com/microsoft/onefuzz) integration [detection file](https://github.com/microsoft/onefuzz/blob/main/docs/getting-started.md#detecting-the-use-of-onefuzz);

Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data
Expand Down