Skip to content

Commit

Permalink
Merge pull request #52 from theseion/log-markers
Browse files Browse the repository at this point in the history
  • Loading branch information
fzipi authored Apr 9, 2022
2 parents 2ec51c5 + 6d23a91 commit 531f76e
Show file tree
Hide file tree
Showing 22 changed files with 807 additions and 749 deletions.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
repos:
- repo: git://github.com/dnephin/pre-commit-golang
- repo: https://github.com/dnephin/pre-commit-golang
rev: v0.4.0
hooks:
- id: go-fmt
Expand Down
74 changes: 43 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@

This software should be compatible with the [Python version](https://pypi.org/project/ftw/).

I wrote this one to get more insights on the original version, and trying to shed some lights on the internals. There are many assumptions on the inner workings that I needed to dig into the code to know how they worked.
I wrote this one to get more insights on the original version, and trying to shed some light on the internals. There are many assumptions on the inner workings that I needed to dig into the code to know how they worked.

My goals are:
- get a compatible `ftw` version, with no dependencies and easy to deploy
- be CI/CD extremely friendly
- be extremely CI/CD friendly
- be fast (if possible)
- add features like:
- syntax checking on the test files
Expand All @@ -27,40 +27,30 @@ My goals are:

Go to the [releases](https://github.com/fzipi/go-ftw/releases) page and get the one that matches your OS.

If you have Go installed and configured to run Go binaries from your shell you can also run
```bash
go install github.com/fzipi/go-ftw@latest
```

## Example Usage

To run tests you need:
1. a WAF (doh!)
2. a file where the waf stores the logs
3. a config file, or environment variables, with the information to get the logs and how to parse them (I might embed this for the most commonly used, like Apache/Nginx)
3. a config file, or environment variables, with the information to get the logs and how to parse them (I might embed this for the most commonly used, like Apache/NGiNX)

By default, _ftw_ would search for a file in `$PWD` with the name `.ftw.yaml`. Example configurations for `apache` and `nginx` below:

```yaml
---
logfile: '../coreruleset/tests/logs/modsec2-apache/apache2/error.log'
logtype:
name: 'apache'
timeregex: '\[([A-Z][a-z]{2} [A-z][a-z]{2} \d{1,2} \d{1,2}\:\d{1,2}\:\d{1,2}\.\d+? \d{4})\]'
timeformat: 'ddd MMM DD HH:mm:ss.S YYYY'
```
For nginx, as logs will be to the second, you need to add the amount of time you want to truncate to. This will for example discard anything less than one second:
```yaml
---
logfile: '../coreruleset/tests/logs/modsec3-nginx/nginx/error.log'
logtype:
name: 'nginx'
timeregex: '(\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2})'
timeformat: 'YYYY/MM/DD HH:mm:ss'
timetruncate: 1s
```
Time format specification follows the one used by [gostradamus](https://github.com/bykof/gostradamus#token-table).
If your webserver uses a different time format, please [create an issue](https://github.com/fzipi/go-ftw/issues/new/choose) and we can extend the documentation to cover it.
I normally perform my testing using the [Core Rule Set](https://github.com/coreruleset/coreruleset/).
You can start the containers from that repo using docker-compose:
Expand Down Expand Up @@ -95,16 +85,15 @@ Global Flags:

```

After merging [this PR](https://github.com/coreruleset/coreruleset/pull/2080), no changes will be needed.
Until that happens, you can get and apply the [patch](https://patch-diff.githubusercontent.com/raw/coreruleset/coreruleset/pull/2080.patch), using `patch -p1 < 2080.patch`.

Then you can run your tests using:
Here's an example on how to run your tests:

`ftw run -d tests -t`
```bash
ftw run -d tests -t
```

And the result should be similar to:

```
```bash
❯ ./ftw run -d tests -t

🛠️ Starting tests!
Expand All @@ -126,14 +115,14 @@ And the result should be similar to:
running 944300-328: ✔ passed 5.873305ms
running 944300-329: ✔ passed 5.828122ms
➕ run 2354 total tests in 18.923445528s
skept 7 tests
skipped 7 tests
🎉 All tests successful!
```
Happy testing!

## Additional features

You can add functions to your tests, to simplify bulk writing, or even read values from the environment while executing. This is because `data:` sections in tests will be parse for Go [text/template](https://golang.org/pkg/text/template/) additional syntax, and with the power of additional [Sprig functions](https://masterminds.github.io/sprig/).
You can add functions to your tests, to simplify bulk writing, or even read values from the environment while executing. This is because `data:` sections in tests are parsed with Go [text/template](https://golang.org/pkg/text/template/), and also are given the power of additional [Sprig functions](https://masterminds.github.io/sprig/).

This will allow you to write tests like this:

Expand All @@ -147,7 +136,7 @@ Will be expanded to:
data: 'foo=%3d++++++++++++++++++++++++++++++++++'
```
But also, you can get values from the environment dinamically when the test is run:
But also, you can get values from the environment dynamically when the test is run:
```yaml
data: 'username={{ env "USERNAME" }}
Expand All @@ -163,7 +152,7 @@ Other interesting functions you can use are: `randBytes`, `htpasswd`, `encryptAE

## Overriding test results

Sometimes you have tests that work well in some platform combination, e.g. Apache + modsecurity2, but fail in other, e.g. Nginx + modsecurity3. Taking that into account, you can override test results using the `testoverride` config param. The test will be run, but the _result_ would be overriden, and your comment will be printed out.
Sometimes you have tests that work well for some platform combinations, e.g. Apache + modsecurity2, but fail for others, e.g. NGiNX + modsecurity3. Taking that into account, you can override test results using the `testoverride` config param. The test will be run, but the _result_ would be overriden, and your comment will be printed out.

Example:

Expand All @@ -184,11 +173,34 @@ testoverride:

You can combine any of `ignore`, `forcefail` and `forcepass` to make it work for you.

## Truncating logs
## How log parsing works
The log output from your WAF is parsed and compared to the expected output.
The problem with log files is that they aren't updated in real time, e.g. because the
web server / WAF has an internal buffer, or because there's some `fsync` magic involved).
To make log parsing consistent and guarantee that we will see output when we need it,
go-ftw uses "log markers". In essence, unique log entries are written _before_ and _after_
every test stage. go-ftw can then search for these markers.

The [container images for Core Rule Set](https://github.com/coreruleset/modsecurity-crs-docker) can be configured to write these marker log lines by setting
the `CRS_ENABLE_TEST_MARKER` environment variable. If you are testing a different WAF
you will need to instrument it with the same idea (unless you are using "cloud mode").
The rule for CRS looks like this:
```
# Write the value from the X-CRS-Test header as a marker to the log
SecRule REQUEST_HEADERS:X-CRS-Test "@rx ^.*$" \
"id:999999,\
phase:1,\
log,\
msg:'%{MATCHED_VAR}',\
pass,\
t:none"
```

Log files can get really big. Searching patterns are performed using reverse text search in the file. Because the test tool is *really* fast, we sometimes see failures in nginx depending on how fast the tests are performed, mainly because log times in nginx are truncated to one second.
The rule looks for an HTTP header named `X-CRS-Test` and writes its value to the log,
the value being the UUID of a test stage.

To overcome this, you can use the new config value `logtruncate: True`. This will, as it says, call _truncate_ on the file, actively modifying it between each test. You will need permissions to write the logfile, implying you might need to call the go-ftw binary using sudo.
You can configure the name of the HTTP header by setting the `logmarkerheadername`
option in the configuration to a custom value (the value is case insensitive).

## License
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Ffzipi%2Fgo-ftw.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Ffzipi%2Fgo-ftw?ref=badge_large)
28 changes: 13 additions & 15 deletions check/base.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
package check

import (
"time"

"github.com/fzipi/go-ftw/config"
"github.com/fzipi/go-ftw/test"
"github.com/fzipi/go-ftw/waflog"
Expand All @@ -19,13 +17,9 @@ type FTWCheck struct {
func NewCheck(c *config.FTWConfiguration) *FTWCheck {
check := &FTWCheck{
log: &waflog.FTWLogLines{
FileName: c.LogFile,
TimeRegex: c.LogType.TimeRegex,
TimeFormat: c.LogType.TimeFormat,
Since: time.Now(),
Until: time.Now(),
TimeTruncate: c.LogType.TimeTruncate,
LogTruncate: c.LogTruncate,
FileName: c.LogFile,
StartMarker: nil,
EndMarker: nil,
},
expected: &test.Output{},
overrides: &c.TestOverride,
Expand All @@ -34,12 +28,6 @@ func NewCheck(c *config.FTWConfiguration) *FTWCheck {
return check
}

// SetRoundTripTime sets the time the roundtrip took so we can check logs with it
func (c *FTWCheck) SetRoundTripTime(since time.Time, until time.Time) {
c.log.Since = since
c.log.Until = until
}

// SetExpectTestOutput sets the combined expected output from this test
func (c *FTWCheck) SetExpectTestOutput(t *test.Output) {
c.expected = t
Expand Down Expand Up @@ -106,3 +94,13 @@ func (c *FTWCheck) SetCloudMode() {
}
c.expected.Status = status
}

// SetStartMarker sets the log line that marks the start of the logs to analyze
func (c *FTWCheck) SetStartMarker(marker []byte) {
c.log.StartMarker = marker
}

// SetEndMarker sets the log line that marks the end of the logs to analyze
func (c *FTWCheck) SetEndMarker(marker []byte) {
c.log.EndMarker = marker
}
14 changes: 0 additions & 14 deletions check/base_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,17 @@ package check
import (
"sort"
"testing"
"time"

"github.com/fzipi/go-ftw/config"
"github.com/fzipi/go-ftw/test"
)

var yamlApacheConfig = `---
logfile: 'tests/logs/modsec2-apache/apache2/error.log'
logtype:
name: 'apache'
timeregex: '\[([A-Z][a-z]{2} [A-z][a-z]{2} \d{1,2} \d{1,2}\:\d{1,2}\:\d{1,2}\.\d+? \d{4})\]'
timeformat: 'ddd MMM DD HH:mm:ss.S YYYY'
`

var yamlNginxConfig = `---
logfile: 'tests/logs/modsec3-nginx/nginx/error.log'
logtype:
name: 'nginx'
timeregex: '(\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2})'
timeformat: 'YYYY/MM/DD HH:mm:ss'
timetruncate: 1s
testoverride:
ignore:
'942200-1': 'Ignore Me'
Expand All @@ -42,10 +32,6 @@ func TestNewCheck(t *testing.T) {

c := NewCheck(config.FTWConfig)

if c.log.TimeTruncate != time.Second {
t.Errorf("Failed")
}

for _, text := range c.overrides.Ignore {
if text != "Ignore Me" {
t.Errorf("Well, didn't match Ignore Me")
Expand Down
44 changes: 3 additions & 41 deletions check/logs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,35 +8,23 @@ import (
"github.com/fzipi/go-ftw/utils"
)

var nginxLogText = `2021/03/16 12:40:19 [info] 17#17: *2495 ModSecurity: Warning. Matched "Operator ` + "`" + `Within' with parameter ` + "`" + `GET HEAD POST OPTIONS' against variable ` + "`" + `REQUEST_METHOD' (Value: ` + "`" + `OTHER' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-911-METHOD-ENFORCEMENT.conf"] [line "27"] [id "911100"] [rev ""] [msg "Method is not allowed by policy"] [data "OTHER"] [severity "2"] [ver "OWASP_CRS/3.3.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272/220/274"] [tag "PCI/12.1"] [hostname "172.19.0.3"] [uri "/"] [unique_id "161589841954.023243"] [ref "v0,5"], client: 172.19.0.1, server: modsec3-nginx, request: "OTHER / HTTP/1.1", host: "localhost"
2021/03/16 12:40:19 [info] 17#17: *2495 ModSecurity: Warning. Matched "Operator ` + "`" + `Pm' with parameter ` + "`" + `AppleWebKit Android' against variable ` + "`" + `REQUEST_HEADERS:User-Agent' (Value: ` + "`" + `ModSecurity CRS 3 Tests' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1360"] [id "920300"] [rev ""] [msg "Request Missing an Accept Header"] [data ""] [severity "5"] [ver "OWASP_CRS/3.3.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [tag "paranoia-level/3"] [hostname "172.19.0.3"] [uri "/"] [unique_id "161589841954.023243"] [ref "v0,5v63,23"], client: 172.19.0.1, server: modsec3-nginx, request: "OTHER / HTTP/1.1", host: "localhost"
2021/03/16 12:40:19 [info] 17#17: *2495 ModSecurity: Warning. Matched "Operator ` + "`" + `Ge' with parameter ` + "`" + `5' against variable ` + "`" + `TX:ANOMALY_SCORE' (Value: ` + "`" + `7' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "138"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 7)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "172.19.0.3"] [uri "/"] [unique_id "161589841954.023243"] [ref ""], client: 172.19.0.1, server: modsec3-nginx, request: "OTHER / HTTP/1.1", host: "localhost"
2021/03/16 12:40:19 [info] 17#17: *2497 ModSecurity: Warning. Matched "Operator ` + "`" + `Within' with parameter ` + "`" + `GET HEAD POST OPTIONS' against variable ` + "`" + `REQUEST_METHOD' (Value: ` + "`" + `OTHER' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-911-METHOD-ENFORCEMENT.conf"] [line "27"] [id "911100"] [rev ""] [msg "Method is not allowed by policy"] [data "OTHER"] [severity "2"] [ver "OWASP_CRS/3.3.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272/220/274"] [tag "PCI/12.1"] [hostname "172.19.0.3"] [uri "/"] [unique_id "161589841970.216949"] [ref "v0,5"], client: 172.19.0.1, server: modsec3-nginx, request: "OTHER / HTTP/1.1", host: "localhost"
2021/03/16 12:40:19 [info] 17#17: *2497 ModSecurity: Warning. Matched "Operator ` + "`" + `Pm' with parameter ` + "`" + `AppleWebKit Android' against variable ` + "`" + `REQUEST_HEADERS:User-Agent' (Value: ` + "`" + `ModSecurity CRS 3 Tests' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1360"] [id "920300"] [rev ""] [msg "Request Missing an Accept Header"] [data ""] [severity "5"] [ver "OWASP_CRS/3.3.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [tag "paranoia-level/3"] [hostname "172.19.0.3"] [uri "/"] [unique_id "161589841970.216949"] [ref "v0,5v63,23"], client: 172.19.0.1, server: modsec3-nginx, request: "OTHER / HTTP/1.1", host: "localhost"
2021/03/16 12:40:19 [info] 17#17: *2497 ModSecurity: Warning. Matched "Operator ` + "`" + `Ge' with parameter ` + "`" + `5' against variable ` + "`" + `TX:ANOMALY_SCORE' (Value: ` + "`" + `7' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "138"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 7)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "172.19.0.3"] [uri "/"] [unique_id "161589841970.216949"] [ref ""], client: 172.19.0.1, server: modsec3-nginx, request: "OTHER / HTTP/1.1", host: "localhost"
`

var apacheLogText = `[Tue Jan 05 02:21:09.637165 2021] [:error] [pid 76:tid 139683434571520] [client 172.23.0.1:58998] [client 172.23.0.1] ModSecurity: Warning. Pattern match "\\\\b(?:keep-alive|close),\\\\s?(?:keep-alive|close)\\\\b" at REQUEST_HEADERS:Connection. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "339"] [id "920210"] [msg "Multiple/Conflicting Connection Header Data Found"] [data "close,close"] [severity "WARNING"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [hostname "localhost"] [uri "/"] [unique_id "X-PNFSe1VwjCgYRI9FsbHgAAAIY"]
var logText = `[Tue Jan 05 02:21:09.637165 2021] [:error] [pid 76:tid 139683434571520] [client 172.23.0.1:58998] [client 172.23.0.1] ModSecurity: Warning. Pattern match "\\\\b(?:keep-alive|close),\\\\s?(?:keep-alive|close)\\\\b" at REQUEST_HEADERS:Connection. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "339"] [id "920210"] [msg "Multiple/Conflicting Connection Header Data Found"] [data "close,close"] [severity "WARNING"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [hostname "localhost"] [uri "/"] [unique_id "X-PNFSe1VwjCgYRI9FsbHgAAAIY"]
[Tue Jan 05 02:21:09.637731 2021] [:error] [pid 76:tid 139683434571520] [client 172.23.0.1:58998] [client 172.23.0.1] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1230"] [id "920300"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [tag "paranoia-level/2"] [hostname "localhost"] [uri "/"] [unique_id "X-PNFSe1VwjCgYRI9FsbHgAAAIY"]
[Tue Jan 05 02:21:09.638572 2021] [:error] [pid 76:tid 139683434571520] [client 172.23.0.1:58998] [client 172.23.0.1] ModSecurity: Warning. Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "91"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "localhost"] [uri "/"] [unique_id "X-PNFSe1VwjCgYRI9FsbHgAAAIY"]
[Tue Jan 05 02:21:09.647668 2021] [:error] [pid 76:tid 139683434571520] [client 172.23.0.1:58998] [client 172.23.0.1] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "87"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 3, 2, 0, 0"] [ver "OWASP_CRS/3.3.0"] [tag "event-correlation"] [hostname "localhost"] [uri "/"] [unique_id "X-PNFSe1VwjCgYRI9FsbHgAAAIY"]
`

func TestAssertApacheLogContainsOK(t *testing.T) {
func TestAssertLogContainsOK(t *testing.T) {
err := config.NewConfigFromString(yamlApacheConfig)
if err != nil {
t.Errorf("Failed!")
}
logName, _ := utils.CreateTempFileWithContent(apacheLogText, "test-apache-*.log")
logName, _ := utils.CreateTempFileWithContent(logText, "test-*.log")
defer os.Remove(logName)
config.FTWConfig.LogFile = logName

c := NewCheck(config.FTWConfig)

since := utils.GetFormattedTime("2021-01-05T00:30:26.371Z")
until := utils.GetFormattedTime("2021-01-06T18:30:26.371Z")

c.SetRoundTripTime(since, until)
c.SetLogContains(`id "920300"`)

// c.SetNoLogContains(`Something that is not there`)
Expand All @@ -49,29 +37,3 @@ func TestAssertApacheLogContainsOK(t *testing.T) {
// t.Errorf("Failed !")
// }
}

func TestAssertNginxLogContainsOK(t *testing.T) {
err := config.NewConfigFromString(yamlNginxConfig)
if err != nil {
t.Errorf("Failed!")
}
logName, _ := utils.CreateTempFileWithContent(nginxLogText, "test-nginx-*.log")
defer os.Remove(logName)
config.FTWConfig.LogFile = logName

c := NewCheck(config.FTWConfig)

since := utils.GetFormattedTime("2021-03-15T00:30:26.371Z")
until := utils.GetFormattedTime("2021-03-18T18:30:26.371Z")

c.SetRoundTripTime(since, until)
c.SetLogContains(`id "911100"`)

if !c.AssertLogContains() {
t.Errorf("Failed !")
}

if c.AssertNoLogContains() {
t.Error("No log contains failed")
}
}
3 changes: 2 additions & 1 deletion cmd/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ var runCmd = &cobra.Command{
log.Fatal().Err(err)
}

os.Exit(runner.Run(include, exclude, showTime, quiet, tests))
currentRun := runner.Run(include, exclude, showTime, quiet, tests)
os.Exit(currentRun.Stats.TotalFailed())
},
}

Expand Down
Loading

0 comments on commit 531f76e

Please sign in to comment.