add flow support in template (i.e javascript scripting)

[tarunKoyalwar]

Proposed changes

Complete Documentation of flow is available at https://github.com/projectdiscovery/nuclei/tree/feat-nuclei-template-flow/v2/pkg/tmplexec/flow#readme

example flow templates are available in integration_test at https://github.com/projectdiscovery/nuclei/tree/feat-nuclei-template-flow/integration_tests/flow

nuclei/v2/pkg/tmplexec/flow/README.md

Lines 1 to 319 in f5ac33a

	# flow
	flow is a new template engine/backend introduced in v3 which primarily adds 2 most awaited features
	- conditional execution of requests (ex: `flow: dns() && http()`)
	- request execution orchestration (iterate over a slice, request execution order, if/for statement)
	both of these features are implemented using javascript (ECMAScript 5.1) with [goja](https://github.com/dop251/goja) backend.
	## conditional execution
	Many times when writing complex templates we might need to add some extra checks (or conditional statements) before executing certain part of request
	An ideal example of this would be when bruteforcing wordpress login with default usernames and passwords. If we try to write a template for this it would be something like this
	```yaml
	id: wordpress-bruteforce
	info:
	name: WordPress Login Bruteforce
	author: pdteam
	severity: high
	http:
	- method: POST
	path:
	- "{{BaseURL}}/wp-login.php"
	payloads:
	username:
	- admin
	- guest
	- testuser
	password:
	- password123
	- qwertyuiop
	- letmein
	body: "log=§username§&pwd=§password§&wp-submit=Log+In"
	attack: clusterbomb
	matchers:
	- type: word
	words:
	- "ERROR"
	part: body
	negative: true
	```
	but if we carefully re-evaluate this template, we can see that template is sending 9 requests without even checking, if the url actually exists or target site is actually a wordpress site. before v3 it was possible to do this by adding a extractor and sending additional content in say url fragment and it would fail if request was not successful and another way would be writing a workflow (2 templates and 1 workflow file i.e total 3 files for 1 template) but this is `hacky` and not a good solution.
	With addition of flow in Nuclei v3 we can re-write this template to first check if target is a wordpress site, if yes then bruteforce login with default credentials and this can be achieved by simply adding one line of content i.e `flow: http("check-wp") && http("bruteforce")` and nuclei will take care of everything else.
	```yaml
	id: wordpress-bruteforce
	info:
	name: WordPress Login Bruteforce
	author: pdteam
	severity: high
	flow: http("check-wp") && http("bruteforce")
	http:
	- id: check-wp
	method: GET
	path:
	- "{{BaseURL}}/wp-login.php"
	matchers:
	- type: word
	words:
	- "WordPress"
	part: body
	- type: word
	words:
	- "wp-content"
	part: body
	matchers-condition: and
	- id: bruteforce
	method: POST
	path:
	- "{{BaseURL}}/wp-login.php"
	payloads:
	username:
	- admin
	- guest
	- testuser
	password:
	- password123
	- qwertyuiop
	- letmein
	body: "log=§username§&pwd=§password§&wp-submit=Log+In"
	attack: clusterbomb
	matchers:
	- type: word
	words:
	- "ERROR"
	part: body
	negative: true
	```
	**Note:** this is just a example template with poor matchers. refer 'nuclei-templates' repo for final template
	The update template now seems straight forward and easy to understand. we are first checking if target is a wordpress site and then executing bruteforce requests. This is just a simple example of conditional execution and flow accepts any Javascript (ECMAScript 5.1) expression/code so you are free to craft any conditional execution logic you want using for,if and whatnot.
	## request execution orchestration
	`conditional execution` is one simple use case of flow but `flow` is much more powerful than that, for example it can be used to
	- iterate over a slice of values and execute requests for each value (ex: [dns-flow-probe](testcases/nuclei-flow-dns.yaml))
	- extract values from one request and iterate over them and execute requests for each value ex: [dns-flow-probe](testcases/nuclei-flow-dns.yaml)
	- get/set values from/to template context (global variables)
	- print/log values to stdout at xyz condition or while debugging
	- adding custom logic during template execution (ex: if status code is 403 => login and then re-run it)
	- use any/all ECMAScript 5.1 javascript (like objects,arrays etc) and build/transform variables/input at runtime
	- update variables at runtime (ex: when jwt expires update it by using refresh token and then continue execution)
	- and a lot more (this is just a tip of iceberg)
	simply put request execution orchestration can be understood as nuclei logic bindings for javascript (i.e two way interaction between javascript and nuclei for a specific template)
	To better understand orchestration we can try to build a template for vhost enumeration using flow. which usually requires writing / using a new tool
	**for basic vhost enumeration a template should**
	- do a PTR lookup for given ip
	- get SSL ceritificate for given ip (i.e tls-grab)
	- extract subject_cn from certificate
	- extract subject_alt_names(SAN) from certificate
	- filter out wildcard prefix from above values
	- and finally bruteforce all found vhosts
	**Now if we try to implement this in template it would be**
	```yaml
	# send a ssl request to get certificate
	ssl:
	- address: "{{Host}}:{{Port}}"
	# do a PTR lookup for given ip and get PTR value
	dns:
	- name: "{{FQDN}}"
	type: PTR
	matchers:
	- type: word
	words:
	- "IN\tPTR"
	extractors:
	- type: regex
	name: ptrValue
	internal: true
	group: 1
	regex:
	- "IN\tPTR\t(.+)"
	# bruteforce all found vhosts
	http:
	- raw:
	- |
	GET / HTTP/1.1
	Host: {{vhost}}
	matchers:
	- type: status
	negative: true
	status:
	- 400
	- 502
	extractors:
	- type: dsl
	dsl:
	- '"VHOST: " + vhost + ", SC: " + status_code + ", CL: " + content_length' tarun@macbook:~/Codebase/nuclei/integration_tes
	```
	**But this template is not yet ready as it is missing core logic i.e how we use all these obtained data and do bruteforce**
	and this is where flow comes into picture. flow is javascript code with two way bindings to nuclei. if we write javascript code to orchestrate vhost enumeration it is as simple as
	```javascript
	ssl();
	dns();
	for (let vhost of iterate(template["ssl_subject_cn"],template["ssl_subject_an"])) {
	set("vhost", vhost);
	http(); }
	```
	With just extra 5 lines of javascript code template can now perform vhost enumeration and this can be run on scale with all awesome features of nuclei with various supported inputs like ASN,CIDR,URL etc
	In above Js code we are using some Nuclei JS bindings lets understand what they do
	- `ssl()` => execute ssl request
	- `dns()` => execute dns request
	- `template["ssl_subject_cn"]` => get value of `ssl_subject_cn` from template context (global variables)
	- `iterate()` => this is a nuclei helper function which iterates any type of value (array,map,string,number) while handling empty / nil values
	- `set("vhost",vhost)` => creates new variable `vhost` in template and assigns value of `vhost` to it
	- `http()` => execute http request
	This template is now missing one last thing i.e
	- removing wildcard prefix (*.) in subject_cn,subject_an
	- trailing `.` in PTR value
	and this can be done using either JS methods of using DSL helper functions as shown in below template
	```yaml
	id: vhost-enum-flow
	info:
	name: vhost enum flow
	author: tarunKoyalwar
	severity: info
	description: |
	vhost enumeration by extracting potential vhost names from ssl certificate and dns ptr records
	flow: |
	ssl();
	dns({hide: true});
	for (let vhost of iterate(template["ssl_subject_cn"],template["ssl_subject_an"])) {
	vhost = vhost.replace("*.", "")
	set("vhost", vhost);
	http();
	}
	ssl:
	- address: "{{Host}}:{{Port}}"
	dns:
	- name: "{{FQDN}}"
	type: PTR
	matchers:
	- type: word
	words:
	- "IN\tPTR"
	extractors:
	- type: regex
	name: ptrValue
	internal: true
	group: 1
	regex:
	- "IN\tPTR\t(.+)"
	http:
	- raw:
	- |
	GET / HTTP/1.1
	Host: {{trim_suffix(vhost, ".")}}
	matchers:
	- type: status
	negative: true
	status:
	- 400
	- 502
	extractors:
	- type: dsl
	dsl:
	- '"VHOST: " + vhost + ", SC: " + status_code + ", CL: " + content_length'
	```
	### Nuclei JS Bindings
	This section contains a brief description of all nuclei JS bindings and their usage
	**1. Protocol Execution Functions**
	Any protocol that is present in a nuclei template can be called/executed in javascript in format `proto_name()` i.e `http()` , `dns()` , `ssl()` etc
	If we want to execute a specific request of a protocol (ref: see [nuclei-flow-dns](testcases/nuclei-flow-dns-id.yaml)) this can be achieved by either passing
	- index of that request in protocol (ex: `dns(0)`, `dns(1)` etc)
	- id of that request in protocol (ex: `dns("extract-vps")`, `dns("probe-http")` etc)
	For More complex use cases multiple requests of a single protocol can be executed by just specifying their index or id one after another (ex: `dns("extract-vps","1")`)
	**2. Iterate Helper Function**
	Iterate is a nuclei js helper function which can be used to iterate over any type of value (array,map,string,number) while handling empty / nil values.
	This is addon helper function from nuclei to omit boilerplate code of checking if value is empty or not and then iterating over it
	```javascript
	iterate(123,{"a":1,"b":2,"c":3})
	// iterate over array with custom separator
	iterate([1,2,3,4,5], " ")
	```
	**Note:** In above example we used `iterate(template["ssl_subject_cn"],template["ssl_subject_an"])` which removed lot of boilerplate code of checking if value is empty or not and then iterating over it
	**3. Set Helper Function**
	When Iterating over a values/array or some other use case we might want to invoke a request with custom/given value and this can be achieved by using `set()` helper function. When invoked/called it adds given variable to template context (global variables) and that value is used during execution of request/protocol. the format of `set()` is `set("variable_name",value)` ex: `set("username","admin")` etc
	```javascript
	for (let vhost of myArray) {
	set("vhost", vhost);
	http(1)
	}
	```
	**Note:** In above example we used `set("vhost", vhost)` which added `vhost` to template context (global variables) and then called `http(1)` which used this value in request
	**4. Template Context**
	when using `nuclei -jsonl` flag we get lot of data/metadata related to a vulnerability (ex: template details,extracted-values and much more) . A template context is nothing but a map/JSON containing all this data along with internal/unexported data that is only available at runtime (ex: extracted values from previous requests, variables added using `set()` etc). This template context is available in javascript as `template` variable and can be used to access any data from it. ex: `template["ssl_subject_cn"]` , `template["ssl_subject_an"]` etc
	```javascript
	template["ssl_subject_cn"] // returns value of ssl_subject_cn from template context which is available after executing ssl request
	template["ptrValue"] // returns value of ptrValue which was extracted using regex with internal: true
	```
	Lot of times we don't known what all data is available in template context and this can be easily found by printing it to stdout using `log()` function
	```javascript
	log(template)
	```
	**5. Log Helper Function**
	It is a nuclei js alternative to `console.log` and this pretty prints map data in readable format
	**Note:** This should be used for debugging purposed only as this prints data to stdout
	**6. Dedupe**
	Lot of times just having arrays/slices is not enough and we might need to remove duplicate variables . for example in earlier vhost enumeration we did not remove any duplicates as there is always a chance of duplicate values in `ssl_subject_cn` and `ssl_subject_an` and this can be achieved by using `dedupe()` object. This is nuclei js helper function to abstract away boilerplate code of removing duplicates from array/slice
	```javascript
	let uniq = new Dedupe(); // create new dedupe object
	uniq.Add(template["ptrValue"])
	uniq.Add(template["ssl_subject_cn"]);
	uniq.Add(template["ssl_subject_an"]);
	log(uniq.Values())
	```
	And that's it , this automatically converts any slice/array to map and removes duplicates from it and returns a slice/array of unique values
	------
	> Similar to DSL helper functions . we can either use built in functions available with `Javscript (ECMAScript 5.1)` or use DSL helper functions and its upto user to decide which one to uses

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[tarunKoyalwar]

TODO

• fix errors
• code restructuring
• validating availability of variables in template context (same as multiprotocol ??) ✅ ( response variables with proto prefix are now available in template context)
• add unit tests
• change function defination of protocol execution functions and return bool which is matcher-status
• add integration tests based on use cases
• fix missing multiprotocol implementation in code protocol
• add iterate built-in function to handle iterating over (string, slice , map) ✅ ( by default dynamic extracted values are stored in slice/array so there doesn't seem to be a need for this helper yet )
• ✅ added iterate built in function to iterate and handle empty / null values of slices/ strings etc
• deprecate iterate-all
• documentation (temporarily added in flow package it will be moved to appropriate location in new PR)

[ehsandeep]

• internal: true for matchers in multi-protocol ( added new option to accept map / json as input in function iteself ex:

http({hide: true})

• implicit matcher status in flow in case of no matchers used.

[tarunKoyalwar]

POC

• vhost enumeration using flow template and dns, ssl, http protocol

$ chaos -silent -d hackerone.com  | dnsx -silent -json | jq -r '. | .a | select( . != null ) | .[]' | sort -u | uniq | go run . -t ~/test-templates/vhost-enum-flow.yaml 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.1 (latest)
[INF] New templates added in latest release: 198
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 20
[INF] Running httpx on input host
[INF] Found 20 URL from httpx
[vhost-enum-flow] [http] [info] https://104.16.99.52/ [VHOST: www.hackerone.com, SC: 200, CL: 177235]
[vhost-enum-flow] [http] [info] https://104.16.99.52/ [VHOST: api.hackerone.com, SC: 200, CL: 8486]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://52.60.160.16/ [VHOST: ec2-52-60-160-16.ca-central-1.compute.amazonaws.com., SC: 404, CL: 9783]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: cdn-185-199-109-153.github.com., SC: 404, CL: 9623]
[vhost-enum-flow] [http] [info] https://52.60.165.183/ [VHOST: uberflip.com, SC: 404, CL: 9594]
[vhost-enum-flow] [http] [info] https://35.170.195.217/ [VHOST: ec2-35-170-195-217.compute-1.amazonaws.com., SC: 403, CL: 520]
[vhost-enum-flow] [http] [info] https://52.203.118.248/ [VHOST: freshdesk.com, SC: 404, CL: 1129]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://44.218.44.151/ [VHOST: ec2-44-218-44-151.compute-1.amazonaws.com., SC: 403, CL: 520]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://104.16.99.52/ [VHOST: hackerone.com, SC: 302, CL: 92]
[vhost-enum-flow] [http] [info] https://3.98.63.202/ [VHOST: ec2-3-98-63-202.ca-central-1.compute.amazonaws.com., SC: 404, CL: 9780]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: cdn-185-199-108-153.github.com., SC: 404, CL: 9623]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: www.github.com, SC: 404, CL: 9575]
[vhost-enum-flow] [http] [info] https://3.229.75.209/ [VHOST: ec2-3-229-75-209.compute-1.amazonaws.com., SC: 403, CL: 520]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://52.60.160.16/ [VHOST: uberflip.com, SC: 404, CL: 9588]
[vhost-enum-flow] [http] [info] https://52.60.165.183/ [VHOST: ec2-52-60-165-183.ca-central-1.compute.amazonaws.com., SC: 404, CL: 9797]
[vhost-enum-flow] [http] [info] https://185.199.109.153/ [VHOST: www.github.com, SC: 404, CL: 9575]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://52.203.118.248/ [VHOST: ec2-52-203-118-248.compute-1.amazonaws.com., SC: 403, CL: 520]
[vhost-enum-flow] [http] [info] https://35.170.195.217/ [VHOST: freshdesk.com, SC: 404, CL: 1127]
[vhost-enum-flow] [http] [info] https://3.98.63.202/ [VHOST: uberflip.com, SC: 404, CL: 9601]
[vhost-enum-flow] [http] [info] https://44.218.44.151/ [VHOST: freshdesk.com, SC: 404, CL: 1127]
[vhost-enum-flow] [http] [info] https://185.199.108.153/ [VHOST: www.github.com, SC: 404, CL: 9575]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: cdn-185-199-111-153.github.com., SC: 404, CL: 9623]
[vhost-enum-flow] [http] [info] https://3.229.75.209/ [VHOST: freshdesk.com, SC: 404, CL: 1127]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://52.60.160.16/ [VHOST: uberflip.com, SC: 404, CL: 9589]
[vhost-enum-flow] [http] [info] https://104.16.100.52/ [VHOST: hackerone.com, SC: 302, CL: 92]
[vhost-enum-flow] [http] [info] https://52.60.165.183/ [VHOST: uberflip.com, SC: 404, CL: 9594]
[vhost-enum-flow] [http] [info] https://185.199.111.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://104.16.100.52/ [VHOST: www.hackerone.com, SC: 200, CL: 177235]
[vhost-enum-flow] [http] [info] https://3.98.63.202/ [VHOST: uberflip.com, SC: 404, CL: 9589]
[vhost-enum-flow] [http] [info] https://104.16.100.52/ [VHOST: api.hackerone.com, SC: 200, CL: 8486]
[vhost-enum-flow] [http] [info] https://35.169.15.7/ [VHOST: ec2-35-169-15-7.compute-1.amazonaws.com., SC: 403, CL: 520]
[vhost-enum-flow] [http] [info] https://35.169.15.7/ [VHOST: freshdesk.com, SC: 404, CL: 1127]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: www.github.com, SC: 404, CL: 9575]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: githubusercontent.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: cdn-185-199-110-153.github.com., SC: 404, CL: 9623]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: github.io, SC: 301, CL: 0]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: github.com, SC: 404, CL: 9115]
[vhost-enum-flow] [http] [info] https://185.199.110.153/ [VHOST: github.com, SC: 404, CL: 9115]

[tarunKoyalwar]

example template using most/all features

id: vhost-enum-flow

info:
  name: vhost enum flow
  author: tarunKoyalwar
  severity: info
  description: |
        vhost enumeration by extracting potential vhost names from ssl certificate and dns ptr records

flow: |
  let uniq = new Dedupe();
  ssl();
  dns();
  uniq.Add(template["ptrValue"])
  uniq.Add(template["ssl_subject_cn"]);
  uniq.Add(template["ssl_subject_an"]);
  for (let vhost of uniq.Values()) {
    vhost = vhost.replace("*.", "")
    set("vhost", vhost);
    http();
  }  

ssl:
  - address: "{{Host}}:{{Port}}"

dns:
  - name: "{{FQDN}}"
    type: PTR

    extractors:
      - type: regex
        name: ptrValue
        internal: true
        group: 1
        regex:
          - "IN\tPTR\t(.+)" 

http:
  - raw:
      - |
        GET / HTTP/1.1
        Host: {{trim_suffix(vhost, ".")}}        

    matchers:
      - type: status
        negative: true
        status:
          - 400
          - 502

    extractors:
      - type: dsl
        dsl:
          - '"VHOST: " + vhost + ", SC: " + status_code + ", CL: " + content_length'

[tarunKoyalwar]

PR Changelog

• Flow support added
• resolved race condition in utils
• resolved race condition in tlsx (depends on fix data race in x509->response tlsx#331)
• Complete redesign of template Context to make it concurrency safe and more
• Complete redesign of template execution (replaced outdated implementation of common/executer
• Partial redesign of multiprotocol with context of template execution package tmplexec

[tarunKoyalwar]

Template for #3700

id: vpc-logging-not-enabled
info:
  name: VPC flow logging Not enabled
  author: princechaddha
  severity: low
  description: |
    Checks if the Amazon CloudWatch alarmed is enabled for Sign In without MFA
  reference:
    - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeVpcs.html
    - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeFlowLogs.html
  remediation: Enabling VPC Flow Logs
  tags: cloud,devops,aws,amazon,vpc,ec2,logging

self-contained: true
signature: aws
variables:
  service: ec2
  regions:
    - us-west-1
    - us-east-1

flow: |
  for(let region of iterate(template["regions"])){
      set("region",region);
      http("extract-vpcs");
      for(let vpcId of iterate(template["vpcs"])){
          set("vpcId",vpcId);
          http("extract-flow-logs");
      }
  }  

http:
  - id: extract-vpcs
    raw:
     - |
        POST https://{{service}}.{{region}}.amazonaws.com/ HTTP/1.1
        Host: {{service}}.{{region}}.amazonaws.com
        Content-Type: application/x-www-form-urlencoded; charset=utf-8
        User-Agent: aws-cli/2.9.15 Python/3.11.1 Darwin/22.3.0 source/arm64 prompt/off command/ec2.describe-vpcs

        Action=DescribeVpcs&Version=2016-11-15
    
    extractors:
      - type: regex
        name: vpcs
        part: body
        internal: true
        group: 1
        regex:
          - '<vpcId>([a-z-0-9]+)<\/vpcId>'

  - id: extract-flow-logs
    raw:
      - |
        POST https://{{service}}.{{region}}.amazonaws.com/ HTTP/1.1
        Host: {{service}}.{{region}}.amazonaws.com
        Content-Type: application/x-www-form-urlencoded; charset=utf-8
        User-Agent: aws-cli/2.9.15 Python/3.11.1 Darwin/22.3.0 source/arm64 prompt/off command/ec2.describe-flow-logs

        Action=DescribeFlowLogs&Version=2016-11-15&Filter.1.Name=resource-id&Filter.1.Value.1={{vpcId}}

    iterate-all: true
    matchers-condition: and
    matchers:
      - type: word
        part: body
        words:
          - "</DescribeFlowLogsResponse>"

      - type: word
        words:
          - "<flowLogId>"
        negative: true

    extractors:
      - type: dsl
        dsl:
          - '"VPC flow logging is not enabled for " + vpcId + " in " + region + " region"'

Demo

Without Override from Cli

$  go run . -t ~/test-templates/vpc-logging-not-enabled.yaml                               

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.1 (latest)
[INF] New templates added in latest release: 198
[INF] Templates loaded for current scan: 1
[vpc-logging-not-enabled] [http] [low] https://ec2.us-west-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-0a7b0a4148d2b5c14 in us-west-1 region]
[vpc-logging-not-enabled] [http] [low] https://ec2.us-east-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-0cc801e39bf787771 in us-east-1 region]
[vpc-logging-not-enabled] [http] [low] https://ec2.us-east-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-072e8d17809966803 in us-east-1 region]

With Override from CLI using -var flag

$ go run . -t ~/test-templates/vpc-logging-not-enabled.yaml -V "regions=regions.txt" -lfa 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.0-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.0.0-dev (development)
[INF] Current nuclei-templates version: v9.6.1 (latest)
[INF] New templates added in latest release: 198
[INF] Templates loaded for current scan: 1
[vpc-logging-not-enabled] [http] [low] https://ec2.us-east-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-072e8d17809966803 in us-east-1 region]
[vpc-logging-not-enabled] [http] [low] https://ec2.us-east-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-0cc801e39bf787771 in us-east-1 region]
[vpc-logging-not-enabled] [http] [low] https://ec2.us-west-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-0a7b0a4148d2b5c14 in us-west-1 region]
[vpc-logging-not-enabled] [http] [low] https://ec2.ap-south-1.amazonaws.com/ [VPC flow logging is not enabled for vpc-047617a9feaf980c0 in ap-south-1 region]

regions.txt

us-east-1
us-west-1
ap-south-1

[tarunKoyalwar]

created follow up issue for adding for request annotation as suggested by @Mzack9999

• for request annotation support in http protocol #4081

[Mzack9999]

lgtm!

Few observations:

• The mechanism for scoping context variables via set seems a bit counter-intuitive, as the variable is scoped instead for the whole template. Quoting Advanced Scripting Engine #641 (comment) I think a declarative approach is more intuitive and readable, from the previous work:

args = {
    'server': server,
}
http(args);
http("another-id", args)

Regardless, awesome job!