Posted in scanning on January 3, 2023 by Jonathan Walker ‐ 10 min read
When new security research is out, the spin up time to scour assets for vulnerabilities can be a long and tedious task. Spending time learning about the latest findings, how to exploit them, and what conditions are required in order to exploit them. How can you stay on top of it all when it is a constant battle repeating itself?
That is the exact problem projects like Nuclei are made for; To help researchers identify known issues through a powerful templating language to ensure you do not miss out. It can also help you identify issues such as known subdomain takeovers, exposed panels, network services, misconfigurations, exposed files, and the list goes on.
It does an outstanding job when you have a limited number of assets and are performing scans individually. Though the moment you have hundreds or thousands of assets, scaling vertically is no longer an option.
Performing scanning at scale is exactly what Nuclear Pond is meant to achieve. You can launch thousands of scans without having to worry about cost, waiting for extended periods of time, and customize how many scans you want to perform in parallel for far less than a cup of coffee. Once the scans are complete you can choose to upload them to S3 for querying with Athena or just view the output as if it were running on your own machine. These scans can launch hundreds of instances of Nuclei all at the same time on the cheapest compute available on AWS with Lambda.
Scans can help you visualize your attack surface and vulnerabilities such as:
- Identify assets vulnerable to subdomain takeovers
- Provide a comprehensive identification on exposed assets such as Jenkins, Elasticsearch, Grafana, and many more
- Locate exposed files, logs, configurations, backups, etc.
- Misconfigurations that can lead to information disclosure
- Network services such as ssh, ftp, telnet, mysql, etc.
- Known vulnerabilities in exposed assets
- Published CVEs that can be easily exploited
What is Nuclei
Nuclei is a tool that allows you to create configuration files to validate a wide variety of security issues. Contributing to Nuclei and Nuclei Templates is a breeze. When running nuclei, you can specify what security issues you are trying to identify and launch a scan against your hosts. This can often be slow on your local machine even with rate limits and concurrency settings configured. While they have done an outstanding job creating a fast scanner, doing so at scale can be difficult. Lets take a deep dive into each component of Nuclei.
ProjectDiscovery maintains the repository nuclei-templates which contains various templates for the nuclei scanner provided by them and the community. Contributions are welcome and straight forward to add to based on previous examples and you can reference my pull request to get a sense of just how easy it is.
This is to help you understand what scans you can perform with Nuclei and the options available to you.
- Filter by tags
-tags takeoverallows you to identify known takeovers
- Filter by templates directory
-t dnsexecutes all templates under dns directory
- Filter by specific template
- Exclude by tags
-etags cveexcludes searching for vulnerabilities
Detecting Network Services
Here is an example in which we want to enumerate some protocols running on scanme.nmap.org with the network detection templates. This will help us identify network services running on the specified host. This allows us to currently check over 40 different network based services such as ssh, smtp, mysql, mongodb, telnet, etc.
$ nuclei -u scanme.nmap.org -t network/detection __ _ ____ __ _______/ /__ (_) / __ \/ / / / ___/ / _ \/ / / / / / /_/ / /__/ / __/ / /_/ /_/\__,_/\___/_/\___/_/ v2.8.3 projectdiscovery.io [INF] Using Nuclei Engine 2.8.3 (latest) [INF] Using Nuclei Templates 9.3.2 (latest) [INF] Templates added in last update: 57 [INF] Templates loaded for scan: 42 [INF] Targets loaded for scan: 1 [openssh-detect] [network] [info] scanme.nmap.org:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13]
Takeovers can be a common occurrence when you manage thousands of zones within your infrastructure and mistakes certainly occur in which deprecating assets may not complete in the correct order or completely. This can lead to dangling assets that can be taken over by an attacker. The repository Can I take over XYZ is an excellent resource if you want to learn what the current landscape looks like at this time.
Nuclei currently has over 70 different templates to detect if you are currently vulnerable to a takeover and here is an example as to how check to see if a domain is vulnerable.
$ nuclei -u https://jsdkjfskjsfdkdjfds.s3.amazonaws.com/ -tags takeover __ _ ____ __ _______/ /__ (_) / __ \/ / / / ___/ / _ \/ / / / / / /_/ / /__/ / __/ / /_/ /_/\__,_/\___/_/\___/_/ v2.8.3 projectdiscovery.io [INF] Using Nuclei Engine 2.8.3 (latest) [INF] Using Nuclei Templates 9.3.3 (latest) [INF] Templates added in last update: 238 [INF] Templates loaded for scan: 74 [INF] Targets loaded for scan: 1 [INF] Templates clustered: 69 (Reduced 68 HTTP Requests) [aws-bucket-takeover] [http] [high] https://jsdkjfskjsfdkdjfds.s3.amazonaws.com/
Think of Nuclear Pond as just a way for you to run Nuclei in the cloud. You can use it just as you would on your local machine but run them in parallel and with however many hosts you want to specify. All you need to think of is the nuclei command line flags you wish to pass to it.
Setup & Installation
To install Nuclear Pond, you need to configure the backend terraform module. You can do this by running
terraform apply or leveraging terragrunt. After your backend infrastructure has been created, it’s time for you to install
$ go install github.com/DevSecOpsDocs/nuclearpond@latest
Command line flags
The most important flags are as follows. These are flags for
nuclearpond run which must proceed each of the following.
-bwhich commands how many hosts per invocation(number of hosts / batches = nuclei lambda invocations)
-oflag allows you to specify outputs such as
cmdfor the output of Nuclei and
s3for data lake analysis of findings
-cflag allows you to specify how many threads to invoke lambda (1 is the default but >10 is recommended at scale)
-a $(echo -ne "-t dns" | base64)flag allows you to pass
-t dnsto Nuclei
-fis your backend function name and
-ris your region you have deployed the function to
- The environment variables below can replace
The backend configures a Lambda function which includes the Nuclei binary within a layer which is located in
/opt/nuclei. The function accepts an event with your targets, arguments, and output type. Nuclear Pond allows you to invoke this lambda function by taking your targets, arguments, and output in parallel by splitting up your targets into batches.
- Maximum execution time of fifteen minutes which if combined with
-cNuclei flags should not be an issue
- You should avoid using the
-o file.txtas that file remains in lambda
- Since the input includes raw flags being sent to exec it is vulnerable to RCE
- Be careful making modifications to the infrastructure, such as adding network interfaces, as it could increase risk
While I strongly recommend against not including filters for your scan and running it against all templates, it can be done within a couple of minutes with
-rl 1000 -c 50 which can potentially bring down your target. So use caution and always make sure you have permission to do so. This tool is primarily built for a targeted approach.
$ nuclearpond run -t devsecopsdocs.com -a $(echo -ne "-rl 1000 -c 50 -silent" | base64) -o cmd _ _ _ ____ _ | \ | | _ _ ___ | | ___ __ _ _ __ | _ \ ___ _ __ __| | | \| | | | | | / __| | | / _ \ / _` | | '__| | |_) | / _ \ | '_ \ / _` | | |\ | | |_| | | (__ | | | __/ | (_| | | | | __/ | (_) | | | | | | (_| | |_| \_| \__,_| \___| |_| \___| \__,_| |_| |_| \___/ |_| |_| \__,_| devsecopsdocs.com 2023/01/03 10:15:07 Running nuclei against the target devsecopsdocs.com 2023/01/03 10:15:07 Running with 1 threads 2023/01/03 10:17:02 Scan complete with output: [aws-cloudfront-service] [http] [info] https://devsecopsdocs.com [aws-bucket-service] [http] [info] https://devsecopsdocs.com [xss-deprecated-header] [http] [info] https://devsecopsdocs.com [1; mode=block] [robots-txt-endpoint] [http] [info] https://devsecopsdocs.com/robots.txt [nameserver-fingerprint] [dns] [info] devsecopsdocs.com [ns-1309.awsdns-35.org.,ns-1822.awsdns-35.co.uk.,ns-487.awsdns-60.com.,ns-579.awsdns-08.net.] [s3-detect] [http] [info] https://devsecopsdocs.com/%c0 [tls-version] [ssl] [info] devsecopsdocs.com [tls13] 2023/01/03 10:17:02 Completed all parallel operations in 1m54.972950992s , best of luck!
Data Lake Output
This output is recommended when leveraging Nuclear Pond as once the script invokes, all of the work is handed off to the cloud for you to analyze another time. This output is known as
s3 and you can output it by specifying
-o s3. You can also specify
-l targets.txt and
-b 10 to invoke the lambda functions in batches of 10 targets per execution.
$ nuclearpond run -t devsecopsdocs.com -a $(echo -ne "-t dns -silent" | base64) -o s3 _ _ _ ____ _ | \ | | _ _ ___ | | ___ __ _ _ __ | _ \ ___ _ __ __| | | \| | | | | | / __| | | / _ \ / _` | | '__| | |_) | / _ \ | '_ \ / _` | | |\ | | |_| | | (__ | | | __/ | (_| | | | | __/ | (_) | | | | | | (_| | |_| \_| \__,_| \___| |_| \___| \__,_| |_| |_| \___/ |_| |_| \__,_| devsecopsdocs.com 2023/01/03 10:22:12 Running nuclei against the target devsecopsdocs.com 2023/01/03 10:22:12 Running with 1 threads 2023/01/03 10:22:13 Saved results in s3://test-nuclei-runner-artifacts/findings/2023/01/03/18/nuclei-findings-cd17c344-ec06-48da-96d6-728debf01c57.json 2023/01/03 10:22:13 Completed all parallel operations in 1.165510457s , best of luck!
Scanning at Scale
Now lets run Nuclear Pond as it was intended to do, at scale on a significant amount of targets. Here I have around 500k targets, decided to batch 2k targets per execution with
-b 2000, and run 200 individual threads locally with
-c 200 to run lambda functions asynchronously.
$ nuclearpond run -l ~/Desktop/500k.txt -a $(echo -ne "-t dns/mx-fingerprint.yaml -rl 1000 -c 50 -silent" | base64) -o s3 -b 2000 -c 200 _ _ _ ____ _ | \ | | _ _ ___ | | ___ __ _ _ __ | _ \ ___ _ __ __| | | \| | | | | | / __| | | / _ \ / _` | | '__| | |_) | / _ \ | '_ \ / _` | | |\ | | |_| | | (__ | | | __/ | (_| | | | | __/ | (_) | | | | | | (_| | |_| \_| \__,_| \___| |_| \___| \__,_| |_| |_| \___/ |_| |_| \__,_| devsecopsdocs.com 2023/01/03 10:25:51 Running nuclear pond against 580880 targets 2023/01/03 10:25:51 Splitting targets into 291 individual executions 2023/01/03 10:25:51 Running with 200 threads 2023/01/03 10:25:58 Saved results in s3://test-nuclei-runner-artifacts/findings/2023/01/03/18/nuclei-findings-dd51d033-40fa-46ba-80a1-6b95803aed18.json ... 2023/01/03 10:26:27 Saved results in s3://test-nuclei-runner-artifacts/findings/2023/01/03/18/nuclei-findings-0745ab34-1c2e-451b-8736-8aea93a5ae41.json 2023/01/03 10:26:27 Completed all parallel operations in 36.215408938s , best of luck!
To explore your findings in Athena all you need to do is perform the following query! The database and the table should already be available to you. You may also have to configure query results if you have not done so already. Once you are comfortable with querying Athena, it would be best to move over to help you visualize your results such as grafana.
select * from nuclei_db.findings_db limit 10;
In order to get down into queries a little deeper, I thought I would give you a quick example. In the select statement we drill down into
"matched-at" column must be in double quotes due to
- character, and you are searching only for high and critical findings generated by Nuclei.
SELECT info.name, host, type, info.severity, "matched-at", info.description, template, dt FROM "nuclei_db"."findings_db" where host like '%devsecopsdocs.com' and info.severity in ('high','critical')
The backend infrastructure, all within terraform module. I would strongly recommend reading the readme associated to it as it will have some important notes.
- Lambda function
- S3 bucket
- Stores nuclei binary
- Stores configuration files
- Stores findings
- Glue Database and Table
- Allows you to query the findings in S3
- IAM Role for Lambda Function
What are you waiting for? Get started today. Contributions are welcome and looking forward to seeing issues created!