IaC Scan Runner
The IaC Scan Runner is a REST API service used to scan IaC (Infrastructure as Code) package by performing various checks in order to find possible errors and improvements. The IaC Scan Runner’s source code is available in xlab-si/iac-scan-runner GitHub repository.
Scanner and check reference
The scanner is the main component of the IaC Scan Runner and it initiates the scanning process, which makes the supplied IaC go through multiple checks.
IaC Scan Runner currently supports the following IaC checks that can be executed as part of one IaC scan:
IaC Check |
Target IaC entity |
Enabled (by default) |
Needs configuration |
|---|---|---|---|
TOSCA |
yes |
no |
|
Ansible |
yes |
no |
|
Ansible |
no |
yes |
|
Terraform |
yes |
no |
|
Terraform |
yes |
no |
|
Terraform |
yes |
no |
|
YAML |
yes |
no |
|
Python |
yes |
no |
|
Python |
yes |
no |
|
Python packages |
yes |
no |
|
Git repositories |
yes |
no |
|
Git repositories |
yes |
no |
|
Markdown files |
yes |
no |
|
Docker |
yes |
no |
|
Nginx configuration |
yes |
no |
|
Shell scripts |
yes |
no |
|
JavaScript |
yes |
no |
|
TypeScript |
yes |
no |
|
HTML |
yes |
no |
|
CSS and other styles |
yes |
no |
|
Java |
yes |
no |
|
Multiple components |
yes |
no |
|
Multiple components |
no |
yes |
|
Multiple components |
no |
yes |
The following subsections explain the necessary API actions for each check.
xOpera TOSCA validation
xOpera is an orchestration project that includes xOpera TOSCA orchestrator called opera - a lightweight
TOSCA orchestrator, which uses a separate component called xOpera TOSCA YAML parser. This opera-tosca-parser
Python library (also comes with CLI) is able to parse and validate OASIS TOSCA templates and CSAR files
(also see xOpera validate check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Ansible Lint
Ansible Lint is a command-line tool for linting playbooks, roles and collections aimed towards any Ansible users (see Ansible Lint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Steampunk Scanner
Steampunk Scanner (developed by XLAB Steampunk) is a quality scanner for Ansible playbooks, where you can scan the playbook and get an instant quality score with tips on how to improve it. The steampunk-scanner CLI enables the use of Ansible scanner from the console with the ability to scan Ansible task files, playbooks, roles and collections (see Steampunk Scanner check).
Check ID (from the API) |
|
Enabled (by default) |
no |
Configured (by default) |
no |
Documentation |
TFLint
TFLint is a a pluggable Terraform linter.
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
tfsec
tfsec is a security scanner for your Terraform code (see tfsec check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Terrascan
Terrascan is a static code analyzer for IaC and defaults to scanning Terraform (see Terrascan check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
yamllint
yamllint is a linter for YAML files that checks for syntax validity, key repetition and cosmetic problems such as lines length, trailing spaces, indentation, etc. (see yamllint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Pylint
Pylint is a Python static code analysis tool that checks for errors in Python code, tries to enforce a coding standard and looks for code smells (see Pylint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Bandit
Bandit is a tool designed to find common security issues in Python code (see Bandit check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Safety
Safety is a is a PyUp CLI tool that checks your installed Python dependencies for known security vulnerabilities (see PyUp Safety check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Gitleaks
Gitleaks is a SAST tool for detecting hardcoded secrets like passwords, API keys, and tokens in Git repos (see Gitleaks check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
git-secrets
git-secrets is a tool that prevents you from committing secrets and credentials into Git repositories (see git-secrets check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Markdown lint
Markdown lint is a tool to check markdown files and flag style issues (see Markdown lint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
hadolint
hadolint is a Dockerfile linter (see hadolint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Gixy
Gixy is a tool to analyze Nginx configuration (see Gixy check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
ShellCheck
stylelint is a static analysis tool for shell scripts (see ShellCheck check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
ESLint
ESLint is a tool for identifying and reporting on patterns found in ECMAScript/JavaScript code (see ESLint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
TypeScript ESLint
TypeScript ESLint enables ESLint to support TypeScript (see TypeScript ESLint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
HTMLHint
HTMLHint is the static code analysis tool you need for your HTML (see HTMLHint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
stylelint
stylelint is a mighty, modern linter that helps you avoid errors and enforce conventions in your styles (see stylelint check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Checkstyle
Checkstyle is a tool for checking Java source code for adherence to a Code Standard or set of validation rules (see Checkstyle check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
cloc
cloc counts blank lines, comment lines, and physical lines of source code in many programming languages (see cloc check).
Check ID (from the API) |
|
Enabled (by default) |
yes |
Configured (by default) |
yes |
Documentation |
Snyk
Snyk helps you find, fix and monitor known vulnerabilities in open source (see Snyk check).
Check ID (from the API) |
|
Enabled (by default) |
no |
Configured (by default) |
no |
Documentation |
SonarScanner
SonarScanner is the official scanner used to run code analysis on SonarQube and SonarCloud (see SonarScanner check).
Check ID (from the API) |
|
Enabled (by default) |
no |
Configured (by default) |
no |
Documentation |
Tip
If you do not wish to supply your user token within the config file, pass it as a secret in the API.
REST API
This section focuses on the IaC Scan Runner REST API service.
Installation
You can run the REST API using a public xscanner/runner Docker image as follows:
# run IaC Scan Runner REST API in a Docker container and
# navigate to localhost:8080/swagger or localhost:8080/redoc
docker run --name iac-scan-runner -p 8080:80 xscanner/runner
Tip
Other methods of running are also explained in xlab-si/iac-scan-runner GitHub repository.
Note
The Docker image is large because we use many different tools for scanning. For the future releases. we will try to shrink it down as much as possible.
Usage
After the setup you will see that the OpenAPI Specification and interactive Swagger UI API documentation are
available on /swagger, whereas ReDoc generated API reference documentation is accessible on /redoc.
You can also retrieve an OpenAPI document that conforms to the OpenAPI Specification as JSON file on
/openapi.json or as YAML file on /openapi.yaml (or /openapi.yml).
The IaC Scan Runner API can be used to interact with the main IaC inspection component and initialize IaC scans. The API includes various IaC checks that can be filtered and configured. User can choose to execute all or just the selected checks as a part of one IaC scan. After the scanning process the API will return all the check results.
REST API endpoint |
Description |
|---|---|
Retrieve and filter checks |
|
Enable check for running |
|
Disable check for running |
|
Configure check |
|
Initiate IaC scan |
The API endpoints are further described below.
- GET /checks
This endpoint lets you retrieve and filter the supported IaC checks. You can filter checks by their keynames (use the keyword request parameter) and find out whether they are already enabled (set the enabled parameter) or configured (set the configured parameter). Checks can also be filtered by their target entity (set the target_entity_type parameter) - here we have three types of checks - IaC (they only check the code), component (they check IaC requirements and dependencies in order to find vulnerabilities) and check that are both IaC and component. Each IaC check in the API has its unique name so that it can be distinguished from other checks. When no filter is specified, the endpoint lists all IaC checks.
Example request:
$ curl -X 'GET' 'http://127.0.0.1:8000/checks?keyword=Terraform&enabled=true'
import requests URL = 'http://127.0.0.1:8000/checks?keyword=Terraform&enabled=true' response = requests.get(URL) print(response.json())
Example response:
[ { "name": "tflint", "description": "A Pluggable Terraform Linter", "enabled": true, "configured": true, "target_entity_type": "IaC" }, { "name": "tfsec", "description": "Security scanner for your Terraform code", "enabled": true, "configured": true, "target_entity_type": "IaC" }, { "name": "terrascan", "description": "Terrascan is a static code analyzer for IaC (defaults to scanning Terraform)", "enabled": true, "configured": true, "target_entity_type": "IaC" } ]
- Query Parameters
keyword (string) – optional keyword from check name or description
enabled (boolean) – search for checks that are enabled or not
configured (string) – search for checks that are configured or not
target_entity_type (string) – search by target entity (one of
IaC,component,IaC and component)
- Status Codes
200 OK – Successful Response
404 Not Found – Bad Request
422 Unprocessable Entity – Validation Error
- PUT /checks/{check_name}/enable
IaC checks can be enabled (can be used for scanning) or disabled (cannot be used for scanning). Most of the local checks are enabled by default and some of them that are advanced, take longer time or require additional configuration are disabled and have to be enabled before the scanning. This endpoint can be used to enable a specific IaC check (selected by the check_name parameter), which means that it will become available for running within IaC scans.
Example request:
$ curl -X 'PUT' 'http://127.0.0.1:8000/checks/snyk/enable'
import requests URL = 'http://127.0.0.1:8000/checks/snyk/enable' response = requests.put(URL) print(response.json())
Example response:
"Check: snyk is now enabled and available to use."- Parameters
check_name (string) – check that you want to enable for running
- Status Codes
200 OK – Successful Response
400 Bad Request – Bad Request
422 Unprocessable Entity – Validation Error
- PUT /checks/{check_name}/disable
This endpoint can be used to disable a specific IaC check (selected by the check_name parameter), which means that it will become unavailable for running within IaC scans.
Example request:
$ curl -X 'PUT' 'http://127.0.0.1:8000/checks/pylint/disable'
import requests URL = 'http://127.0.0.1:8000/checks/pylint/enable' response = requests.put(URL) print(response.json())
Example response:
"Check: pylint is now disabled and cannot be used."- Parameters
check_name (string) – check that you want to disable for running
- Status Codes
200 OK – Successful Response
400 Bad Request – Bad Request
422 Unprocessable Entity – Validation Error
- PUT /checks/{check_name}/configure
This endpoint is used to configure a specific IaC check (selected by the check_name parameter). Most IaC checks do not need configuration as they already use their default settings. However, some of them - especially the remote service checks (such as Snyk) require to be configured before using them within IaC scans. Some checks will have to be enabled before they can be configured. The configuration of IaC check takes two optional multipart request body parameters - config_file and secret. The former (config_file) can be used to pass a check configuration file (which is supported by almost every check) that is specific to every check and will override the default check settings. The latter (secret) is meant for passing sensitive data such as passwords, API keys, tokens, etc. These secrets are often used to configure the remote service checks - usually to authenticate the user via some token that has been generated in the remote service user profile settings. Some IaC checks support both the aforementioned request body parameters and some support one of them or none. The API will warn you in case of any configuration problems.
Example request:
$ curl -X 'PUT' 'http://127.0.0.1:8000/checks/sonar-scanner/configure' -H 'Content-Type: multipart/form-data' -F 'config_file=@sonar-project.properties;type=text/plain' -F 'secret=56bf-example-token-f007'
import requests URL = 'http://127.0.0.1:8000/checks/sonar-scanner/configure' multipart_form_data = { 'config_file': ('sonar-project.properties', open('/path/to/sonar-project.properties', 'rb')), 'secret': (None, '56bf-example-token-f007') } response = requests.put(URL, files=multipart_form_data) print(response.json())
Example response:
"Check: sonar-scanner has been configured successfully."- Parameters
check_name (string) – check that you want to configure before scanning
- Form Parameters
config_file – optional check configuration file
secret – optional secret for configuration (password, API token, etc.)
- Status Codes
200 OK – Successful Response
400 Bad Request – Bad Request
422 Unprocessable Entity – Validation Error
Warning
Be careful not to expose your secrets directly in your IaC.
- POST /scan
This is the main endpoint that is used to scan the IaC and gather the results from the executed IaC checks. The request body is treated as multipart (multipart/form-data type) and has two parameters. The first one is iac and is required. Here, the user passes his (compressed) IaC package (currently limited to zip or tar). The second parameter is checks and is an optional array of checks, which the user wants to executed as a part of his IaC scan. The IaC checks are selected by their unique names. If the user does not specify that field, all the enabled checks are executed. The API will warn you if there are any nonexistent, disabled or un-configured checks that you wanted to use. After the scanning process the API will return results of all checks (their outputs and return codes).
Example request:
$ curl -X 'POST' 'http://127.0.0.1:8000/scan' -H 'Content-Type: multipart/form-data' -F 'iac=@scaling-example.zip' -F 'checks=bandit,ansible-lint'
import requests URL = 'http://127.0.0.1:8000/scan' multipart_form_data = { 'iac': ('scaling-example.zip', open('/path/to/scaling-example.zip', 'rb')), 'checks': (None, 'bandit,ansible-lint') } response = requests.put(URL, files=multipart_form_data) print(response.json())
Example response:
{ "bandit": { "output": "[main]\tINFO\tprofile include tests: None\n[main]\tINFO\tprofile exclude tests: None\n[main]\tINFO\tcli include tests: None\n[main]\tINFO\tcli exclude tests: None\n[main]\tINFO\trunning on Python 3.8.10\nRun started:2021-08-25 11:23:29.960356\n\nTest results:\n\tNo issues identified.\n\nCode scanned:\n\tTotal lines of code: 0\n\tTotal lines skipped (#nosec): 0\n\nRun metrics:\n\tTotal issues (by severity):\n\t\tUndefined: 0\n\t\tLow: 0\n\t\tMedium: 0\n\t\tHigh: 0\n\tTotal issues (by confidence):\n\t\tUndefined: 0\n\t\tLow: 0\n\t\tMedium: 0\n\t\tHigh: 0\nFiles skipped (0):\n", "rc": 0 }, "ansible-lint": { "output": "WARNING Listing 6 violation(s) that are fatal\n\u001b[34mservice.yaml\u001b[0m:32: \u001b[91myaml\u001b[0m \u001b[2mtoo many spaces inside braces\u001b[0m \u001b[2;91m(braces)\u001b[0m\n\u001b[34mservice.yaml\u001b[0m:32: \u001b[91myaml\u001b[0m \u001b[2mtoo many spaces inside brackets\u001b[0m \u001b[2;91m(brackets)\u001b[0m\n\u001b[34mservice.yaml\u001b[0m:35: \u001b[91myaml\u001b[0m \u001b[2mtoo many spaces inside braces\u001b[0m \u001b[2;91m(braces)\u001b[0m\n\u001b[34mservice.yaml\u001b[0m:35: \u001b[91myaml\u001b[0m \u001b[2mtoo many spaces inside brackets\u001b[0m \u001b[2;91m(brackets)\u001b[0m\n\u001b[34mservice.yaml\u001b[0m:45: \u001b[91myaml\u001b[0m \u001b[2mtoo many spaces inside brackets\u001b[0m \u001b[2;91m(brackets)\u001b[0m\n\u001b[34mservice.yaml\u001b[0m:62: \u001b[91myaml\u001b[0m \u001b[2mtoo many spaces inside brackets\u001b[0m \u001b[2;91m(brackets)\u001b[0m\nYou can skip specific rules or tags by adding them to your configuration file:\n\u001b[2m# .ansible-lint\u001b[0m\n\u001b[94mwarn_list\u001b[0m: \u001b[2m# or 'skip_list' to silence them completely\u001b[0m\n - yaml \u001b[2m# Violations reported by yamllint\u001b[0m\n\nFinished with \u001b[1;36m6\u001b[0m \u001b[1;35mfailure\u001b[0m\u001b[1m(\u001b[0ms\u001b[1m)\u001b[0m, \u001b[1;36m0\u001b[0m \u001b[1;35mwarning\u001b[0m\u001b[1m(\u001b[0ms\u001b[1m)\u001b[0m on \u001b[1;36m9\u001b[0m files.\n", "rc": 2 } }
- Parameters
scan_response_type (string) – JSON (default) or YAML scan response
- Form Parameters
iac – IaC file (currently limited to zip or tar)
checks – optional array of the selected checks
- Status Codes
200 OK – Successful Response
400 Bad Request – Bad Request
422 Unprocessable Entity – Validation Error
Note
All API endpoints try to use JSON responses.
CLI
The IaC Scan Runner CLI enables easier setup of IaC Scan Runner in console environments.
Prerequisites
The Scan Runner CLI requires Python 3 and a virtual environment. In a typical modern Linux environment, we should already be set. In Ubuntu, however, we might need to run the following commands:
$ sudo apt update
$ sudo apt install -y python3-venv python3-wheel python-wheel-common
Installation
IaC Scan Runner CLI is distributed as Python iac-scan-runner package that is regularly published on PyPI.
The simplest way to test iac-scan-runner is to install it into virtual environment:
$ mkdir ~/iac-scan-runner && cd ~/iac-scan-runner
$ python3 -m venv .venv && . .venv/bin/activate
(.venv) $ pip install --upgrade pip
(.venv) $ pip install iac-scan-runner
The development version of the package is available on TestPyPI and the installation goes as follows.
(.venv) $ pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ iac-scan-runner
Commands
iac-scan-runner currently allows users to execute the following shell commands:
CLI command |
Purpose and description |
|---|---|
|
print OpenAPI Specification |
|
install the IaC Scan Runner prerequisites |
|
run the IaC Scan Runner REST API |
Tip
All the CLI commands are equipped with -h/--help option to help you.
Note
If you have any problems with IaC Scan Runner please have a look at the existing GitHub issues in xlab-si/iac-scan-runner/issues or open a new one yourself.