Skip to content

G
gitlab-ce
Commit 42234def authored by Russell Dickenson's avatar Russell Dickenson
Browse files
Options

Merge branch '334578-document-new-variable-in-dast-api-docs' into 'master' 

Document OVERRIDES_CMD_VERBOSE env var for API Fuzzing and DAST API

See merge request gitlab-org/gitlab!74556
parents 8814eae6 ec851be7
Show whitespace changes
Inline Side-by-side
Showing with 361 additions and 20 deletions
doc/user/application_security/api_fuzzing/index.md
View file @ 42234def
......@@ -461,7 +461,7 @@ Follow these steps to provide the bearer token with `FUZZAPI_OVERRIDES_ENV`:
```
1. To validate that authentication is working, run an API fuzzing test and review the fuzzing logs
and the test API's application logs.
and the test API's application logs. See the [overrides section](#overrides) for more information about override commands.
##### Token generated at test runtime
......@@ -495,7 +495,7 @@ variables:
FUZZAPI_PROFILE: Quick
FUZZAPI_OPENAPI: test-api-specification.json
FUZZAPI_TARGET_URL: http://test-deployment/
FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
```
To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and
......@@ -537,7 +537,7 @@ variables:
FUZZAPI_PROFILE: Quick-10
FUZZAPI_OPENAPI: test-api-specification.json
FUZZAPI_TARGET_URL: http://test-deployment/
FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_CMD: renew_token.py
FUZZAPI_OVERRIDES_INTERVAL: 300
```
......@@ -577,6 +577,9 @@ profile increases as the number of tests increases.
|[`FUZZAPI_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. |
|[`FUZZAPI_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. |
|[`FUZZAPI_OVERRIDES_CMD`](#overrides) | Overrides command. |
|[`FUZZAPI_OVERRIDES_CMD_VERBOSE`](#overrides) | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. |
|`FUZZAPI_PRE_SCRIPT` | Run user command or script before scan session starts. |
|`FUZZAPI_POST_SCRIPT` | Run user command or script after scan session has finished. |
|[`FUZZAPI_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). |
|[`FUZZAPI_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. |
|[`FUZZAPI_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. |
......@@ -756,7 +759,7 @@ variables:
FUZZAPI_PROFILE: Quick
FUZZAPI_OPENAPI: test-api-specification.json
FUZZAPI_TARGET_URL: http://test-deployment/
FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
```
#### Using a CI/CD variable
......@@ -801,16 +804,28 @@ variables:
If the value must be generated or regenerated on expiration, you can provide a program or script for
the API fuzzer to execute on a specified interval. The provided script runs in an Alpine Linux
container that has Python 3 and Bash installed. If the Python script requires additional packages,
it must detect this and install the packages at runtime. The script creates the overrides JSON file
as defined above.
container that has Python 3 and Bash installed.
You have to set the environment variable `FUZZAPI_OVERRIDES_CMD` to the program or script you would like
to execute. The provided command creates the overrides JSON file as defined previously.
You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for
your overrides command. In this case, we recommend setting the `FUZZAPI_PRE_SCRIPT` to the file path of a script which
provides those prerequisites. The script provided by `FUZZAPI_PRE_SCRIPT` is executed once, before the analyzer starts.
See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management)
page for information about installing Alpine Linux packages.
You must provide three CI/CD variables, each set for correct operation:
- `FUZZAPI_OVERRIDES_FILE`: File generated by the provided command.
- `FUZZAPI_OVERRIDES_CMD`: Command to generate JSON file.
- `FUZZAPI_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically.
- `FUZZAPI_OVERRIDES_INTERVAL`: Interval in seconds to run command.
Optionally:
- `FUZZAPI_PRE_SCRIPT`: Script to install runtimes or dependencies before the analyzer starts.
```yaml
stages:
- fuzz
......@@ -822,11 +837,167 @@ variables:
FUZZAPI_PROFILE: Quick
FUZZAPI_OPENAPI: test-api-specification.json
FUZZAPI_TARGET_URL: http://test-deployment/
FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_CMD: renew_token.py
FUZZAPI_OVERRIDES_INTERVAL: 300
```
#### Debugging overrides
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8.
By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `FUZZAPI_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing.
It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and follow a naming convention.
Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished.
Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Please notice two things in the script:
- Log file is saved in the location indicated by the environment variable `CI_PROJECT_DIR`.
- Log file name should match `gl-*.log`.
```python
#!/usr/bin/env python
# Example of an overrides command
# Override commands can update the overrides json file
# with new values to be used. This is a great way to
# update an authentication token that will expire
# during testing.
import logging
import json
import os
import requests
import backoff
# [1] Store log file in directory indicated by env var CI_PROJECT_DIR
working_directory = os.environ['CI_PROJECT_DIR']
# [2] File name should match the pattern: gl-*.log
log_file_path = os.path.join(working_directory, 'gl-user-overrides.log')
# Set up logger
logging.basicConfig(filename=log_file_path, level=logging.DEBUG)
# Use `backoff` decorator to retry in case of transient errors.
@backoff.on_exception(backoff.expo,
(requests.exceptions.Timeout,
requests.exceptions.ConnectionError),
max_time=30)
def get_auth_response():
return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD']))
# In our example, access token is retrieved from a given endpoint
try:
# Performs a http request, response sample:
# { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" }
response = get_auth_response()
# Check that the request is successful. may raise `requests.exceptions.HTTPError`
response.raise_for_status()
# Gets JSON data
response_body = response.json()
# If needed specific exceptions can be caught
# requests.ConnectionError : A network connection error problem occurred
# requests.HTTPError : HTTP request returned an unsuccessful status code. [Response.raise_for_status()]
# requests.ConnectTimeout : The request timed out while trying to connect to the remote server
# requests.ReadTimeout : The server did not send any data in the allotted amount of time.
# requests.TooManyRedirects : The request exceeds the configured number of maximum redirections
# requests.exceptions.RequestException : All exceptions that related to Requests
except requests.exceptions.RequestException as requests_error:
# logs exceptions related to `Requests`
logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}')
raise
except requests.exceptions.JSONDecodeError as json_decode_error:
# logs errors related decoding JSON response
logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}')
raise
except Exception as e:
# logs any other error
logging.error(f'Error, unknown error while retrieving access token. Error message: {e}')
raise
# computes object that holds overrides file content.
# It uses data fetched from request
overrides_data = {
"headers": {
"Authorization": f"Token {response_body['Token']}"
}
}
# log entry informing about the file override computation
overrides_file_path = os.path.join(
working_directory, "api-fuzzing-overrides.json")
logging.info("Creating overrides file: %s" % overrides_file_path)
# attempts to overwrite the file
try:
if os.path.exists(overrides_file_path):
os.unlink(overrides_file_path)
# overwrites the file with our updated dictionary
with open(overrides_file_path, "wb+") as fd:
fd.write(json.dumps(overrides_data).encode('utf-8'))
except Exception as e:
# logs any other error
logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}')
raise
# logs informing override has finished successfully
logging.info("Override file has been updated")
# end
```
In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `FUZZAPI_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command.
As for example, the following script `user-pre-scan-set-up.sh`:
```shell
#!/bin/bash
# user-pre-scan-set-up.sh
# Ensures python dependencies are installed
echo "**** install python dependencies ****"
python3 -m ensurepip
pip3 install --no-cache --upgrade \
pip \
backoff
echo "**** python dependencies installed ****"
# end
```
You have to update your configuration to set the `FUZZAPI_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example:
```yaml
stages:
- fuzz
include:
- template: API-Fuzzing.gitlab-ci.yml
variables:
FUZZAPI_PROFILE: Quick
FUZZAPI_OPENAPI: test-api-specification.json
FUZZAPI_TARGET_URL: http://test-deployment/
FUZZAPI_PRE_SCRIPT: user-pre-scan-set-up.sh
FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
FUZZAPI_OVERRIDES_CMD: renew_token.py
FUZZAPI_OVERRIDES_INTERVAL: 300
```
In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in your overrides command.
### Exclude Paths
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0.
......
doc/user/application_security/dast_api/index.md
View file @ 42234def
......@@ -533,7 +533,7 @@ variables:
DAST_API_PROFILE: Quick
DAST_API_OPENAPI: test-api-specification.json
DAST_API_TARGET_URL: http://test-deployment/
DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
DAST_API_OVERRIDES_FILE: dast-api-overrides.json
```
To validate that authentication is working, run an DAST API test and review the job logs and
......@@ -575,13 +575,12 @@ variables:
DAST_API_PROFILE: Quick
DAST_API_OPENAPI: test-api-specification.json
DAST_API_TARGET_URL: http://test-deployment/
DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
DAST_API_OVERRIDES_FILE: dast-api-overrides.json
DAST_API_OVERRIDES_CMD: renew_token.py
DAST_API_OVERRIDES_INTERVAL: 300
```
To validate that authentication is working, run an DAST API test and review the job logs and
the test API's application logs.
To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. See the [overrides section](#overrides) for more information about override commands.
### Configuration files
......@@ -648,6 +647,9 @@ can be added, removed, and modified by creating a custom configuration.
|[`DAST_API_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. |
|[`DAST_API_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. |
|[`DAST_API_OVERRIDES_CMD`](#overrides) | Overrides command. |
|[`DAST_API_OVERRIDES_CMD_VERBOSE`](#overrides) | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.6. |
|`DAST_API_PRE_SCRIPT` | Run user command or script before scan session starts. |
|`DAST_API_POST_SCRIPT` | Run user command or script after scan session has finished. |
|[`DAST_API_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). |
|[`DAST_API_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. |
|[`DAST_API_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. |
......@@ -829,7 +831,7 @@ variables:
DAST_API_PROFILE: Quick
DAST_API_OPENAPI: test-api-specification.json
DAST_API_TARGET_URL: http://test-deployment/
DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
DAST_API_OVERRIDES_FILE: dast-api-overrides.json
```
#### Using a CI/CD variable
......@@ -873,17 +875,182 @@ variables:
#### Using a command
If the value must be generated or regenerated on expiration, you can provide a program or script for
the DAST API scanner to execute on a specified interval. The provided script runs in an Alpine Linux
container that has Python 3 and Bash installed. If the Python script requires additional packages,
it must detect this and install the packages at runtime. The script creates the overrides JSON file
as defined above.
the DAST API scanner to execute on a specified interval. The provided command runs in an Alpine Linux
container that has Python 3 and Bash installed.
You have to set the environment variable `DAST_API_OVERRIDES_CMD` to the program or script you would like
to execute. The provided command creates the overrides JSON file as defined previously.
You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for
your overrides command. In this case, we recommend setting the `DAST_API_PRE_SCRIPT` to the file path of a script which
provides those prerequisites. The script provided by `DAST_API_PRE_SCRIPT` is executed once, before the analyzer starts.
See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management)
page for information about installing Alpine Linux packages.
You must provide three CI/CD variables, each set for correct operation:
- `DAST_API_OVERRIDES_FILE`: File generated by the provided command.
- `DAST_API_OVERRIDES_CMD`: Command to generate JSON file.
- `DAST_API_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically.
- `DAST_API_OVERRIDES_INTERVAL`: Interval in seconds to run command.
Optionally:
- `DAST_API_PRE_SCRIPT`: Script to install runtimes or dependencies before the scan starts.
```yaml
stages:
- dast
include:
- template: DAST-API.gitlab-ci.yml
variables:
DAST_API_PROFILE: Quick
DAST_API_OPENAPI: test-api-specification.json
DAST_API_TARGET_URL: http://test-deployment/
DAST_API_OVERRIDES_FILE: dast-api-overrides.json
DAST_API_OVERRIDES_CMD: renew_token.py
DAST_API_OVERRIDES_INTERVAL: 300
```
#### Debugging overrides
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8.
By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `DAST_API_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing.
It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and following a naming convention.
Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished.
Following our example, we provided `renew_token.py` in the environment variable `DAST_API_OVERRIDES_CMD`. Please notice two things in the script:
- Log file is saved in the location indicated by the environmental variable `CI_PROJECT_DIR`.
- Log file name should match `gl-*.log`.
```python
#!/usr/bin/env python
# Example of an overrides command
# Override commands can update the overrides json file
# with new values to be used. This is a great way to
# update an authentication token that will expire
# during testing.
import logging
import json
import os
import requests
import backoff
# [1] Store log file in directory indicated by env var CI_PROJECT_DIR
working_directory = os.environ['CI_PROJECT_DIR']
# [2] File name should match the pattern: gl-*.log
log_file_path = os.path.join(working_directory, 'gl-user-overrides.log')
# Set up logger
logging.basicConfig(filename=log_file_path, level=logging.DEBUG)
# Use `backoff` decorator to retry in case of transient errors.
@backoff.on_exception(backoff.expo,
(requests.exceptions.Timeout,
requests.exceptions.ConnectionError),
max_time=30)
def get_auth_response():
return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD']))
# In our example, access token is retrieved from a given endpoint
try:
# Performs a http request, response sample:
# { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" }
response = get_auth_response()
# Check that the request is successful. may raise `requests.exceptions.HTTPError`
response.raise_for_status()
# Gets JSON data
response_body = response.json()
# If needed specific exceptions can be caught
# requests.ConnectionError : A network connection error problem occurred
# requests.HTTPError : HTTP request returned an unsuccessful status code. [Response.raise_for_status()]
# requests.ConnectTimeout : The request timed out while trying to connect to the remote server
# requests.ReadTimeout : The server did not send any data in the allotted amount of time.
# requests.TooManyRedirects : The request exceeds the configured number of maximum redirections
# requests.exceptions.RequestException : All exceptions that related to Requests
except requests.exceptions.RequestException as requests_error:
# logs exceptions related to `Requests`
logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}')
raise
except requests.exceptions.JSONDecodeError as json_decode_error:
# logs errors related decoding JSON response
logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}')
raise
except Exception as e:
# logs any other error
logging.error(f'Error, unknown error while retrieving access token. Error message: {e}')
raise
# computes object that holds overrides file content.
# It uses data fetched from request
overrides_data = {
"headers": {
"Authorization": f"Token {response_body['Token']}"
}
}
# log entry informing about the file override computation
# the location of the overrides json file is also CI_PROJECT_DIR
overrides_file_path = os.path.join(
working_directory, "dast-api-overrides.json")
logging.info("Creating overrides file: %s" % overrides_file_path)
# attempts to overwrite the file
try:
if os.path.exists(overrides_file_path):
os.unlink(overrides_file_path)
# overwrites the file with our updated dictionary
with open(overrides_file_path, "wb+") as fd:
fd.write(json.dumps(overrides_data).encode('utf-8'))
except Exception as e:
# logs any other error
logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}')
raise
# logs informing override has finished successfully
logging.info("Override file has been updated")
# end
```
In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `DAST_API_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command.
As for example, the following script `user-pre-scan-set-up.sh`
```shell
#!/bin/bash
# user-pre-scan-set-up.sh
# Ensures python dependencies are installed
echo "**** install python dependencies ****"
python3 -m ensurepip
pip3 install --no-cache --upgrade \
pip \
backoff
echo "**** python dependencies installed ****"
# end
```
You have to update your configuration to set the `DAST_API_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example:
```yaml
stages:
- dast
......@@ -895,11 +1062,14 @@ variables:
DAST_API_PROFILE: Quick
DAST_API_OPENAPI: test-api-specification.json
DAST_API_TARGET_URL: http://test-deployment/
DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
DAST_API_PRE_SCRIPT: user-pre-scan-set-up.sh
DAST_API_OVERRIDES_FILE: dast-api-overrides.json
DAST_API_OVERRIDES_CMD: renew_token.py
DAST_API_OVERRIDES_INTERVAL: 300
```
In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in our overrides command.
### Exclude Paths
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7