Installing CloudShark via Docker

CloudShark in Docker

Since CloudShark 3.9, users are able to deploy in a Docker environment. Everybody’s container environment is different. This document provides an overview of the environment and required settings intended to be used as an starting point for your container management platform of choice.

The CloudShark Image

CloudShark Docker is distributed only from the QA Cafe Customer Lounge. The download consists of a directory containing:

A copy of the installer for the latest version
A Dockerfile needed to build the image
An example docker-compose.yml file
A .tgz of additional dependencies for Docker
A README.md with detailed information.

Additional Containers

3rd-party containers providing a database and caching services are also specified in the docker-compose config, including:

MariaDB v10
Redis v3.2
Memcache v1.6.9

CloudShark relies on publicly available versions of these services that have been published on Docker Hub.

Required: The QA Cafe License App Server

Running CloudShark in a Docker requires it can connect to and communicate with the QA Cafe License Application running outside of the containerized deployment. This may be on the docker host, or a separate machine.

QALA is available to be downloaded as an RPM for Rocky Linux 8.

Volumes

PCAP Data

It is a very good idea to store PCAP data outside of your Docker. The following path should be mounted from an external docker volume:

/usr/cloudshark/data

Auto-Import

If you wish to configure Auto-Import features, you will also need to make a volume available within the CloudShark container that maps to an autoimport location.

By mounting a host directory at this location, you can move files on your host and have them processed by CloudShark within Docker. You will need to configure this path in the Auto-Import settings in CloudShark.

The Database

Additionally, the MariaDB database container should also have external storage supplied and mounted under:

/var/lib/mysql

Environment Variables

The following environment variables are used to configure CloudShark to talk to external services. They must be specified.

DATABASE_URL=mysql://<user>:<password>@<db-addr>/<db-name>
REDIS_URL=redis://<redis-addr>/
MEMCACHE=<memcache-addr>:11211
QACAFE_LICENSE_SERVER=<external IP>

Caveats

External Authentication

External Authentication using LDAP/AD is not supported when running CloudShark within a container. SAML and OAuth are still available.

Sample docker-compose.yaml

Below is our sample docker-compose.yaml file showing these services working together. While Docker does not recommend using compose in production, it still serves as a concrete, easy to translate example of how all the service interact.

version: "3.8"

volumes:
  data:
  db:
  config:

services:
  app:
    image: cloudshark-app
    build: .
    environment:
      - DATABASE_URL=mysql://root@mariadb/cloudshark
      # To connect to mariadb using a password remove the line above and uncomment the following line
      # - DATABASE_URL=mysql://root:password@mariadb/cloudshark
      - REDIS_URL=redis://redis/
      - MEMCACHE=memcached:11211
      # Uncomment the following line to set the IP/Hostname of your QALA server. See README.md for details.
      # - QACAFE_LICENSE_SERVER=[QALA-ADDRESS]
    depends_on:
      - mariadb
      - memcached
      - redis
    volumes:
      - data:/usr/cloudshark/data
      - config:/usr/cloudshark/etc
    ports:
      - "443:443"
      - "80:80"
    restart: unless-stopped

  memcached:
    image: memcached:1.6.9

  redis:
    image: redis:latest

  mariadb:
    image: mariadb:10
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
      # To set a password for mariadb remove the line above and uncomment the following line
      # - MYSQL_ROOT_PASSWORD=password
      - MYSQL_DATABASE=cloudshark

Notes:

The DATABASE_URL environment variable must match the credentials defined in the mariadb image’s environment. This example uses an empty root password for the database. If you would like to use a password, you can configure one by setting the MYSQL_ROOT_PASSWORD environment variable. More documentation to configure MariaDB can be found here.
If the mariadb image has already been started with the environment variable MYSQL_ALLOW_EMPTY_PASSWORD=yes then setting the MYSQL_ROOT_PASSWORD will not update the password. To update the password connect to the mariadb CLI by running docker compose exec mariadb mariadb -uroot and issue the following commands:
```
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> ALTER USER 'root'  IDENTIFIED BY 'password';
MariaDB [(none)]> quit;
```

Custom Threat Assessment Rules

If you are using custom Suricata rules (such as ET Pro) you will need to update them from within your running container. CloudShark ships with suricata-update installed which should be used to manage your rule sources.

To maintain additional rules, you should also define and mount a docker volume similar to db, config, and data:

/var/lib/suricata

By mounting this as an external volume, you may rebuild your docker image without losing your configured rule sources.

Running suricata-update

Running suricata-update must be done within the context of the Docker container. First, use docker ps to determine which container to run your command in:

# docker ps
CONTAINER ID IMAGE           COMMAND                CREATED        STATUS        PORTS                                    NAMES
900ed2470c82 cloudshark_app  "/usr/bin/supervisor…" 26 minutes ago Up 25 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp cloudshark_app_1
54b8b649244e mariadb:10      "docker-entrypoint.s…" 26 minutes ago Up 25 minutes 3306/tcp                                 cloudshark_mariadb_1
701bb03b3a08 redis:3.2       "docker-entrypoint.s…" 26 minutes ago Up 25 minutes 6379/tcp                                 cloudshark_redis_1
3d2774876583 memcached:1.6.9 "docker-entrypoint.s…" 26 minutes ago Up 25 minutes 11211/tcp                                cloudshark_memcached_1

Then, run /usr/bin/suricata-update in that Container to enable et/pro:

# docker exec -it 900ed2470c82 /usr/bin/suricata-update enable-source et/pro
20/4/2022 -- 14:48:35 - <Info> -- Using data-directory /var/lib/suricata.
20/4/2022 -- 14:48:35 - <Info> -- Using Suricata configuration /etc/suricata/suricata.yaml
20/4/2022 -- 14:48:35 - <Info> -- Using /usr/share/suricata/rules for Suricata provided rules.
20/4/2022 -- 14:48:35 - <Info> -- Found Suricata version 5.0.7 at /usr/sbin/suricata.
The source et/pro requires a subscription. Subscribe here:
   https://www.proofpoint.com/us/threat-insight/et-pro-ruleset
Emerging Threats Pro access code (secret-code): xxxxxxx
20/4/2022 -- 14:48:50 - <Info> -- Source et/pro enabled

Subsequent runs of suricata-update will now pull in ETPro rules and store them in /var/lib/suricata/rules.

# docker exec -it 900ed2470c82 /usr/bin/suricata-update

The Docker Host machine should be in charge of periodically running this command in order to keep your rules up to date.

To restart Suricata within your container, execute the following command:

# docker exec -it 900ed2470c82 pkill -SIGUSR2 -f suricata

Application Log Files

Application log files are stored under /tmp in the Docker container. To list the contents of this directory first use docker ps to determine the Container ID of the Docker container running the cloudshark_app image:

# docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS         PORTS                                     NAMES
1f05be41120b   cloudshark-app    "/usr/bin/supervisor…"   4 seconds ago   Up 3 seconds   0.0.0.0:4433->443/tcp, :::4433->443/tcp   3110-app-1
9ccd42ae95bb   memcached:1.6.9   "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   11211/tcp                                 3110-memcached-1
5712416dbcb8   mariadb:10.8.2    "docker-entrypoint.s…"   4 seconds ago   Up 4 seconds   3306/tcp                                  3110-mariadb-1
99cd7818cd4c   redis:latest      "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   6379/tcp                                  3110-redis-1

Running ls /tmp | grep supervisor in the container will display the available application log files:

# docker exec 23d6c8323ae8 ls /tmp | grep supervisor
cloudshark-autoimport-stderr---supervisor-1ru41jaz.log
cloudshark-autoimport-stdout---supervisor-k2uwzcco.log
cloudshark-search-worker-1-stderr---supervisor-mgb48h_1.log
cloudshark-search-worker-1-stdout---supervisor-adunrcps.log
cloudshark-search-worker-2-stderr---supervisor-ckx5jzs0.log
cloudshark-search-worker-2-stdout---supervisor-en0hppd1.log
cloudshark-traceframe-stderr---supervisor-lh5pz2xo.log
cloudshark-traceframe-stdout---supervisor-im4n5a43.log
cloudshark-web-stderr---supervisor-c1ess23u.log
cloudshark-web-stdout---supervisor-miu3ix8w.log
cron-stderr---supervisor-kmbr8tz6.log
cron-stdout---supervisor-s2bo3yr5.log
nginx-stderr---supervisor-ruigkm2c.log
nginx-stdout---supervisor-ku2r0al2.log
setenv-stderr---supervisor-12qxham2.log
setenv-stdout---supervisor-cewsysxy.log
suricata-stderr---supervisor-unqa8a6k.log
suricata-stdout---supervisor-y3hhdkeh.log

Individual application log files can be copied from the container using docker cp:

docker cp 23d6c8323ae8:/tmp/cloudshark-web-stderr---supervisor-c1ess23u.log ~/

An individual application log file can also be viewed in real-time by running tail -f within the container:

docker exec 23d6c8323ae8 tail -f /tmp/cloudshark-web-stdout---supervisor-miu3ix8w.log

The entire /tmp/ directory can also be copied using docker cp:

docker cp 23d6c8323ae8:/tmp ~/

Product

Search Results