Articles in this section

See more

Data Retention Installation Guide

Avatar
Elena H
  • Updated
Follow

Data Retention Installation Guide: Starting SaaS Data Retention Docker Image

This document will describe the process of starting the SAAS Data Retention docker image for the SAAS Data Retention Service.

Overview

The SAAS Data Retention bot will capture all messages sent within a specific Wickr SAAS network. You have several options to configure how these messages will be stored and where they will be stored. You have the option to store the captured messages and files using S3 or the default option is to save the captured messages and files locally. The possible storage options are the following:

  • Store all captured messages and files locally. It is your responsibility to move these files to another system for long term storage and to make sure the host disk drive does not run out of space.
  • Store all captured messages and files on an S3 bucket. The Data Retention bot will save all decrypted messages and files to the S3 bucket you define. Once stored successfully they will be removed from the host machine.
  • Store all captured messages and files encrypted on an S3 bucket. The Data Retention bot will re-encrypt all captured messages and files using a key that you supply and save them to the S3 bucket you define. Once re-encrypted and stored successfully they will be removed from the host machine. NOTE: You will need software to decrypt the message files and files.

Messages are saved in files which are limited to a specific configurable size or time limit. These files are saved locally or on the defined AWS S3 bucket.

You will need to use a host with sufficient memory and disk space to operate the SAAS Data Retention bot.

The SAAS Data Retention bot has options to use other AWS services such as AWS Secrets Manager, AWS S3, AWS KMS and AWS SNS. Details on how to use these services is described later. If you plan on using these optional AWS services you will need to host the SAAS Data Retention docker image on an appropriate host that has sufficient privileges to access these AWS services.

The SAAS Data Retention bot can be configured completely using environment variables or using the AWS Secrets Manager. The Data Retention bot’s challenge/password can be set using an environment value, the AWS secret or entered on the command line. Sections below will define the list of environment and AWS secrets variables, and examples of how to define them.

Challenge / Password

The first time you start the Data Retention bot you will need to supply the challenge (initial password) that was created on the Wickr admin console. As mentioned above you can use one of the following methods to do this:

  • the WICKRIO_BOT_PASSWORD environment variable
  • the password value in the AWS secret identified by the AWS_SECRET_NAME environment variable
  • enter the password when prompted by the Data Retention bot. You will need to run the Data Retention bot with interactive TTY access (i.e with the -ti options). Then depending on how you start the bot you may not see the “password;” prompt, if so type the ENTER key and it will appear. When you type in the password it will hide the value with a “*” value for each character. You will be prompted two times to enter the password.

Note: When you are setting up the Data Retention bot the first time a new password will be generated. This generated password will become the password for the Data Retention bot. The new password will be displayed like the following example:

********************************************************************
**** GENERATED PASSWORD
**** DO NOT LOSE THIS PASSWORD, YOU WILL NEED TO ENTER IT EVERY TIME
**** TO START THE BOT
 "HuU7ED55wRAW4lGgC7fU2odn"
 ********************************************************************

WARNING: You must save the password in a safe place. If you lose the password you will not be able to re-install the SAAS Data Retention Bot. Do not share this password with anyone, if they get this password they will have the ability to start the Data Retention for your network.

The challenge/password is only required when you initially setup the Data Retention bot instance. A unique key value will be generated internally which is unique to the instance of the Data Retention bot. If you need to re-install the Data Retention bot for some reason you will use the generated password value, the original challenge value is NOT valid after the initial installation of the Data Retention bot.

How to configure the Data Retention bot

You will use environment variables and/or AWS secrets to configure the SAAS Data Retention bot. The following sections describe these variables.

Environment Variables

Environment variables are set using the “-e” option when you start the docker container (see the example sections). There are environment variables to define the Data Retention service’s bot user credentials, the name of the AWS secret to use, identify AWS S3 bucket information, and received message handling configuration information.

The following are environment variables that identify Data Retention bot credentials:

  • WICKRIO_BOT_NAME : the name of the Data Retention bot account [required]
  • WICKRIO_BOT_PASSWORD : the password/challenge value associated with the bot account. The bot password can also be stored in the AWS secret, which is more secure. [required unless using AWS Secrets, or entering via the terminal]

The following are environment variables that are used to configure the default Data Retention streaming capability. You can identify where to capture the messages and the files. You can configure the base name of the messages file as well as the size limit or time limit for the received messages files. These values can only be configured using environment variables.

  • WICKRIO_COMP_MESGDEST : this is the path name to the directory where messages will be streamed. (default: /tmp/<botname>/compliance/messages) [optional]
  • WICKRIO_COMP_FILEDEST : this is the path name to the directory where files will be streamed. (default: /tmp/<botname>/compliance/attachments) [optional]
  • WICKRIO_COMP_BASENAME : this is the base name for the received messages files. (default: receivedMessages) [optional]
  • WICKRIO_COMP_FILESIZE : this is the maximum file size for a received messages file. When the max size is reached a new file will be started. (default: 1000000000) [optional]
  • WICKRIO_COMP_TIMEROTATE : this number limits the number of minutes a the Data Retention service will put received messages into a specific received messages file. You can only use the file size or this value to limit the size of the received messages file (default: 0) [optional]

AWS Service Environment Variables

The following are environment variables that identify default values:

  • AWS_DEFAULT_REGION : the default AWS region to use for AWS services like Secrets Manager (not used for S3 or KMS). If this is environment variable is not defined then the us-east-1 region will be the default.

The following environment variable identifies the AWS secret to use when using AWS secrets manager to store/retrieve the credentials needed by the Data Retention service:

  • AWS_SECRET_NAME : the AWS secret that contains the credentials needed by the Data Retention bot. [optional]
  • AWS_SECRET_REGION : the AWS region that the AWS secret is located in. If you are using AWS secrets and this value is not defined the AWS_DEFAULT_REGION value will be used. [optional]

The following are environment variables that are used to identify the AWS S3 bucket information if you want the Data Retention service to stream messages and files to AWS S3. You can also configure these values in the AWS secrets.

  • WICKRIO_S3_BUCKET_NAME : the AWS S3 bucket name, if you want to stream capture messages/files to AWS S3. This value can also be set using the AWS Secrets Manager. [optional]
  • WICKRIO_S3_REGION : the region where the AWS S3 bucket is located. This value can also be set using the AWS Secrets Manager. [optional]
  • WICKRIO_S3_FOLDER_NAME : this is a name you can optionally set if you want to use the same AWS bucket for multiple Data Retention bots. This folder name will be prepended to the key for any messages and files written to the S3 bucket [optional]

If you intend to use AWS S3 buckets for the captured messages/files with client side encryption you will need to define the ARN value associated with the KMS master key. The following environment variables are needed to initiate client side encryption:

  • WICKRIO_KMS_MSTRKEY_ARN : the ARN value that identifies the AWS KMS master key used to encrypt the message files and files on the Data Retention bot before they are pushed to the AWS S3 bucket.
  • WICKRIO_KMS_REGION : the region where the KMS master key is located

If you intend to use the AWS SNS capability, to receive events from the Data Retention bot, the following environment variables will define how to publish messages to the appropriate AWS SNS topic. If AWS SNS is setup to be used, the Data Retention bot will send events to the defined SNS topic. Types of events sent include startup, shutdown, as well as error conditions. Supported events are described later in this document.

  • WICKRIO_SNS_TOPIC_ARN: the ARN value associated with the AWS SNS topic that you want Data Retention service events sent to.

The SAAS Data Retention bot has the capability to generate AWS CloudWatch metrics. If activated, the metrics will be generated every 60 seconds. If you want CloudWatch metrics to be generated use the following environment variable to activate this capability:

  • WICKRIO_METRICS_TYPE: Set the value of this environment variable to "cloudwatch" to generate metrics to the associated CloudWatch.

The list of metrics supported will be described in a section later in this document.

AWS Secret Values

The secret in the AWS Secrets Manager can have the following values in it:

  • password : this is the value of the Data Retention bot’s password or challenge value.
  • s3_bucket_name : this is the name of the S3 Bucket to store captured messages/files. If not set, the default file streaming will be used [optional]
  • s3_region : this is the region where the S3 Bucket is located [optional]
  • s3_folder_name : this is a name you can optionally set if you want to use the same AWS bucket for multiple Data Retention bots. This folder name will be prepended to the key for any messages and files written to the S3 bucket [optional]
  • kms_master_key_arn : the ARN value that identifies the AWS KMS master key used to encrypt the message files and files on the Data Retention bot before they are pushed to the AWS S3 bucket.
  • kms_region : the region where the KMS master key is located
  • sns_topic_arn : the ARN value associated with the AWS SNS topic that you want Data Retention service events sent to.

AWS Services

If you plan on using any of the AWS services supported by the Wickr SAAS Data Retention service you will need to make sure the host (EC2 instance for example) has the appropriate IAM Role and Policy to access them. You can configure the Data Retention service to use AWS Secrets Manager, AWS S3, AWS SNS as well as AWS KMS services. The following IAM Policy will allow access to the specific actions of these AWS services:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "secretsmanager:GetSecretValue",
                "sns:Publish",
                "cloudwatch:PutMetricData",
                "kms:GenerateDataKey"
            ],
            "Resource": "*"
        }
    ]
}

You can create an IAM Policy that is stricter by identifying the specific objects for each service that you want to allow the containers on your host to access. Remove the actions for the AWS services that you do not intend to use. For example, if you intent to use an AWS S3 bucket only then remove the following actions:

  • secretsmanager:GetSecretValue
  • sns:Publish
  • kms:GenerateDataKey

If you are using an EC2 instance to host your Data Retention bot create an IAM role, using the EC2 common case and assign a policy using the policy definition from above.

How to start the Data Retention bot

Before you start the Data Retention service’s bot you need to determine how you want to configure it. If you are not running in an environment which will not have access to AWS services then your options are limited. In that case you will use the default message streaming options. You should decide whether you want to limit the size of the captured message files to size or time (see the environment variables in previous section).

If you are running the Data Retention service’s bot in an AWS environment and you want to use AWS services then you should create a secret in the AWS secrets manager to contain the credentials, and create the AWS S3 bucket to keep the received messages and files. Once these AWS services are setup you can proceed to starting the Data Retention service’s docker image.

The following sections show some possible ways to configure and start your Data Retention bot.

Basic Startup (no AWS services)

The following is the basic way to start the SAAS Data Retention bot, password is defined using environment variable. It will start using the default file streaming using the default values defined in the environment variables above.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 -e WICKRIO_BOT_PASSWORD='password' \
wickr/bot-compliance-cloud:latest

Basic Startup, Password Prompted (no AWS services)

The following is the basic way to start the SAAS Data Retention bot, password is entered when prompted by the Data Retention bot. It will start using the default file streaming using the default values defined in the environment variables above.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 wickr/bot-compliance-cloud:latest
 
docker attach compliance_1234567890_bot
.
.
.
Enter the password:************
Re-enter the password:************

Note: it is very important to run the bot using the -ti options so that you receive the password prompt. Also, it is important that you run the “docker attach <container ID | container name>“ immediately after starting the docker image so that you get the password prompt. We suggest running both these commands in a script. If you attach to the docker image and don’t see the prompt hit the ENTER key and you will see the prompt.

Rotate Received Message Files Every 15 Minutes (no AWS services)

The following docker command will start the SAAS Data Retention bot using environment variables and configure it to rotate the received messages files to limit to 15 minutes.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 -e WICKRIO_BOT_PASSWORD='password' \
-e WICKRIO_COMP_TIMEROTATE=15 \
wickr/bot-compliance-cloud:latest

Using AWS Secrets Manager with Password

You can use the AWS Secrets Manager to identify the Data Retention bot’s password. When you start the Data Retention bot you will need to set an environment variable that identifies the AWS Secret where this information is located.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 -e AWS_SECRET_NAME='wickrpro/alpha/new-3-bot' \
wickr/bot-compliance-cloud:latest

The ‘wickrpro/compliance/compliance_1234567890_bot’ secret has the following secret value in it, shown as plaintext:

{
    "password":"password"
}

Using AWS Secrets Manager with Password and S3

You can use the AWS Secrets Manager to host the credentials and the AWS S3 bucket information. When you start the Data Retention bot you will need to set an environment variable that identifies the AWS Secret where this information is located.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 -e AWS_SECRET_NAME='wickrpro/alpha/compliance_1234567890_bot' \
wickr/bot-compliance-cloud:latest

The ‘wickrpro/compliance/compliance_1234567890_bot’ secret has the following secret value in it, shown as plaintext:

{
    "password":"password",
    "s3_bucket_name":"bot-compliance",
    "s3_region":"us-east-1",
    "s3_folder_name":"network1234567890"
}

Messages and files received by the bot will be put in the “bot-compliance'” bucket in the folder named “network1234567890”

Using AWS Secrets Manager with Password, S3 and KMS Encryption

You can use the AWS Secrets Manager to host the credentials, the AWS S3 bucket and AWS KMS master key information. When you start the Data Retention bot you will need to set an environment variable that identifies the AWS Secret where this information is located.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 -e AWS_SECRET_NAME='wickrpro/alpha/compliance_1234567890_bot' \
wickr/bot-compliance-cloud:latest

The ‘wickrpro/compliance/compliance_1234567890_bot’ secret has the following secret value in it, shown as plaintext:

{
    "password":"password",
    "s3_bucket_name":"bot-compliance",
    "s3_region":"us-east-1",
    "s3_folder_name":"network1234567890",
    "kms_master_key_arn":"arn:aws:kms:us-east-1:1234567890:key/12345678-1234-abcde-a617-abababababab",
    "kms_region":"us-east-1"
}

Messages and files received by the bot will be encrypted using the KMS key identified by the ARN value, then put in the “bot-compliance'” bucket in the folder named “network1234567890”.

NOTE: Make sure you have the appropriate IAM policy setup.

Using environment variables

If you do not want to use the AWS Secrets Manager to host the credentials associated with the Data Retention bot you can start the docker image with environment variables.

docker run -v /opt/compliance_1234567890_bot:/tmp/compliance_1234567890_bot \
-d --restart on-failure:5 --name="compliance_1234567890_bot" -ti \
 -e WICKRIO_BOT_NAME='compliance_1234567890_bot' \
 -e WICKRIO_BOT_PASSWORD='password' \
 -e WICKRIO_S3_BUCKET_NAME='bot-compliance' \
-e WICKRIO_S3_FOLDER_NAME='network1234567890' \
 -e WICKRIO_S3_REGION='us-east-1' \
wickr/bot-compliance-cloud:latest

You can use environment values to identify the Data Retention bot’s credentials, information about AWS S3 buckets, and/or configuration information for the default file streaming. You MUST identify the name of the Data Retention bot (i.e. WICKRIO_BOT_NAME) as an environment variable.

Stopping a Data Retention Bot

The software running on the Data Retention bot will capture SIGTERM signals and gracefully shutdown. You can use the ‘docker stop <CONTAINER ID|NAME>’ command to issue the SIGTERM command to the Data Retention bot’s docker image. For example:

docker stop compliance_1234567890_bot

Getting Logs

By default the software running on the Data Retention bot’s docker image will be output to log files in the /tmp/<botname>/logs directory. They will rotate to a maximum of 5 files. You can also get the logs by running the following command:

docker logs compliance_1234567890_bot

Metrics and Events for the Data Retention bot

This section describes the metrics and events that are currently supported by the 5.116 version of the SAAS Data Retention bot.

Metrics (AWS CloudWatch)

The SAAS Data Retention bot metrics are generated by the bot on one minute intervals and transmitted to the CloudWatch service associated with the account the SAAS Data Retention docker image is running on.

These are the existing metrics supported by the Saas Data Retention bot.

Metric Description
Messages_Rx messages received
Messages_Rx_Failed failures to process received messages
Messages_Saved messages saved to the received messages file
Messages_Saved_Failed failures to save messages to the received messages file
Files_Saved files received
Files_Saved_Bytes number of bytes for files received
Files_Saved_Failed failures to save files
Logins logins (normally this will be 1 for each interval)
Login_Failures failures to login (normally this will be 1 for each interval)
S3_Post_Errors errors posting message files and files to S3 bucket
Watchdog_Failures watchdog failures
Watchdog_Warnings watchdog warnings

Metrics are generated to be consumed by AWS CloudWatch. The namespace that will be used for bots is “WickrIO”. Each metric has an array of dimensions, the following is the list of dimensions that will be posted with the above metrics:

Dimension Value
Id the bot's username
Device description of specific bot device/instance. Useful if multiple bot instances
Product one of WickrPro_, WickrEnterprise_ with the type appended: Alpha, Beta, Production
BotType bot type, "Compliance" for the complliance bots
Network the ID of the associated network

Events (AWS SNS)

This section lists and describes the events that are generated by the Saas Data Retention bot client. These events will be posted to the AWS SNS topic defined by the ARN value identified by the WICKRIO_SNS_TOPIC_ARN environment variable or the sns_topic_arn secret value.

Events generated by the SAAS Data Retention bot will be sent as JSON strings. As of the 5.116 release the following values will be include in the events:

Name Value
complianceBot the username of the Data Retention bot
dataTime the date and time when the event occurred
device description of the specific bot device/instance. Useful if multiple bot instances
dockerImage the Docker image associated with the bot
dockerTag the tag/version of the Docker image
message the event message, see possible values below
notificationType for events the value will be "Bot Event"
severity either "normal" or "critical"

You will need to subscribe to the SNS topic so that you can receive the events. If you subscribe using an email address you will receive an email similar to the following when an event is generated:

{
"complianceBot": "compliance_85884765_bot",
 "dateTime": "2022-10-12T13:05:39",
 "device": "Desktop 7be8fde46ed5",
 "dockerImage": "wickr/bot-compliance-cloud",
 "dockerTag": "6.18.19.01",
 "message": "Logged in",
 "notificationType": "Bot Event",
 "severity": "normal"
}

The following sections list the events that are supported in the 5.116 version.

Critical Events

These are events that will cause the bot to stop or restart. The number of restarts will be limited so as to not cause other issues.

Login Failures

These are events generated when the bot client fails to login. The following are the possible event messages that can be generated. Each message will indicate the reason for the login failure.

Event Type Event Message
failedlogin bad credentials check password
failedlogin user not found
failedlogin account or device is suspended
provisioning User exited the command
provisioning bad password for the config.wickr file
provisioning cannot read the config.wickr file
failedlogin logins all failed!
failedlogin new user but database already exists!

More Critical Events

Event Type Event Message
Suspended Account WickrIOClientMain::slotAdminUserSuspend: code(%1): reason: %2“
BotDevice Suspended Device is suspended!
WatchDog The SwitchBoard system is down for more than <x> minutes
S3 Failures Failed to put file <file-name> on S3 bucket. Error: <AWS-error>
Fallback Key SERVER SUBMIITED FALLBACK KEY: Is not a recognized client active fallback key. Please submit logs to desktop engineering.

Normal Events

These are normal events that warn the user about normal operating occurrences. Too many occurrences of these types of events within a specific time period may be cause for concern.

Device added to account

This event is generated when a new device is added to this Data Retention bot account. In some situations this can be an important indication to the network owner that someone has created an instance of the Data Retention bot. The following is the message for this event:

A device has been added to this account!

Bot Logged in

This event is generated when the bot has successfully logged in. The following is the message for this event:

Logged in

Shutting down

This event is generated when the bot is shutting down. If the user did not explicitly initiate this, it could be an indication of a problem. The following is the message for this event:

Shutting down

Updates Available

This event is generated when the SAAS Data Retention bot is started and it identifies that there is a newer version of the associated Docker image available. This event will be generated when the bot starts and on a daily basis. This event includes the “versions” array field which identifies the new versions that are available. For example, the following is a sample of what this event would look like:

{
  "complianceBot": "compliance_85884765_bot",
  "dateTime": "2022-10-12T13:05:55",
  "device": "Desktop 7be8fde46ed5",
  "dockerImage": "wickr/bot-compliance-cloud",
  "dockerTag": "6.16.20.01",
  "message": "There are updates available",
  "notificationType": "Bot Event",
  "severity": "normal",
  "versions": [
    "6.18.19.01"
  ]
}

Related articles

Comments

0 comments

Article is closed for comments.