Use Case

An application which needs secrets to be operated is installed on an EC2 Instance. Secrets stored in a configuration file should be handled in a way, that they could be checked in in a git repository and decrypted on the target machine. There they will be stored in the environment and used by the application.

In this lab, Mozilla SOPS is used to encrypt the secrets in a file. SOPS itself acts as an editor for such encrypted files and is able to deal with YAML, JSON, ENV, INI and BINARY (in this example .env) file formats. Furthermore, SOPS is able to deal with various Keys (AWS KMS, GCP KMS, Azure Key Vault, age, and PGP), in this case we'll use AWS KMS.

Strengths

There is no sophisticated handling of keys (e.g. GPG Keys) needed to deal with the secrets, the whole key handling happens in AWS. If users have the permission to access the KMS Key, they are able to read and write the secrets. We don't have to ship the secret to the EC2 Instance, as an assigned IAM Role allows the Instance to use the secret. As we'll see later, the usage of SOPS is pretty straightforward on the server-side. As SOPS supports lots of cloud providers and also GPG, it can be considered as cloud-agnostic (surely, the mechanisms to create keys and roles change).

Weakness

Nevertheless, the secret exist in clear-text on the server (as environment variables or files).

What we'll do in this lab

Big Picture

References:

As it will be easier to deal with SOPS from your machine, please install the AWS CLI on your machine. The Installation Instructions can be found here.

Create a security group for SSH

aws ec2 create-security-group --group-name sops-lab-ssh-in --description "SSH Traffic for SOPS Lab Instances"
aws ec2 authorize-security-group-ingress \
    --group-name sops-lab-ssh-in \
    --protocol tcp \
    --cidr 0.0.0.0/0 \
    --port 22

Create an Instance

aws ec2 run-instances --image-id ami-087c17d1fe0178315 --instance-type t2.micro --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=sops-instance}]" --key-name vockey --security-groups "sops-lab-ssh-in" --iam-instance-profile Name=LabInstanceProfile --output table

The EC2 instance has been created with an SSH Key which is available in the AWS Academy environment. You can fetch the keys the same way you got your AWS CLI Configuration (AWS Academy -> AWS Details -> SSH Key).

For the further progress of this lab, it's the easiest way to store the PEM Key in ~/.ssh/labsuser.pem

Get the Name of this Instance / Connect to the Instance

INSTANCE_HOSTNAME=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=sops-instance" --query "Reservations[].Instances[].PublicDnsName" --out text | xargs)

echo "ssh -i ~/.ssh/labsuser.pem ec2-user@${INSTANCE_HOSTNAME}"

ssh -i ~/.ssh/labsuser.pem ec2-user@${INSTANCE_HOSTNAME}

You will need Mozilla SOPS on both (the developer machine and the application server).

Therefore, download the application from Releases.

On Mac (Homebrew)

Install SOPS using brew install sops

On Linux

curl -sfL https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux -o /tmp/sops
chmod a+x /tmp/sops
sudo cp /tmp/sops /usr/local/bin/

Key Management Service

Get the ARN of your Key from the Console

 
  ARN="The ARN you noted down"
  sops --kms $ARN secret.env

APP_SECRET="my-super-secret-string"

scp -i ~/.ssh/labsuser.pem secret.env ec2-user@${INSTANCE_HOSTNAME}:/tmp/secret.env

ssh -i ~/.ssh/labsuser.pem ec2-user@${INSTANCE_HOSTNAME}

Install the sample application

curl -sfL https://github.com/fhb-codelabs/secret-demo-app/releases/download/0.0.1/secret-demo-app-linux-amd64 -o /tmp/secret-demo-app
chmod a+x /tmp/secret-demo-app

if [[ ! -d /opt/demo-app ]]; then 
  sudo mkdir /opt/demo-app
fi

sudo mv /tmp/secret-demo-app /opt/demo-app

Set the environment and start the application

/opt/demo-app/secret-demo-app

2021/10/08 16:56:49 No port is set, defaulting to 8080
2021/10/08 16:56:49 Secret to start this up is missing, the APP_SECRET environment variable might be unset

export $(sops --decrypt /tmp/secret.env | xargs)

/opt/demo-app/secret-demo-app

Verify

curl http://localhost:8080

> curl localhost:8080
Hi there, your secret is my-super-secret-string

If we create a virtual machine without the instance profile, we shouldn't be able to decrypt the secret.

aws ec2 run-instances --image-id ami-087c17d1fe0178315 --instance-type t2.micro --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=sops-instance-2}]" --key-name vockey --security-groups "sops-lab-ssh-in" --output table

INSTANCE_HOSTNAME_2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=sops-instance-2" --query "Reservations[].Instances[].PublicDnsName" --out text | xargs)

echo "ssh -i ~/.ssh/labsuser.pem ec2-user@${INSTANCE_HOSTNAME_2}"

ssh -i ~/.ssh/labsuser.pem ec2-user@${INSTANCE_HOSTNAME_2}

curl -sfL https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux -o /tmp/sops
chmod a+x /tmp/sops
sudo cp /tmp/sops /usr/local/bin/ 

scp -i ~/.ssh/labsuser.pem secret.env ec2-user@${INSTANCE_HOSTNAME_2}:/tmp/secret.env

sops --decrypt /tmp/secret.env

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  arn:aws:kms:us-east-1:<REDACTED>: FAILED
    - | Error decrypting key: NoCredentialProviders: no valid
      | providers in chain. Deprecated.
      | 	For verbose messaging see
      | aws.Config.CredentialsChainVerboseErrors

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.