Oops, I Did It Again: Automatically Encrypting Secrets

Got a heads-up from OpenAI that they'd invalidated my token because I'd exposed it publicly! Oops. This isn't the first time I've done it, but this will be the first time I take a computational approach towards avoiding it.

(tl;dr use the code and brief instructions in this gist.)

First: Damage Control

I shared three secret tokens actually, about 24 hours ago when I pushed an update of my .bashrc to a public repo. This repo was there for a long time, but I guess I'd been careful about pushing anything sensitive to it: Specifically, I've been using this code block to separate sensitive environment variables out from others:

if [ -f ~/.bash_private ]; then
    . ~/.bash_private
fi

But I guess I mistakenly moved towards keeping everything in a single "run control" (.bashrc) file.

To their credit, OpenAI alerted me before anyone else, and in fact Netlify and GitHub hadn't detected it after the first 24 hours. Not that it's their job! But kudos in any case to OpenAI.

I removed all the tokens from the other two providers, and made the .bashrc repo private for now.

Next: Encryption

I'd first heard of keeping secrets in public repos by encrypting them in this section of ThePrimeagen's "Developer Productivity" course, but I clearly haven't put that into action. The command he uses for encrypting or decrypting is

$ ansible-vault <encrypt/decrypt> <filename>

after having installed ansible. Easy enough! As a bonus, the algorithm it uses is probably quantum-safe. 🤖

There are several other viable options for two-way encryption of secrets, Mozilla's SOPS being one, FYI.

Alerts For Unencrypted Secret Files

Now, how to abort and alert when committing code which contains unencrypted secrets? This DevOps StackExchange question seems to be a good start. Let's see if I can get a pre-commit hook to do this for my .bashrc file:

#!/bin/sh
#
# Put this file in .git/hooks/pre-commit
# from https://devops.stackexchange.com/questions/16317/how-to-ansible-vault-files-as-they-are-committed-to-git?newreg=a7e3811180a7401ea428511549dd5a09

set -o nounset

FILE_PATTERN=".bashrc|.env"   # 👈 I've also included a pattern for .env files
ENCRYPTED_PATTERN="$ANSIBLE_VAULT"

is_encrypted() {
  local file=$1
  if ! git show :"$file" | grep --quiet "^${ENCRYPTED_PATTERN}"; then
    echo "Located a staged file that should be encrypted:
> ${file}
"
    echo "Please un-stage this file. If you are adding or updating this file, please encrypt it before staging."
    echo "Alternatively, you can git checkout the latest encrypted version of the file before committing.
"
    echo "Remember... Only YOU Can Prevent Secret Leakage."
    exit 1
  fi
}

echo "Running pre-commit checks..."
git diff --cached --name-only | grep "${FILE_PATTERN}" | while IFS= read -r line; do
  is_encrypted "${line}"
done

If I pop that in the repo where .bashrc is and try to commit it without encrypting it first, it stops me and displays:

Running pre-commit checks...
Located a staged file that should be encrypted:
> .bashrc

Please un-stage this file. If you are adding or updating this file, please encrypt it before staging.
Alternatively, you can git checkout the latest encrypted version of the file before committing.

Remember... Only YOU Can Prevent Secret Leakage.

That's pretty awesome! Let's make it a little more foolproof by making this pre-commit run for all commits to local repos:

$ mkdir ~/.githooks
$ git config --global core.hooksPath ~/.githooks
$ mv .git/hooks/pre-commit ~/.githooks
$ chmod +x ~/.githooks/pre-commit

Automatically Encrypt Files On Commit

How to automate the encryption of these secret files, though? There's no such automation in the StackExchange answer, but there is a bash script provided to encrypt or decrypt manually. What's more, there's an improved version of that script in this blog post, the improvement being that it skips any already encrypted files which match the FILE_PATTERN instead of breaking out of the while loop.

Let's incorporate the relevant logic into our globally installed pre-commit hook:

#!/bin/sh
# ~/.githooks/pre-commit

set -o nounset

FILE_PATTERN=".bashrc|.env"
ENCRYPTED_PATTERN="$ANSIBLE_VAULT"

encrypt_if_needed_then_add() {
  local file=$1
  if ! git show :"$file" | grep --quiet "^${ENCRYPTED_PATTERN}"; then
    echo "Located a staged file that should be encrypted:
> ${file}
"
    encrypt $file
    git add $file
  fi
}

encrypt() {
  local file=$1
  echo "$file";
  password_type=--ask-vault-password
  if [ -f "$HOME/.vault_password" ]
  then
      password_type="--vault-password-file $HOME/.vault_password"
  fi
  ansible-vault encrypt $password_type $file
}

echo "Running pre-commit checks..."
git diff --cached --name-only | grep "${FILE_PATTERN}" | while IFS= read -r line; do
  encrypt_if_needed_then_add "${line}"
done

I've also put the vault password in a new file ~/.vault_password so I don't have to enter it for every ansible-vault command. This works beautifully on commit, and won't try to encrypt files which are already encrypted:

$ ls -a
.        ..        .bashrc    .env        .git        .gitignore

$ cat .env  # 👈 let's say this one is already encrypted
$ANSIBLE_VAULT;1.1;AES256
30643665626462663736373366386138313966616234313939626563333833363439396530366633
3763383837393934323861313138646137316362333231380a623530396135326536353062333534
34333133383734623161636135616165653833306463336161353032376263316531666339666639
3863663839323835370a333738303064623230653164616531343438386166353461613962626631
3964

$ cat .bashrc  # 👈 and this one isn't
export OPENAI_KEY="sk-hithisisntreallymykey0Kivelearnedmylesson"

$ git add --all && git commit -m "first"
Running pre-commit checks...
Located a staged file that should be encrypted:
> .bashrc

.bashrc
Encryption successful

Automatically Decrypt Files After Commit

The pre-commit hook will work for making sure the secrets are encrypted, but I actually want to immediately decrypt those secrets once the commit is completed, so I can use them locally, modify them, etc. If I copy pre-commit to a file post-commit and make these changes, this should accomplish it:

#!/bin/sh
# ~/.githooks/post-commit

set -o nounset

FILE_PATTERN=".bashrc|.env"
ENCRYPTED_PATTERN="$ANSIBLE_VAULT"

decrypt_if_needed() {
  local file=$1
  if git show :"$file" | grep --quiet "^${ENCRYPTED_PATTERN}"; then
    echo "Located a committed file that should be decrypted:
> ${file}
"
    decrypt $file
  fi
}

decrypt() {
  local file=$1
  echo "$file";
  password_type=--ask-vault-password
  if [ -f "$HOME/.vault_password" ]
  then
      password_type="--vault-password-file $HOME/.vault_password"
  fi
  ansible-vault decrypt $password_type $file
}

echo "Running post-commit checks..."
files=$(ls -a | grep "${FILE_PATTERN}")
for file in $files; do
  echo $file
  decrypt_if_needed $file
done

Now, when .bashrc is unencrypted, this is what happens on commit:

$ git add --all && git commit -m "first"
Running pre-commit checks...
Located a staged file that should be encrypted:
> .bashrc

.bashrc
Encryption successful

Running post-commit checks...
.bashrc
Located a committed file that should be decrypted:
> .bashrc

.bashrc
Decryption successful
[trunk 0798892] first

$ git status
On branch trunk
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   .bashrc

no changes added to commit (use "git add" and/or "git commit -a")

$ cat .bashrc
export OPENAI_KEY="sk-hithisisntreallymykey0Kivelearnedmylesson"

The secret file is unencrypted and there's an unstaged change to it. This is as expected, since the post-commit hook decrypted it. Not a 100% perfect solution since I'll constantly have unstaged changes showing for my secret files, but I'm willing to live with it.

What About Post-Checkout and Post-Pull?

One flaw I can't live with is that if I check out a different branch or commit of this repo, the secrets-containing files will be encrypted. This will also happen when pulling from a remote repo.

I'm pretty sure the quick solution for this is:

$ cp ~/.githooks/post-commit ~/.githooks/post-checkout
$ cp ~/.githooks/post-commit ~/.githooks/post-merge

It works perfectly for checkout, but I haven't tested it for pull/merge yet.

The End

I think this will suffice for automatically encrypting secrets on commit. The biggest point of failure is FILE_PATTERN, which I'll have to be mindful of and modify as needed.

Zev Averbach

American software engineer in Switzerland. Constant builder, learner, and teacher. 🌼