Securely store secrets in Git using SOPS and Azure Key Vault

Kubernetes is usually called a declarative system because the desired state is defined in YAML files. To apply these YAML files to Kubernetes, GitOps is often used as a continuous delivery process. When implementing GitOps, Git becomes the source of truth for deployments. It compares the desired state in Git with the current state and applies changes to Kubernetes. Kubernetes objects describe the state of the cluster. Examples of objects are pods, services, namespaces, config maps, and secrets. All of these are defined in YAML files and committed to Git. It should not come as a surprise that storing secrets in Git comes with challenges. Here is where SOPS comes into play. SOPS (Secrets OPerationS) is a popular tool to encrypt secrets. SOPS works with JSON, YAML, and INI files. Instead of encrypting the entire file, it only encrypts the values and keeps the keys in plain text. This means the files stay readable, which is great for maintainability.

Let's take a simple YAML file that includes credentials.

      firstname: John
      lastname: Doe
      emailAddress: [email protected]
      credentials:
        username: john.doe
        password: this-is-a-password
    

SOPS comes with a CLI. It includes commands to encrypt and decrypt secrets: sops -e --encrypted-regex '^(password)$' example.yaml > example.encrypted.yaml After running this command, the file below is created. As you can see, only the password is encrypted. The rest of the file remains readable.

      firstname: John
      lastname: Doe
      emailAddress: [email protected]
      credentials:
          username: john.doe
          password: ENC[AES256_GCM,data:JLZDP1NHn730fyzRM8ZWtVB/,iv:4JXAmYUfZ9ANoMgyFcO7H6tKsY1wUq4nljVLHMp6FE4=,tag:vIKbzuzT7M56tu89nkg/jg==,type:str]
      sops:
          kms: []
          gcp_kms: []
          azure_kv:
              - vault_url: https://[NAME].vault.azure.net
                name: [NAME_OF_KEY]
                version: 2463ae16286b4cc8a04f0a2d4bc951d8
                created_at: "2022-12-23T10:48:16Z"
                enc: fJkjVV3EwQNs_2XxNUEBiCtjA6mcPgBtKY1TdwLCjUHRANkGxLxgZBByTzNBo1usyfUtqERKrZIMeNbB6p8CFmS0xJ9Mr0kc0E01CHfQBC3egJJs3O0sZvwMiw-N3-NI-ViUmhP7QRpo7BtPjwHkYEvO8M4bYUDsiB5JTSS1QQBCmGx_uiIUj5Mx6Fqn7Lg1uQGpV-g63PZ6bqU7ZuQ1AQ4cSz7NYVXK6cMbuGL_DUdqyQHQ86LaEiFwyy_Qg-A5vuhwDALYnz65iVx5qgvRlkeAOHrBRqqFtLxCMJEXHUCQ58hB-5BEuiuRNoGTIpln7r4C-rGsY8Z2TQWZePCKaA
          hc_vault: []
          age: []
          lastmodified: "2022-12-23T10:48:17Z"
          mac: ENC[AES256_GCM,data:PXwT5dv3fWKCr9++UBvv3Lr8lMo6suHJ5fDcwbsx2ZgbXalm7LRHkUMjKb2EPsJEnEGBSzNkeuGfwvCVzL7O5UywW/XpGCfxB0JqglghRPGlJOZOdDGswOBe6lkVHzf2lCP6yjV0ay4Z4IKfog9OmyNitKceJt8T3Xmj04b1Mns=,iv:TiQNVJHP8RbOkzwLDweMy8c/n2KD7jLfBjEo/X3jvYM=,tag:ffhEcPePEfMUEGXTELm0uw==,type:str]
          pgp: []
          encrypted_regex: ^(password)$
          version: 3.7.3
    

How does SOPS work?

SOPS uses AES256 (Advanced Encryption Standard) to encrypt and decrypt data. It uses symmetric encryption, meaning one encryption key is used to encrypt and decrypt. In the encrypted code snippet, you saw that SOPS understands YAML, and instead of encrypting the entire file, it only encrypts the values. SOPS follows the envelope encryption practice. Envelope encryption uses a Data Encryption Key (DEK) to encrypt data then a Key Encryption Key (KEK) is used to encrypt the Data Encryption Key. Each time the sops -e command is executed, it generates a 256-bit data key.

To encrypt the data key, SOPS integrates with Azure Key Vault, AWS KMS, GCP KMS, and HashiCorp Vault. After the data key is encrypted, it's stored in the enc field along with other metadata about the KMS. In the example, I'm using Azure Key Vault. Before you execute sops -e you must be logged in (az login) with a user that has access to Key Vault and has permissions to the action:

• Microsoft.Key Vault/vaults/keys/encrypt/action

For decrypting:

• Microsoft.Key Vault/vaults/keys/decrypt/action

You can tell SOPS to use Key Vault by passing the argument --azure-kv https://[NAME].vault.azure.net/keys/[NAME_OF_KEY]/[VERSION]. Or by creating the .sops.yaml file in the same directory or one of the ancestor directories. This is more convenient when working with multiple files.

      creation_rules:
        - azure_Key Vault: https://[NAME].vault.azure.net/keys/[NAME_OF_KEY]/[VERSION]
    

When executing sops -d, SOPS will ask Azure Key Vault to decrypt the data key. SOPS will then use AES265 and the data key to decrypt the data.

The encrypted file also contains the mac field. Message Authentication Code (MAC) ensures the authenticity of the encrypted file when it arrives at the receiver. What happens is that SOPS takes the file structure, encrypts it, and stores it in the mac field. When a bad actor changes the file, and you execute sops -d, it will throw an exception: MAC mismatch.

Rotate the Data Encryption Key with SOPS

An extra security measure is to implement key rotation. SOPS allows us to rotate the Data Encryption Key. The command sops -r example.encrypted.yaml > example.encrypted.new.yaml first decrypts the file, generates a new Data Encryption Key, and encrypts the file with the new key. Lastly, Key Vault encrypts the Data Encryption Key.

Using SOPS with Kubernetes and GitOps

When implementing a CI/CD pipeline, there are two possible mechanisms pull and push. Push has been around for the longest time. The CI/CD pipeline initiates the deployment and connects to the Kubernetes cluster to push new data. Usually, with this approach, secrets are stored in the CI/CD tool. With a pull mechanism, a cluster operator regularly scans the Git repository to identify new changes. If the operator detects a change, it will update the state of the cluster. A benefit of this approach is that no external client has admin access to the cluster.

When using GitOps, Git is the source of truth. All manifest, including secrets, are stored in Git. Keeping secrets in Git doesn't sound like a best practice. Except if those secrets are encrypted. The most popular tools used in this scenario are SOPS and Seal secrets. Let's make a quick comparison:

Use Azure Key Vault for key management

A KMS (Key Management System) is a centralized storage for managing cryptographic keys. Cryptographic keys are used to secure data in transit and rest. A company must secure these keys at all costs because losing them could mean that a bad actor can decrypt the data. Azure Key Vault is a KMS that securely stores keys in an HSM at rest. When creating a key, you have two options software-protected and HSM-protected keys. For both options, the key is stored in an HSM. The difference is where the cryptographic operations (encrypt, decrypt, etc.) are executed. For software-protected keys, these are executed in software on a VM, and for HSM-protected keys, in an HSM.

An HSM (hardware security module) is a physical device that secures keys and can execute cryptographic operations like encrypting and decrypting data, verifying and signing signatures, and more. Azure is using nCipher nShield HSMs.

Encrypt data using Azure Key Vault

In the previous section, I explained that SOPS asks Key Vault to encrypt (and decrypt) the data key. Key Vault uses RSA keys to asymmetric encrypt and decrypt data. SOPS calls the Key Vault's encrypt endpoint to encrypt the data key. However, this is not mandatory. To improve performance, a client can also encrypt the data locally using the public key from Key Vault. With asymmetric encryption, data can only be decrypted using the private key. Because the private key doesn't leave Azure, using the Key Vault's decrypt endpoint is mandatory.

As I explained, under the hood, SOPS calls Azure Key Vault to encrypt and decrypt the data key. In the next section, I'll explain how you can leverage Azure Key Vault to store data at rest securely. Azure uses this approach internally as well to store data at rest.

1. The frontend application sends data to the backend application.
2. The backend application gets the public key from Azure Key Vault.
   1. Even though Azure offers an endpoint for encryption, it's only necessary when using symmetric keys. With asymmetric keys, encryption can be done by the client using the public key from Key Vault.
3. The backend application uses the public key to encrypt the data.
4. Store the encrypted data securely along with the id and version of the key.
   1. It's important to save the version of the key if key rotation is enabled. The same version should be used to decrypt the data.

Below an example how to encrypt data in C#.

    string Encrypt(string text)
    {
        var publicKey = keyClient.GetKey(keyName);

        using (var rsa = publicKey.Value.Key.ToRSA())
        {      
            var byteData = Encoding.Unicode.GetBytes(text);

            var encryptedText = rsa.Encrypt(byteData, RSAEncryptionPadding.OaepSHA256);

            var result = Convert.ToBase64String(encryptedText);

            return result;
        } 
    }
    

Decrypt data with Azure Key Vault

1. The frontend application requests data from the backend application.
2. The backend applications calls the storage to get encrypted data.
3. The backend application calls Azure Key Vault to decrypt the data.
   1. Decrypting data requires the private key. This key is stored safely in Azure and never exposed. Key Vault offers an endpoint to decrypt data.
   2. Pass the public key id and the version (use the same version of the key that was used to encrypt the data, if a different version is passed, Key Vault cannot decrypt the data.)
4. The backend application returns the decrypted data to the frontend application.

Below an example how to decrypt data in C#.

      void Decrypt(string encryptedText)
      {
          var publicKey = keyClient.GetKey(keyName);
      
          var cryptoClient = new CryptographyClient(publicKey.Value.Id, credentials);
      
          var encryptedBytes = Convert.FromBase64String(encryptedText);
          var dencryptResult = cryptoClient.Decrypt(EncryptionAlgorithm.RsaOaep256, encryptedBytes);
          var decryptedText = Encoding.Unicode.GetString(dencryptResult.Plaintext);
          
          Console.WriteLine($"Decrypted text: {decryptedText}");
      }
    

Usually, you don't have to implement this when using Azure services for storing data (storage account, SQL, etc.). Azure handles the encryption and decryption for you.

Use key rotation to keep your data safe

Rotating cryptographic keys is an industry-standard. If your keys fall into the wrong hands, key rotation helps keep your data safe. In Azure Key Vault, we can define a rotation policy per key. The policy automatically rotates keys after an expiration time. It creates a new version (becomes the latest) of the key. The old versions are still available. This is important because a client of that key could have encrypted data with a specific key version. If that version is deleted, there is no way of decrypting that data. Preferably, clients of a rotated key re-encrypt the data with the latest key version. Key Vault sends messages to Eventgrid when a key is rotated. This allows clients to subscribe to these messages and take action.

Summary

SOPS is a CLI tool to encrypt secrets. It understands the structure of JSON, YAML, and INI files. SOPS only encrypts the values. The keys remain are readable, which is great for maintainability. SOPS follows the envelope encryption practice. A Data Encryption Key is used to encrypt the data, and a Key Encryption Key is used to encrypt the Data Encryption Key. SOPS integrates with KMS systems like Azure Key Vault to encrypt the data key.

In the last section of this article, I explained how you could leverage Azure Key Vault to store data at rest securely.

In my GitHub repository, you find Terraform code to create a Key Vault in Azure and an example project to encrypt/ decrypt data with Azure Key Vault.