2023 Update: Instead of using Cloud KMS, you may have better success using Secret Manager, or built-in secrets

Sometimes, you just need a scheduled task to make the machine go 'beep boop'. With Google Cloud Scheduler, you can make parts of Google Cloud go 'beep boop' in a timely, but secure, manner.

In this example, we'll be creating a Twitter bot that tweets daily. This is based on the real-world example @python2sunset.

We'll be using the following components:

• Cloud Scheduler, which targets a
• Cloud Pub/Sub topic with a specific payload, which
• Cloud Functions listens to, using
• Cloud KMS encrypted environment variables.

In our example, we'll be posting to a Twitter account. You'll need a set of API keys to use with Twitter, which you can get by applying for a developer account and registering your app.

Table of contents:

• Example code: beepboop
• Configuring our actuators
• Creating encrypted secrets
• Deploying our function

Example code: beepboop

Let's work backwards through this list and start with the function:

This function accepts an event and context, which our Pub/Sub message payload will supply. We haven't created this payload yet, but we can make it anything we want. In this case, we're going to use a dictionary:

{"tweet": "🤖"}

Our event data will be a base64 encoded representation of this string, which we can decode and then safely parse to extract our data. We're only continuing on with our processing if we see a specific payload (a dictionary with a "tweet" key). Using this method we can tell our function to tweet any string, functionality we will mention later.

This function also has a third-party package dependency (python-twitter) which we'll need to ensure we include in our provisioning.

Configuring our actuators

Now we have our function code, we need the parts that make our function run (the 'actuators', "a device that causes a machine or other device to operate"). For that, we'll need to create a Pub/Sub topic and Scheduler job that will trigger our function. We'll name all of these literally, prepending our function name.

For each of these components, we'll link to the related section of the Google Cloud Platform Console with an an example screenshot, and the equivalent gcloud command. You can choose to use either or.

For the topic, we'll need a new Pub/Sub topic called beepboop-topic:

gcloud pubsub topics create beepboop-topic 

For the job, we'll need a new Cloud Scheduler job:

• called beepboop-job,
• on a daily frequency (following cron standards),
• in Greenwich Mean Time (GMT),
• targeting Pub/Sub and the beepboop-topic topic we just created,
• with our aforementioned payload {"tweet": "🤖"}

gcloud scheduler jobs create pubsub beepboop-job \
    --schedule "0 0 * * *"  \
    --topic beepboop-topic \
    --message-body "{'tweet': '🤖'}"

The schedule "0 0 * * *" means "daily at midnight" (minute 0, hour 0, every day of the month, every month of the year, and every day of the week).

Given our function passes through any value of "tweet", we could setup:

• Tweeting "🤖" at midnight, and
• tweeting "👾" at midday

by creating a second scheduled job, targeting the same topic, with these different payload values. This allows us to keep the one function code that performs multiple tasks ✨

Creating encrypted secrets

Before we get to the function, we need to do something about those four environment variables. They are super secret values that if exposed can have nasty consequences. In our case, people can tweet as us, access our account, and our direct messages.

So, we need to encrypt them. For this, we'll use Cloud KMS for encryption and decryption. There are other ways to do secrets in Google Cloud. If you're using another part of Google Cloud such as App Engine or Cloud Run, berglas (before v0.5.0) simplifies this process by wrapping around Cloud KMS. Newer versions of berglas use Secret Manager (a topic for another post).

We're going to create a key on a keyring, then encrypt our secrets, add these as environment variables for our function via the console, and decrypt them at run time.

First, we create a keyring to store our key on.

gcloud kms keyrings create beepboop-keyring --location global

Then a key on our keyring:

gcloud kms keys create beepboop-key \
    --purpose encryption \
    --keyring beepboop-keyring \
    --location global

From here, we can encrypt our secrets. For this, we'll need the combination keyring/key identifier known as the resource. For our key, this resource will be projects/beepboop-project/locations/global/keyRings/beepboop-keyring/cryptoKeys/beepboop-key

For any project, location, keyring and key, this will be:

projects/PROJECT/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY

You can generate the specific resource identifier for your key by:

• going to the Key on the Keyring in the Google Cloud Platform Console, clicking the "⋮" icon, and selecting "Copy Resource ID"
• using the gcloud CLI
  • gcloud kms keys list --location global --keyring beepboop-keyring --format="value(name)"

Now we have our key, we can encrypt our values. Since we have four values to encrypt, and later decrypt, we need to consider the most efficient way to do this.

To make our application twelve factor, we can use four separate environment variables. For each of our secret values, we can encrypt them using the same key, then decrypt them in our function. For the command-line deployment method, we can put all these values into a yaml file, which will then be processed for us. For the console method, we will have to copy the values in separately.

For ease of ensuring we don't get any encoding issues, since we're using strings, we'll use a method similar to our reference article, using base64 encoding -- but processing all the keys in bulk.

Using this small code snippet, we can take a yaml file of key-value pairs, and encrypt only the value, and returning a new set of key-encryptedvalue pairs:

Given a yaml file secrets.yaml, which contains one KEY_RESOURCE_NAME of your resource identifier, and any other number of key-value pairs, it will encrypt all of them (apart from the KEY_RESOURCE_NAME itself), and output the result into a secrets.yaml.enc file (the .enc extension is just a flag to us that this file is the encrypted version)

$ cat secrets.yaml
KEY_RESOURCE_NAME: "projects/beepboop-project/locations/global/keyRings/beepboop-keyring/cryptoKeys/beepboop-key"
CONSUMER_TOKEN: AAAAAAAAAAAAAAAAAAAAAAAA
CONSUMER_SECRET: YIJJJjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj
ACCESS_TOKEN: 1111111111111111111-0oaaaaaaaaaaaaaaaaaaaaaaaa
ACCESS_SECRET: z4hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

$ pip install google-cloud-kms pyyaml
$ python encrypt.py

$ cat secrets.yaml.enc
ACCESS_SECRET: CiQAgOOBPVx...==
ACCESS_TOKEN: CiQAgOOBPeaDhy/NP3rYYP5hc9yJnZZvw+kiIHMdbXWWQhknBQwS...==
CONSUMER_SECRET: CiQAgOOBPQm49gQd+N34gd9NyM7GoY8y//3n5qEqsbbLwxEVEv4SWAb...==
CONSUMER_TOKEN: CiQAgOOBPSp0zXOkD8q1+1tlIA7kWua0CkzncZapHOxhjIsRCS4SQgAb...==
KEY_RESOURCE_NAME: projects/beepboop-project/locations/global/keyRings/beepboop-keyring/cryptoKeys/beepboop-key

Our keys may change order, but since we're referencing them by key name, the order doesn't matter.

Side note: if you want to save this setup for later in a git repo, do not commit these files. There is no point making these values secret only to share them on the public internet 😫. To ensure you can't accidentally add them, create a .gitignore file and add "secrets.yaml*" so you don't commit them. To ensure glcoud ignores these files too, add !include:.gitignore to your .gcloudignore file. This will ignore the contents of .gitignore!

To decrypt these secrets, we'll need to adjust our original function:

We'll also need to make sure we update our requirements.txt:

We'll also need to give our function permission to decrypt using this key. (We as an admin user have rights to do this, but our function, by default, does not have permission).

gcloud kms keys add-iam-policy-binding beepboop-key \
    --location global \
    --keyring beepboop-keyring \
    --role roles/cloudkms.cryptoKeyDecrypter \
    --member serviceAccount:beepboop-project@appspot.gserviceaccount.com

The serviceAccount here is the default service account for Cloud Functions. In our local python scripts and in the gcloud command line we act as an admin user, so we have permission to do a lot of things. Our function doesn't (and really shouldn't) have all the permissions we the admin have, so we need to be explicit.

Deploying our function

And finally for the function, we'll need a new Function:

• called beepboop-function,
• triggered on the topic beepboop-topic,
• using our main.py and requirements.txt files from earlier,
• executing the function beepboop, and
• specifying our environment variables

Important note: for the --source flag you specify a folder, and everything apart from the files mentioned in .gcloudignore are uploaded. Since we've been playing with secrets files we want to super make sure we don't upload anything we don't want to. So we should create a folder containing only our code and requirements. Put the main.py and requirements.txt files in their own code/ folder and the gcloud command below will work.

gcloud functions deploy beepboop-function \
    --trigger-topic beepboop-topic \
    --runtime python37 \
    --entry-point beepboop \
    --source code/ \
    --env-vars-file secrets.yaml.enc \
    --no-allow-unauthenticated

Both these methods will result in the same five environment variables.

To test this function, you can go back to the Cloud Scheduler, and click "Run Now":

gcloud scheduler jobs run beepboop-job

If everything went to plan, we should have a response on twitter:

Beep boop!