Cron-ing in the Cloud: How to use Cloud Scheduler to automate routine tasks

Christopher Wilcox - Feb 18 '21 - - Dev Community

Cloud Scheduler triggered Cloud Functions

Last week I wrote about taming my GitHub notifications and I thought I would follow that post up by talking about automating the automation - or - how I use the cloud as my own personal computer :)

I currently do not have a desktop style computer. As part of traveling and liking the ability to move about my house as I do my work, I have moved to relying on laptop computers. There is a downside to this though: I don’t really have a place to easily run scheduled tasks as I never know if the computer will be on at that time. Enter Cloud Scheduler.

Running scripts on a schedule, or an external trigger, is a common practice and Google Cloud Scheduler makes this possible while not having to be sure I have a computer running in my home. Further, the free tier of Cloud Scheduler allows you to specify 3 jobs for free, no matter how many times the job is run. Each job after that is USD$0.10 each. Do note you may also incur costs for excess Cloud Functions execution time, database costs, etc.

Using my script I created for managing my GitHub notifications as motivation, let’s break down how to use Cloud Scheduler and Google Cloud Functions to routinely execute code. Also, shoutout to my coworkers Katie McLaughlin and Martin Omander for their video about this very subject. Check it out on YouTube if that style is easier for you. They leverage Cloud Run instead of Cloud Functions, but it is very similar to this post.

An overview

So, we already have a bit of code running locally. At a high-level to get this in the cloud we need to

  1. Wrap it in an entry point that can be invoked by a Cloud Function
  2. Deploy the code to Cloud Functions using gcloud, a CLI that helps to interact with Google Cloud
  3. Create a Cloud Scheduler job that triggers the Cloud Function on a routine.

Modifying the script to work with Cloud Functions

From my last post, we have the following code to start from.

import getpass
from github3 import login
import keyring

access_token = keyring.get_password('github', 'notifications')
if access_token == None:
   access_token = getpass.getpass("Please enter your GitHub access token (will be saved to keyring)")
   keyring.set_password('github', 'notifications', access_token)

gh = login(token=access_token)

notifications_to_mark = []
for notification in gh.notifications():
   url_segments = notification.subject['url'].split('/')
   number = url_segments[-1]
   repo = url_segments[-3]
   org = url_segments[-4]

   try:
       if notification.subject['type'] == 'PullRequest':
           pr = gh.repository(org, repo).pull_request(int(number))
           if pr.state == 'closed':
               notifications_to_mark.append(notification)

       elif notification.subject['type'] == 'Issue':
           issue = gh.repository(org, repo).issue(int(number))
           if issue.state == 'closed':
               notifications_to_mark.append(notification)
   except Exception as e:
       print(e)

unique_repos = set([n.subject['url'][:-1] for n in notifications_to_mark])
print(f"Found {len(notifications_to_mark)} to mark read across {len(unique_repos)} repositories")

input("Press ENTER to mark items read.")

for n in notifications_to_mark:
   print(f"Marking {n.subject['url']}")
   n.mark()
Enter fullscreen mode Exit fullscreen mode

There are a few things we need to change in this code to make it work as a Cloud Function:

  1. This code is formatted as a runnable script, not a python module.

  2. Cloud Functions cannot request access to keyring while running

  3. As Cloud Functions run an HTTP endpoint, they expect a response.

Create an entrypoint for cloud functions

The next step to take is to move this from being a script that is exec’d on the command line, and move it to a script that a Cloud Function runner can invoke. Cloud Functions expects a Python function so the simplest thing to do is move the code that was previously written under a function.

def mark_read(request):
    gh = login(token=access_token)
    notifications_to_mark = []
    for notification in gh.notifications():
        ...
Enter fullscreen mode Exit fullscreen mode

Replacing keyring use with a Cloud Functions environment variable

As I initially created this to run on my laptop I leveraged a package calledkeyring to store my access token. However, this requires the user to enter their password to gain access to the secret. Since moving to Cloud Functions makes this non-interactive, I moved the secret to an environment variable.

Before 

access_token = keyring.get_password('github', 'notifications')
gh = login(token=access_token)
Enter fullscreen mode Exit fullscreen mode

After 

access_token = os.getenv("GITHUB_ACCESS_TOKEN") # 🤔
gh = login(token=access_token)
Enter fullscreen mode Exit fullscreen mode

You might be saying to yourself “but wait, that is just a plain text environment variable. Should I really just put that like that in the cloud? That seems bad”.

And that would be a good instinct. While you could do that, it would be best to instead use an encrypted secret. Luckily this is relatively straightforward using secret manager. Let’s add a new function, get_access_token to abstract that code from the notifications processor.

Later on this post discusses how to add that secret toGoogle Cloud Secret Manager.

from google.cloud import secretmanager

def get_access_token():
    if "GITHUB_ACCESS_TOKEN_LOCATION" in os.environ:
        client = secretmanager.SecretManagerServiceClient()
        name=os.getenv("GITHUB_ACCESS_TOKEN_LOCATION")
        access_token = client.access_secret_version(name=name).payload.data.decode("UTF-8")
    elif "GITHUB_ACCESS_TOKEN" in os.environ:
        # Allow the use of an environment variable. Though it would be better
        # if a Cloud Secret was used.
        access_token = os.getenv("GITHUB_ACCESS_TOKEN")

    return access_token

def mark_read(request):
    gh = login(token=get_access_token())
   ...
Enter fullscreen mode Exit fullscreen mode

Add a return value

For a Cloud Function to be considered ‘successful’ it should return content. For this reason, I change the final print to return that information. Whatever is returned will be output as the response to the POST request.

    return f"Marked {len(notifications_to_mark)} read across {len(unique_repos)} repositories"
Enter fullscreen mode Exit fullscreen mode

Also, print statements are captured by Google Cloud Logging. Cloud Functions includes simple runtime logging and will gather stdout and stderr. So, while a logging library could be used, print statements are sufficient.

The End Result

So after these changes, the script is:

# requirements.txt
github3.py==1.3.0
google-cloud-secret-manager==2.2.0


import os
from github3 import login
from google.cloud import secretmanager

def get_access_token():
    if "GITHUB_ACCESS_TOKEN_LOCATION" in os.environ:
        client = secretmanager.SecretManagerServiceClient()
        name = os.getenv("GITHUB_ACCESS_TOKEN_LOCATION")
        access_token = client.access_secret_version(name=name).payload.data.decode(
            "UTF-8"
        )
    elif "GITHUB_ACCESS_TOKEN" in os.environ:
        # Allow the use of an environment variable. Though it would be better
        # if a Cloud Secret was used.
        access_token = os.getenv("GITHUB_ACCESS_TOKEN")

    return access_token

def mark_read(request):
    gh = login(token=get_access_token())
    notifications_to_mark = []
    for notification in gh.notifications():
        # url is of form https://api.github.com/repos/googleapis/nodejs-dialogflow/pulls/264'
        # state change indicates pr/issue status change
        url_segments = notification.subject["url"].split("/")
        number = url_segments[-1]
        repo = url_segments[-3]
        org = url_segments[-4]
        try:
            if notification.subject["type"] == "PullRequest":
                pr = gh.repository(org, repo).pull_request(int(number))
                if pr.state == "closed":
                    notifications_to_mark.append(notification)

            elif notification.subject["type"] == "Issue":
                issue = gh.repository(org, repo).issue(int(number))
                if issue.state == "closed":
                    notifications_to_mark.append(notification)
        except Exception as e:
            print(e)

    unique_repos = set([n.subject["url"][:-1] for n in notifications_to_mark])
    print(
        f"Found {len(notifications_to_mark)} to mark read across {len(unique_repos)} repositories"
    )

    for n in notifications_to_mark:
        print(f"Marking {n.subject['url']}")
        n.mark()

    response = f"Marked {len(notifications_to_mark)} read across {len(unique_repos)} repositories"
    print(response)
    return response
Enter fullscreen mode Exit fullscreen mode

Using gcloud to interact with Google Cloud

Now that we have a Cloud Function ready Python module, we need to deploy the code to a Cloud Function. While you could use the Cloud Consoleto do everything below, I’ll be showing how to use gcloud to do this.

You can find instructions for installing gcloud on a variety of platforms here

First thing to do is to create a few environment variables to be used by future commands:

REGION=us-central1
PROJECT_ID=my-project
GITHUB_ACCESS_TOKEN=your-access-token
Enter fullscreen mode Exit fullscreen mode

Next, before we move on, it is a good idea to ensure you are logged in and addressing the correct project.

gcloud auth login
gcloud config set project $PROJECT_ID
Enter fullscreen mode Exit fullscreen mode

You don’t need to run this every time before running commands but, especially if you use multiple cloud projects, it is good to ensure you are targeting the right project.

Permissions

Whenever deploying anything to the cloud, it is good to start by thinking about access policies. In Google Cloud, we can create service accounts with limited permissions. For this instance, the service account needs the ability to invoke a Cloud Function.

This will be used later when configuring Cloud Scheduler.

gcloud iam service-accounts create gh-notifications-sa \
   --display-name "gh notifications service account"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
   --member serviceAccount:gh-notifications-sa@${PROJECT_ID}.iam.gserviceaccount.com \
   --role roles/cloudfunctions.invoker

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
   --member serviceAccount:gh-notifications-sa@${PROJECT_ID}.iam.gserviceaccount.com \
   --role roles/secretmanager.secretAccessor
Enter fullscreen mode Exit fullscreen mode

Add a secret to Secret Manager

Earlier a function was added to use a secret rather than a local environment variable.

Now it’s time to add that secret to Google Cloud.

echo "s3cr3t" | gcloud secrets create github-access-token --data-file=- --replication-policy=automatic
GITHUB_ACCESS_TOKEN_LOCATION=="projects/$PROJECT_ID/secrets/github-access-token/versions/latest"
Enter fullscreen mode Exit fullscreen mode

Deploying your Cloud Function

The next step, now that we have a Service Account, is to deploy the cloud function.

gcloud functions deploy scrub-github-notifications \
      --trigger-http \
      --region us-central1 \
      --runtime python39 \
      --entry-point mark_read \
      --service-account gh-notifications-sa@${PROJECT_ID}.iam.gserviceaccount.com \
      --no-allow-unauthenticated \
      --set-env-vars GITHUB_ACCESS_TOKEN_LOCATION=${GITHUB_ACCESS_TOKEN_LOCATION}
Enter fullscreen mode Exit fullscreen mode

Breaking this down, we use an HTTP Trigger, to run a Python 3.9 function, in the us-central1 region. This function doesn’t allow unauthenticated access which prevents callers that aren’t the specified service account from invoking it. The command also provides an environment variable to configure and the name of the entry-point.

Creating a Cloud Scheduler

Next the Cloud Scheduler that invokes the Cloud Function.

gcloud scheduler jobs create http scrub-gh-notifications-job \
  --description "Scrub GitHub Notifications hourly working hours" \
  --schedule "0 07-18 * * *" \
  --time-zone "US/Pacific" \
  --uri "https://${REGION}-${PROJECT_ID}.cloudfunctions.net/scrub-github-notifications" \
  --http-method POST \
  --oidc-service-account-email gh-notifications-sa@${PROJECT_ID}.iam.gserviceaccount.com \
  --message-body '{"name": "Triggered by Cloud Scheduler"}'
Enter fullscreen mode Exit fullscreen mode

The schedule arguments

Let’s start by looking at the arguments related to frequency.

  --schedule "0 07-18 * * *"
  --time-zone "US/Pacific"
Enter fullscreen mode Exit fullscreen mode

Cloud Scheduler uses unix-cron format. The above says “between 7AM and 6PM, in the specified timezone, run every hour”

The trigger

Now, on to what we are triggering:

  --uri "https://${REGION}-${PROJECT_ID}.cloudfunctions.net/scrub-github-notifications" \
  --http-method POST \
  --oidc-service-account-email gh-notifications-sa@${PROJECT_ID}.iam.gserviceaccount.com \
  --message-body '{"name": "Triggered by Cloud Scheduler"}'
Enter fullscreen mode Exit fullscreen mode

This will make a POST Request to the specified uri, using the service account. It also sends a message-body which, since in this case we don’t require input, just states what triggered the function.

And just like that, my other computer is the cloud

And just like that, you can now let everyone know that you use the cloud as your own personal computer!

For some bonus content, did you know you can test your cloud functions locally? You can use the Python Functions Framework to run your Cloud Function on your machine and experiment before deploying!

pip install functions_framework
functions-framework --target mark_read
Enter fullscreen mode Exit fullscreen mode

For a more in depth look at this you can read more in this postby Dustin Ingram.

Originally published at https://chriswilcox.dev on 18 February 2021

Image
8 Tips To Enhance Your Repair Car Key Game
Image
The People Closest To Traffic Accident Lawyer Near Me Tell You Some Big Secrets
Image
A Look Inside The Secrets Of Mental Health Practitioners
Image
Stwórz Swoją Oazę Spokoju: Najlepsze Meble do Sypialni
Image
Five Reasons To Join An Online Lost Auto Key Replacement Shop And 5 Reasons To Not
Image
The Role of Warehousing Solutions in E-commerce and Retail Success
Image
Not So Hacktoberfest...!

devchallenge, hacktoberfestchallenge, hacktoberfest, opensource
Image
How To Outsmart Your Boss On Houston Asbestos Attorney
Image
10 Self Propelled Wheelchair With Suspension-Friendly Habits To Be Healthy
Image
7. Understanding the Circuit Breaker Pattern

systemdesign, springboot, microservices, knowledgebytes
Image
The Importance of LinkedIn Profile Pictures for Professional Branding
Image
How To Recognize The Gas Boiler Engineer Right For You
Image
Automate Your Scripts with Systemd Services: Benefits and Step-by-Step Guide

automation, linux, cli
Image
Ten Lpg Gas Engineer Near Me That Will Actually Change Your Life
Image
web-site judi online slots terunggul yang berada pada indonesia
Image
How To Make A Profitable Replace Lost Car Keys If You're Not Business-Savvy
Image
The Most Hilarious Complaints We've Seen About Gas Safe Registered Engineer Near Me
Image
Metode Memisah Agen Judi Online Sah serta Gadungan
Image
What NOT To Do In The Lost My Car Keys Industry
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .