Cloud Seed - Deploy faster with GitLab 🚀

Were you at the Cloud Next in October? This big annual Google event where a lot of new features around Google and its cloud platform GCP are announced. This year, there was one linked to GitLab : Cloud Seed.

⛅ 🌱Cloud Seed ?

What is Cloud Seed? This is a new, open source project launched by GitLab Incubation at the end of 2021 and lead by by Sri Rangan.

The objective of Cloud Seed is very simple. It’s to accelerate deployment of your application from GitLab to Google Cloud Platform. This project was opened first for beta testers, we were less than 100 people to try out this project and give some feedback.

At the Cloud Next, Sri announced the first release of Cloud Seed and his opening to everyone. This talk is available on this page.

⏪ Before Cloud Seed, deployments looked like …

Before Cloud Seed, deployments on GCP could be done with several methods. In all cases, continuous deployment (and integration) in GitLab is done by a GitLab homemade tool: GitLab CI.

GitLab CI is “a tool for software development using the continuous methodologies: continuous integration, continuous delivery and continuous deployment”. Based on the Go language, GitLab Runner made these CI/CD operations, defined in pipelines, very fast.

On GitLab CI, there are two concepts to know: “stage” and “job”.

Stage is a step of your pipeline. Like “test”, “build”, “deploy” to test, build and deploy your application.
Job is a little entity that is going to make one operation on your application. Some jobs could be on the same stage.

I’ve created this cheat sheet to explain basics of GitLab CI pipeline

Come back to our deployment on GCP. I focus on solutions only based on GitLab CI. I voluntarily exclude external tools like Ansible and Terraform that can be called from a GitLab CI pipeline. I see two options to deploy a project on GCP.

First one, it’s to use the gcloud Command Line Interface (CLI) as you can use on your computer. A new job based on the image “google/cloud-sdk” allows you to avoid installing this CLI. After configuration and authentication of your project, you can use gcloud command to deploy a Cloud Function, an application with App engine, etc. The GitLab CI job for a deployment of a Cloud Function could look like:

deploy-with-gcloud-image:
  [...]
  image: google/cloud-sdk:&lt;verion>
  script:
    - echo $GCP_SERVICE_KEY | base64 -d > /tmp/gcloud-service-key.json
    - gcloud auth activate-service-account --key-file /tmp/gcloud-service-key.json
    - gcloud functions deploy my_function [...]

If you are working on a bigger project deployed on Google Kubernetes Engine (GKE), you can use the Kubernetes Agent created by GitLab one year ago. This agent replaces an older version of Kubernetes integration, depreciated in 14.5 version and removed in the GitLab version 15 released last year.

How does it work? You configure an agent on your GitLab project. An agent is one or more yaml files. After declaring in the GitLab UI where your agent is, GitLab gives you a docker command to install it on your Kubernetes cluster. And that’s all. After this, each commit on your project will be detected by the agent and your application will be deployed automatically. Nice!

If, despite this, you want or have to deploy anything with a GitLab CI pipeline, this Kubernetes integration offers a “CI/CD tunnel”. This agent is going to simplify the connection between GitLab and your Kubernetes cluster. In a job based on the image “bitnami/kubectl”, after declared your Kubernetes context linked to your agent, you can simply apply Kubernetes yaml file, for example:

deploy-with-cicd-tunnel:
  before_script:
    - kubectl config use-context "group/project:agentk"
  image:
    name: bitnami/kubectl:&lt;version>
    entrypoint: [""]
  script:
    - kubectl apply -f &lt;my kubernetes file>

I created this cheat sheet on this Kubernetes integration:

🚀 Cloud Seed, the deployment facilitator

The main objective of Cloud Seed is to accelerate the deployment of your application. In the previous options, you could see that you need to declare and write a pipeline to deploy your application. Nothing really complicated but this is code mainly copied from other pipelines.

This is on this point that Cloud Seed steps in.

Others objectives are hidden behind the first. If applications deployment in the Cloud is faster and easier, this allows to attract people to decide a migration on the Cloud. Next, if applications are on Cloud, they can use Cloud services, modernizing them.

🔎 How Cloud Seed works ?

Cloud Seed is available on GitLab in the Menu “Infrastructure > Google Cloud”. This allows you to see a new page containing three zones: “Configuration”, “Deployments” et “Database”..

Configuration

The first section is about configuration of connection with your Google account. After entering and choosing your Gmail address, you can set the name of your GCP project you have created before. Next, specify on which branch you can create a service account.

During my tests I didn’t try to create some elements on a specific branch. This is for later.

After this first step, you will see this new service account on the screen. And the last thing to do is to configure a GCP region to execute your actions. By default it’s “us-central1”.

These operations impacted your CI/CD variables. You can check on the “Settings > CI/CD > Variables” menu. Before Cloud Seed, these declarations had to be manually.

Deployments

Now your configuration is done, we can focus on the main objective to Cloud Seed: the deployment of your application.

At the time I write this blog post, there are two deployment services visibles: Cloud Run and Cloud Storage. The first is the only one enabled for this moment. You may request support for additional Google Cloud services via a Cloud Seed feature request. This helps the open source community understand the needs and define future roadmaps.

Cloud Run is a serverless platform which allows you to focus on development. Cloud Run builds for you your application from a Dockerfile, or if this file is missing, try to determine which typology your project is from a list of buildpacks.

Cloud Seed configures your project for Cloud Run with a merge request containing a modification only on the .gitlab-ci.yml file.

include:
- remote: https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/library/-/raw/main/gcp/cloud-run.gitlab-ci.yml

This modification creates a link with another GitLab project, belonging to GitLab, that the objective is to have CI/CD templates for the different GCP services.

Looking this yml file in detail, we can see that the job is simple:

deploy-to-cloud-run:
  stage: deploy
  image: registry.gitlab.com/gitlab-org/incubation-engineering/five-minute-production/library/google-cloud-sdk-for-gitlab:main
  script:
    - cp /cloud-run.sh .
    - ./cloud-run.sh
[...]

The job included in our project will be run on the deploy stage and will create a job called deploy-to-cloud-run. The job executes the script cloud-run.sh, saved in the same project. It’s a shell script which verifies the presence of all variables it needs and executes a gcloud command.

That’s all. If you have a Dockerfile or a project listing in the available Google buildpacks, your application will be deployed on Cloud Run with this job.

If you have a doubt, you can see in the GCP console or with the CLI that you have a service Cloud Run running.

The application, in my example a Quarkus API, is available! 🎉

In the deploy-to-cloud-run job, there is the notion of an environment I previously hide:

environment:
    name: $CI_COMMIT_REF_NAME
    url: $DYNAMIC_URL

These three lines are going to create, in your GitLab project, an environment by branch. Environment has no direct action on GCP but GitLab displays information and creates links with your services.

For example, in the Environment menu I have three branches, only two getting a Cloud Run service associated. “Open” buttons allow you to open your API.

⚠️ For this moment, “Stop” button don’t remove the dedicated Cloud Run services. It only removes the environment on GitLab.

Databases

The last part talks about Database. The objective is simple: allow you to create and get information about databases you need in your project. You can see on the GitLab UI that you can create an instance of Cloud SQL database (Postgres, MySQL or SQL Server) directly on GitLab with some few clicks.

For example, for a Postgres database, you have to give some information, like the name, the machine type, the Postgres version and a reference which allows you to create a link between a branch of your project and an instance of a database.

After this step, of course you can see your instance on the GCP console or with the CLI:

On GitLab, some information are displayed:

Perhaps more information will be displayed in the future, I created an issue to ask that 😄.

🗺️ The roadmap

Like all GitLab projects, the Cloud Seed roadmap is available for everyone: https://gitlab.com/groups/gitlab-org/incubation-engineering/five-minute-production/-/boards

We can see on this that scripts for Cloud Function and App engine are planned for and if you have experience with this, you can contribute!

🔭 The future of Cloud Seed

After some first tests, I was so happy to see that, as a developer I am, I can avoid writing CI/CD jobs for configuring and writing gcloud commands. I agree, it’s not really difficult, but when I develop, I would like to focus on this task, not on scripting gcloud commands. It was very simple to configure your project, feedback is welcome and on discord we can exchange. The GitLab community is once again at the top!

I had this chance to test Cloud Seed during the beta testing phase and this allows me to give my feedback and get a correction faster. So it was logical to start migrating my personal projects interacting with GCP on Cloud Run.

GitLab is a company with the slogan “everyone can contribute”. Contributions, as in all GitLab projects, are welcome and give feedback, try to resolve issues, make Cloud Seed grow and be greater than today! A discord existed and allowed to exchange directly with GitLab members and got responses quickly. Very appreciable!

Some evolutions are planned for in the roadmap on others GCP services and I am waiting with impatience to test this and migrate others projects!

To explain quickly what is Cloud Seed, I created this 14th GitLab cheat sheet:

I have the chance to present Cloud Seed at the DevFest Nantes 2022, the second developer conference in France. The video is available on this link.

If you have any questions or comments about this blog post, you can ping me on Twitter 👋or on the Cloud Seed project Twitter account @OpenCloudSeed.