How to Migrate From Terraform Cloud

Spacelift team - Oct 14 - - Dev Community

When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates. 

Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.

Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit

Getting started with Spacelift Migration Kit

To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:

  1. Install Python3.10 or newer and Poetry
  2. Clone the Spacelift Migration Kit.
  3. Install other dependencies and the spacemk command in a Python virtual environment
  4. Create your config.yml
  5. [Optional] Audit your current setup
  6. Export the configuration
  7. Generate the Terraform/OpenTofu configuration
  8. Create a repository based on the generated code
  9. Create a stack based on the repository
  10. State Migration
  11. [Optional] Customization
  12. Useful information

Let's cover these steps in detail.

How to migrate from Terraform Cloud

1. Install prerequisites

We need to ensure that a version newer than python3.10 is installed and also that poetry is installed.

Installing poetry on MacOS using pipx

brew install pipx
pipx install poetry
pipx ensurepath

Enter fullscreen mode Exit fullscreen mode

To install it on any other operating system, check out this tutorial.

2. Clone the Spacelift migration kit

Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.

git clone git@github.com:spacelift-io/spacelift-migration-kit.git
Enter fullscreen mode Exit fullscreen mode

3. Install spacemk

Go to the Spacelift migration kit directory, and run the following:

poetry install
Creating virtualenv spacemk-liMdtQjr-py3.12 in /Users/flaviuscdinu/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
Enter fullscreen mode Exit fullscreen mode

Now after you have installed spacemk, activate the virtual environment:

poetry shell
Enter fullscreen mode Exit fullscreen mode

4. Create your config.yml

Inside the repository, there is a config.yml.example file, containing the values you need to configure for the migration:

# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
 name: # Valid values: terraform
 settings:
   # Specific to the Terraform exporter (exporter.name: terraform)
   api_endpoint: https://app.terraform.io
   api_token:
   include:
     workspaces: ^example-.*$

generator:
 extra_vars:
   foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"

github:
 api_token:
 endpoint: https://api.github.com

spacelift:
 api:
   api_key_endpoint: https://<ACCOUNT NAME>>.app.spacelift.io/graphql
   api_key_id:
   api_key_secret:
Enter fullscreen mode Exit fullscreen mode

Rename this file, or create a new one that should be called "config.yml".

In this file, you must add the login details to your Terraform Cloud account and your Spacelift account. You can also use environment variables inside this file, to avoid using credentials inside the file (these variables should be prefixed with the "$" sign).

Here is an example that uses environment variables:

# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
 name: terraform
 settings:
   # Specific to the Terraform exporter (exporter.name: terraform)
   api_endpoint: https://app.terraform.io
   api_token: $TFC_TOKEN

generator:
 extra_vars:
   foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"

github:
 api_token: $GITHUB_TOKEN
 endpoint: https://api.github.com

spacelift:
 api:
   api_key_endpoint: https://saturnhead.app.spacelift.io/graphql
   api_key_id: $SPACELIFT_API_KEY_ID
   api_key_secret: $SPACELIFT_API_KEY_SECRET
Enter fullscreen mode Exit fullscreen mode

You will need to export these values inside of your terminal (e.g.):

export TFC_TOKEN=tfctoken
Enter fullscreen mode Exit fullscreen mode

Based on the values above, we will include all workspaces in the migration.

If you don't already have a Spacelift API key, you can go to Settings → API keys and generate one:

tfc migration create spacelift api

After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.

To create an API key for Terraform Cloud check this tutorial, and to create an GitHub token, check this tutorial.

5. [Optional] Audit your current setup

To audit your current setup, you can simply run the spacemk audit command:

Agent Pools: 1 (including 1 with warnings)
Modules: 1
Organizations: 1
Policies: 1 (including 1 with warnings)
Policy Sets: 1
Projects: 4
Providers: 0
Tasks: 0
Teams: 601
Variable Set Variables: 0
Variable Sets: 0
Workspace Variables: 0
Workspaces: 1
Enter fullscreen mode Exit fullscreen mode

This will give you all the information about your Terraform Cloud account, and everything that can be exported.

6. Export your configuration

To export your configuration, simply run spacemk export. This command will generate a json file in the tmp directory, called data.json.

In this file, you will see information about everything that will be migrated and how they are mapping out to Spacelift resources.

{
 "context_variables": [],
 "contexts": [],
 "modules": [
   {
     "_migration_id": "aks",
     "_relationships": {
       "space": {
         "_migration_id": "saturnhead",
         "_source_id": "saturnhead",
         "name": "saturnhead",
         "requires_terraform_workflow_tool": false
       }
     },
     "_source_id": "...",
     "name": "aks",
     "status": "setup_complete",
     "terraform_provider": "az",
     "vcs": {
       "branch": "main",
       "namespace": "flavius-dinu",
       "provider": "github_custom",
       "repository": "terraform-az-aks"
     },
     "visibility": "private"
   }
 ],
 "spaces": [
   {
     "_migration_id": "saturnhead",
     "_relationships": {},
     "_source_id": "saturnhead",
     "name": "saturnhead",
     "requires_terraform_workflow_tool": false
   },
   {
     "_migration_id": "default_project",
     "_relationships": {},
     "_source_id": "...",
     "name": "Default Project",
     "requires_terraform_workflow_tool": false
   },
   ...
   {
     "_migration_id": "sdsd",
     "_relationships": {},
     "_source_id": "...",
     "name": "sdsd",
     "requires_terraform_workflow_tool": false
   }
 ],
 "stack_variables": [],
 "stacks": [
   {
     "_migration_id": "blog_examples",
     "_relationships": {
       "space": {
         "_migration_id": "default_project",
         "_source_id": "...",
         "name": "Default Project",
         "requires_terraform_workflow_tool": false
       }
     },
     "_source_id": "...",
     "autodeploy": false,
     "description": null,
     "has_secret_variables_with_invalid_name": false,
     "has_variables_with_invalid_name": false,
     "name": "blog-examples",
     "slug": "blog-examples",
     "terraform": {
       "version": "1.5.5",
       "workflow_tool": "TERRAFORM_FOSS"
     },
     "vcs": {
       "branch": "main",
       "namespace": "flavius-dinu",
       "project_root": "migration",
       "provider": "github_custom",
       "repository": "blog-examples"
     }
   }
 ]
}
Enter fullscreen mode Exit fullscreen mode

This file can be reviewed and modified to accommodate your needs.

💡 You might also like:

7. Generate the Terraform/OpenTofu configuration

To generate the Terraform/OpenTofu configuration, you will need to use the spacemk generate command.

terraform {
 required_providers {
   spacelift = {
     source  = "spacelift-io/spacelift"
     version = "~> 1.0"
   }
 }
}

resource "spacelift_space" "saturnhead" {
 inherit_entities = true
 name             = "saturnhead"
 parent_space_id  = "root"
}

resource "spacelift_stack" "default_project_blog_examples" {
 branch            = "main"
 manage_state      = false
 name              = "blog-examples"
 project_root      = "migration"
 repository        = "blog-examples"
 space_id          = spacelift_space.default_project.id
 terraform_version = "1.5.5"

 github_enterprise {
   namespace = "flavius-dinu"
 }
}

...

resource "spacelift_module" "saturnhead_aks" {
 branch             = "main"
 name               = "aks"
 repository         = "terraform-az-aks"
 space_id           = spacelift_space.saturnhead.id
 terraform_provider = "az"

 github_enterprise {
   namespace = "flavius-dinu"
 }
}
Enter fullscreen mode Exit fullscreen mode

This will generate the Spacelift configuration required for the migration. Again, the code can be modified to accommodate your needs.

If you are not using the default VCS integration, you will also need to specify the id property in the vcs block (in my case, I will add an id to the github_enterprise block):

github_enterprise {
   namespace = "flavius-dinu"
   id        = "flavius-dinu"
 }
Enter fullscreen mode Exit fullscreen mode

8. Create a repository based on this code

In your VCS, simply create a repository with the exported code. Ensure the vcs provider you are using is configured in Spacelift, and the code repositories you've exported from TFC are available in this vcs provider.

To configure a new VCS provider inside your Spacelift provider, follow the steps from this tutorial for your VCS.

Create the repo with the exported code, this is how it looks like for my GitHub configuration:

tfc migrate kit repository

9. Create a stack based on the repository

Go to your Spacelift account, select Stacks, and the Create Stack option:

create import tfc stack

Add a name to your stack, select a space, and add optional labels and description.

connect to source code tfc migration

In the next step, select the repository containing your migration code as shown above, and specify the branch you want to use.

choose vendor tfc migration kit

Next, you can choose your vendor, it can be either Terraform, or OpenTofu, so I will accept the default Terraform FOSS workflow.

spacelift stack settings

Ensure that the Administrative option is toggled on in the define behavior tab, and accept all other defaults to create the stack.

After the stack is created, trigger a run:

spacelift trigger run stack

After reaching the unconfirmed state, you will see all the resources that it will create, in my case, there are 5 spaces, 1 stack, and 1 module.

spacelift resources tfc migrate

You can easily see that all resources have been migrated successfully.

10. State Migration

If we go to our stack and select resources, we will see that there are no resources in the state:

spacelift stack resources stack

In TFC, however, we have a couple of resources created:

tfc resources created

In our local environment, when we've done the export, the state files were also downloaded:

tfc migration state files

To upload our state files, we must run the following command: spacemk import-state-files-to-spacelift.

Note: if you have changed the name of the stack in your terraform configuration, you will also need to modify the data.json file, to reflect the stack name, when importing the state.

After running the command, if we go back to our stack, we can see the state has been migrated successfully:

successfull state migration

11. [Optional] Customization

While the Migration Kit does a great job when it comes to migrating to Spacelift, we understand that every migration is different and you may need to customize the functionality of the automation. To do that, you can easily implement your own commands and even your own exporter. More details about this can be found here.

Key points

This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud's new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.

To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.

This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.

Written by Flavius Dinu

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .