Jesse WardenTesting & CICD in AWS is something I'm learning. It's hard because ... there seems to be this "AWS way to do things" and "rest of the world", but even the AWS way isn't always documented.
Take integration tests for a Back-end for Front-end...
It's straightforward to take your unit tests using dependency injection, and simply inject real concretes, and voila... integration tests. In a Gitlab/Github pipeline, you can see 'em as a separate test.
AWS? Dude, use CodeDeploy green blue with hooks.
Hooks have a pre/post; you:
- have the pre run integration tests against existing code in QA
- deploy new code and wait for alarms
- if no alarms, 100% traffic is there, then have the post run integration tests against new code in QA
Contrast this with a Gitlab pipeline...
You'd have unit tests, deploy to QA, run integration tests against QA... if they fail, you don't deploy to higher environments. Sure, you could test existing QA. Sure, you could also add manual rollback code.
But why? AWS does it for you, and you can nuke 2 steps now.
The tradeoff? My integration tests are now run in a Lambda, and you have to do some weirdness to get 'em to work locally via Mocha/Jest, etc. It can work, but NOT traditionally how it's done. It's also not homogeneous in the same test suite setup.
Meaning, your integration test code is now a module vs. just a bunch of procedural code in a test module. This means you import into the Lambda and use, and possibly import into the integration tests and use there too if you want to run locally. Cool, but strange, right?
UI's with Amplify is even stranger. Almost all of us use Cypress where we can. Traditionally this is some Cypress Docker container run in the pipeline, but regular Cypress run locally. With Amplify, they manage the container and they manage the pipeline. However...
The tests run before you deploy the code. Now, if some or all of your tests are black/e2e, great, you're still technically testing your UI code against all the services before you deploy it. Deploying UI code is way different than BFF/API in that the infra isn't as huge a deal.
Yes, CloudFront URL's, Route53 on top, or a 3rd party like we use such as CloudFlare can massively impact "well, your code doesn't work when deployed... thanks for nothing, e2e tests". But that stuff isn't transient much. Once you figure it out, you're good.
Except you're not. UI's always take the blame when things break, despite Alarms being proactive with good monitoring, because that's how most people see things are broken. So it behooves you to not only test after you deploy, but also on a cadence. Enter Synthetic Canaries.
Both UI's & API's, they're Lambdas that run e2e tests using Puppeteer if you need it against your UI's &/or API's on some cadence like "every hour". This means even if you don't change the code for 6 months, you'll know possibly before your customers & coworkers something broke.
Now, hold up... we're using both Cypress and Puppeteer now? And... how do we run Puppeteer Canary code locally? This is messy. Most of us in the UI world like Cypress because despite it's procedural style API, it's much more deterministic. Less flaky tests. More confidence.
Puppeteer is the new Selenium. It's powerful, low level, but doesn't have all the high level API goodness that Cypress has. So the theory is you just do the bear minimum to "see if something is wrong" vs. the Behavior Driven Development style of e2e tests that test user stories.
The library is different too; you use require('Synthetics') vs. require('puppeteer'). Fine, we can use dependency injection to allow us run it locally vs. when in the Lambda. Again, though... more of this attitude of "use AWS and the pipeline, forget about quick, local testing".
That works when you're done, but not when you're actively developing and writing the actual tests. This fast feedback cycle is the most important thing, and "debugging your Canary code on deployment 17 of the day" is the path to misery.
Once I figure out how to get this "CodeDeploy pre/post Lambda hooks local vs. remote" and "Synthetic Canary local vs. remote" code down, I'll post how you can utilize in your local build + remote so they act the same, yet allow you to build 'em quickly.
For now... holy crap it's hard to get a standardized CICD pipeline between both UI and API's on AWS. They are just different, and AWS has approached how you build, test, and deploy 'em differently so your perspective needs to shift. Thankfully the skills required are the same.
I will say while I'm super impressed with Gitlab based on what I've learned the past 11 months, I still have this feeling that "things would be easier if I just let CodeDeploy do all the things...". Heavy lift, tho, I reckon.
API's appear to have this "no downtime, safe deployments with automatic rollback" style whereas the UI's have this "fail forward" mentality. It really makes "what you build and in what order and deployment frequency" shift. Just writing stories for co-workers in JIRA is TOUGH af.
