Lorenz VanthilloLet me start at the beginning. In my role as an AWS Cloud Engineer at one of my previous clients, an event-driven architecture was used where third parties constantly sent many events to our AWS environment via EventBridge. For each third party, we provided an event bus with various EventBridge rules.
The challenge here was keeping track of the event structure—how it was organized. Events were frequently updated, leading to many meetings to clarify things.
At the end of 2019, a large part of our problem was solved thanks to EventBridge Schema Discovery. By enabling this on the event buses, schemas were automatically generated based on the received events. This allowed us to generate code bindings from these schemas, which was a great help in our object-oriented environment.
Below you can see a very basic example event of a third party.
{
"version": "0",
"id": "ef21d5fc-a5ba-e2c6-fc4b-a8807455c64d",
"detail-type": "orderType",
"source": "com.company.A",
"account": "xxx",
"time": "2024-08-22T08:04:26Z",
"region": "eu-west-1",
"resources": [],
"detail": {
"orderId": 123456789,
"customer": {
"customerId": "C001",
"name": "John Doe"
},
"orderDate": "2024-08-22"
}
}
AWS discovered a schema for these types of events:
By using the AWS Toolkit for Visual Studio Code, we could easily represent our events as strongly typed objects in our code.
Below is a very basic example on how we used the code bindings.
Output:
123456789
C001
<class 'schema.com_company_a.ordertype.OrderType.OrderType'>
<class 'dict'>
This improved our way of working, but we still encountered an issue. From time to time, third parties would add new attributes to their events. EventBridge would discover these changes, but developers often forgot to update the code bindings for the new schema. Although our implementation was robust enough to prevent breakages when new attributes were added, it resulted in new data that we weren’t utilizing. We had to rely on developers to remember to update their code bindings occasionally, and there was no clear process in place to manage this.
Sometimes a code binding wasn’t updated for months, and occasionally, two developers would update it simultaneously, leading to conflicts or duplicate work.
To handle this better, we decided to build a solution that automatically creates a Jira ticket whenever a third party updates their event and a new schema is discovered.
The solution is available in CloudFormation on my GitHub. Check the README.
The first step was to create an EventBridge rule on our default bus that would trigger whenever a new schema or a schema version update was discovered. This event was then sent to an SQS queue to serve as input for an EventBridge Pipe. Here, we could add additional filtering (optional in this example) and enrich our event using a Lambda function.
For enrichment, we used the describe_schema using boto3.
data = event[0]["input"]["detail"]
try:
response = client.describe_schema(
RegistryName=data["RegistryName"],
SchemaName=data["SchemaName"],
SchemaVersion=data["Version"],
)
except ClientError as e:
raise e
return_data = {
"SchemaName": response["SchemaName"],
"SchemaVersion": response["SchemaVersion"],
"SchemaArn": response["SchemaArn"],
"Content": json.loads(response["Content"]),
}
After enriching our data, we sent it to a Step Function workflow. This workflow, in turn, triggered the AWS-provided AWS-CreateJiraIssue SSM Automation, which automatically created a Jira ticket.
The ticket included details such as the schema name, the new schema version, and the ARN of the schema. (Additional content from the event can also be added if needed.)
+----------------+ +--------+ +-------------------------+ +----------------+ +-------------------------+
| EventBridge | ---> | SQS | ---> | EventBridge Pipe | ---> | Step Function | ---> | SSM Automation Document |
| Rule | | | | (Filtering & Enrichment)| | | | |
+----------------+ +--------+ +-------------------------+ +----------------+ +-------------------------+
Let's demo this solution. Here you see an updated event, based on the original. The attribute status is new.
{
"version": "0",
"id": "dffbd38b-9258-d028-21f3-da0ba3c9e314",
"detail-type": "orderType",
"source": "com.company.A",
"account": "xxx",
"time": "2024-08-22T08:04:26Z",
"region": "eu-west-1",
"resources": [],
"detail": {
"orderId": 123456789,
"status": "Completed",
"customer": {
"customerId": "C001",
"name": "John Doe"
},
"orderDate": "2024-08-22"
}
}
A new schema will be discovered. This will trigger the whole solution. After the Lambda enriched our event, that updated event will be used as input for our Step Function.
The input event of our Step Function is enriched and looks like this.
[
{
"statusCode": 200,
"data": {
"SchemaName": "com.company.A@OrderType",
"SchemaVersion": "2",
"SchemaArn": "arn:aws:schemas:eu-west-1:xxx:schema/discovered-schemas/com.company.A@OrderType",
"Content": {
"openapi": "3.0.0",
"info": {
"version": "1.0.0",
"title": "OrderType"
},
"paths": {},
"components": {
"schemas": {
"AWSEvent": {
"type": "object",
"required": [
"detail-type",
"resources",
"detail",
"id",
"source",
"time",
"region",
"version",
"account"
],
"x-amazon-events-detail-type": "orderType",
"x-amazon-events-source": "com.company.A",
"properties": {
"detail": {
"$ref": "#/components/schemas/OrderType"
},
"account": {
"type": "string"
},
"detail-type": {
"type": "string"
},
"id": {
"type": "string"
},
"region": {
"type": "string"
},
"resources": {
"type": "array",
"items": {
"type": "object"
}
},
"source": {
"type": "string"
},
"time": {
"type": "string",
"format": "date-time"
},
"version": {
"type": "string"
}
}
},
"OrderType": {
"type": "object",
"required": [
"orderId",
"orderDate",
"customer",
"status"
],
"properties": {
"customer": {
"$ref": "#/components/schemas/Customer"
},
"orderDate": {
"type": "string",
"format": "date"
},
"orderId": {
"type": "number"
},
"status": {
"type": "string"
}
}
},
"Customer": {
"type": "object",
"required": [
"customerId",
"name"
],
"properties": {
"customerId": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
}
}
}
}
]
The Step Function workflow will, in turn, trigger the SSM automation and a create a Jira Ticket.
For convenience, I kept the ticket content brief. However, since the Content is also sent as input to the Step Function, it could also be included in the ticket. In that way, you can directly mention the new attributes or changes to the schema in your ticket.
This solution will also be triggered when a completely new event is discovered, as it will create a version 1 of the new event, causing the EventBridge rule to fire.
In this way, we were informed about updates and could schedule them into our sprint. This leads to an acceleration of our development cycle.
I’m aware that this is quite a specific case, but similar solutions can be built by setting up EventBridge rules to trigger on the desired events, and then using enrichment and Step Functions in combination with SSM to create further automations.
