When studying for an AWS certification, we often encounter questions that require a deep understanding of services we might not use every day. Understanding these services only at a theoretical level can lead to confusion and mistakes during the exam. So, how can we improve our comprehension and retention of these complex topics?

One of the best ways to tackle this challenge is by practicing directly with AWS services, replicating question scenarios in a real environment. In this article, I will guide you through the analysis of a specific question about VPC Flow Logs and Kinesis Data Firehose, breaking down each component and showing how you can build a similar practice in your AWS account. This hands-on approach not only reinforces key concepts but also provides real insight into how these services work in practice, helping us turn theory into applicable knowledge.

Domain: Design Solutions for Organizational Complexity

An engineering firm has deployed a critical application on the web servers of an Amazon EC2 instance launched in a VPC. The operations team is looking for a detailed analysis of the traffic from these web servers. They have enabled VPC Flow Logs on the VPC. The logs need to be analyzed using open-source tools in near real-time and visualized to create dashboards.

Proposed Solution:

Ingest the VPC Flow Logs into Amazon Kinesis Data Firehose, which will deliver these logs to Amazon OpenSearch Service for near real-time analysis and visualization of the logs.

Problem Architecture

Involved Services

VPC Flow Logs:

Captures and stores information about the network traffic entering and leaving network interfaces in your Amazon VPC. VPC Flow Logs allow recording details about network connections for analysis and monitoring.

Amazon Kinesis Data Firehose:

A real-time data ingestion service that facilitates loading large volumes of data into storage and analysis services. In this case, Kinesis Data Firehose acts as the intermediary that collects the VPC Flow Logs and sends them to Amazon OpenSearch Service.

Amazon OpenSearch Service:

An AWS-managed platform for search, analysis, and real-time data visualization. It is commonly used to work with logs and telemetry data. OpenSearch Service allows storing and analyzing VPC Flow Logs and creating dashboards to visualize them.

Steps to Replicate the Exercise:

Objective: Configure VPC Flow Logs, send the logs to Kinesis Data Firehose, and visualize the data in Amazon OpenSearch Service.

Create an OpenSearch Service domain to store and analyze the VPC Flow Logs.
Create a Firehose delivery stream to send the flow logs to the OpenSearch Service domain.
Create a VPC Flow Log subscription to the delivery stream.
Explore the VPC Flow Logs on the OpenSearch Service dashboards.
Create a role mapping with an OpenSearch Service user to the Kinesis Data Firehose service role. Since we are using a public access domain for OpenSearch Service, we need to assign the IAM role of the delivery stream to the OpenSearch Service principal user to send bulk logs to the OpenSearch Service domain.
Create an index pattern in the OpenSearch Service dashboards to enable analysis and visualization of the VPC logs.

Prerequisites

As a prerequisite, you need to create an Amazon Simple Storage Service (Amazon S3) bucket to store Firehose delivery stream backups and failed logs.

⚠️ 💰 🤑 Before proceeding, be cautious of the potential costs that may be incurred by executing these steps and remember to delete the resources after completing the exercise ⚠️ 💰 🤑

Step-by-Step Execution of the Tutorial:

Creating an S3 Bucket to Store Backups of Kinesis Data Firehose Messages:

Creating the OpenSearch domain:

After a few minutes, we will see the created domain.

Creating the Kinesis Data Firehose:

Go to the Amazon Kinesis Data Firehose console and create a new Firehose stream.

Select the domain created previously:

Create an S3 bucket to store backups.

Creating VPC Flow Logs:

Go to the VPC service and select the VPC to which you want to add the configuration.

Create a flow log:

Select the Firehose stream that we created previously.

Go to the OpenSearch dashboard.

Since we are using a public access domain for OpenSearch Service, you need to assign the role created for the Firehose delivery stream to the OpenSearch Service dashboard user so that the delivery stream can send bulk logs to the OpenSearch Service domain.

Go to Security > Roles and select the all_access role.

Copy the ARN of the role generated by Kinesis Firehose.

Return to Home, then go to Manage and select Index Patterns.

Create one with the prefix vpcflowlogs*.

At this point, we will be able to see the logs coming from the vpcflowlogs index.

We can also go to the Kinesis Data Firehose metrics to see the flow of messages between VPC and OpenSearch.

This concludes the practice for answering this AWS Architect Professional certification question.

Keep in mind that if you've never used OpenSearch or Kinesis, in this practice you covered basic concepts that, through hands-on experience, can help you retain them long-term.

References:

Question obtained from the simulator provided by https://www.whizlabs.com/
Stream VPC flow logs to Amazon OpenSearch Service via Amazon Kinesis Data Firehose

If you've enjoyed this article, feel free to give a 👏 and ⭐ to the repository.

🤔 Follow me on social media! ⏬

YouTube: https://www.youtube.com/jjoc007
Twitter: https://twitter.com/jjoc007
GitHub: https://github.com/jjoc007
Medium: https://jjoc007.com
LinkedIn: https://www.linkedin.com/in/jjoc007/

Thank you!

Analysis of AWS Certification Question: AWS Solutions Architect Professional — VPC Flow Logs — Kinesis Data Firehose