RDS-Migration-s3-dynamoDB

Venkateswara Reddy - Dec 9 '23 - - Dev Community

youtube: https://youtu.be/73HSRWiBVsE?si=m73C7bj25Nweuvvc

Image description

1.Create DMS(Replication Instance) with specific SG then t3.micro type instance.
2.Create RDS with same SG of DMS & t3.micro rest all are same.
3.We have to create targets. Here my targets are S3 & DynamoDB
4.Create a Iam roles.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.Prerequisites
5.To create IAM we can refer this link and copy paste and create custom IAM policy.

Image description

Here while creating S3 Role IAM we have to choose the DMS.
Then click on create a custom policy then copy paste the code from the above link then create role.

Creation of Target END-POINT:

In the DMS paze we can click on create end point, choose target end point & target identifier we can give any name & then choose the target engine as s3-bucket & we have to provide ARN of the bucket.
Then enter name of the bucket & then bucket folder we can give anything & we can leave end-point settings & use the same VPC what ever we used in the DMS replication & test the connection whether this endpoint is able to communicate with s3 or not & then create the end point.

Creation of Source END-POINT:

Click on DMS then choose endpoint then choose source endpoint then select the rds and choose the which rds we want to migrate.

Image description

Here we have to choose provide access information Manually.

Image description

We have to test the connection, then create the end point.
So finally we created 2 end points.

Image description

RDS:

Note down the DB end point.

IAM for Dynamo-Db To DMS:

Image description

Choose DMS then create custom policy then attach that policy and create role.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.DynamoDB.html

Image description

Upload only these 3 settings which region & Account-ID & dynamodb Name ‘employee’

Endpoint For DyanamoDb:

Image description

Click on DMS click on end point choose target endpoint then Name of the target endpoint we can enter anything & target engine is DynamoDb and ARN of the role.

Image description

Test the connection then create the end-point.
So finally 3 end points are ther.

Create ec2 instance for connecting rds:

Create a ec2 instance with same vpc and SG(inbound-mysql) what ever we used while creating RDS.
If we used different SG while creating ec2 instance we can attach that Sg in the rds by modifying the rds

Image description

Image description

Add data into databases.
Here 1st we are creating one schema if not exist then we are creating databases.

Image description

Here we are creating the table and adding the data into table.

Create the parameter groups:

Image description

Ofter creating choose that and click on edit then modify the checksm & format like this.

Image description

Open rds then click on modify under parameter groups add this parameter what ever we created.

Image description

DB Migration Task:

Image description

We have enter name as anything then select the source and target of the DB

Image description
Here we have to select the only migration or ongoing migration.

Image description

In the database what ever we mention that schema and table
Same like that for dynamo db and s3 2 migration tasks should create.

Image description

We can check the data is migrated or not.
Her we have to choose ;Query with s3 Select’

Image description

Copy data into Dynamodb little fast compared to s3.-

Image description

How many times we do modification that many files will create in s3 in .csv file format.

Image description

. . . . . . . . . . . . .