Migrating your data seems easy theoretically, but when it comes to the technical implementation, different AWS solutions and tools can really get you confused about which offering to choose from.
Different strategies and protocols are put into place when migration is done such as the migration strategy would be different with Amazon S3 Storage services. Various questions pop in one’s head regarding migration, for instance which tool to use, AWS Lambda or Amazon Kinesis Firehose, for processing changes. VaporVM provides you with an easy to follow ‘Go-To-Migration’ mentioned below:
The AWS Database Migration Service (AWS DMS) supports Amazon S3 as a migration target. The services enable you to extract information from any database supported by DMS and write it to Amazon S3 in a format that can be used by almost any application. You can extract the entire database and replicate ongoing changes including additions, deletions, and updates using change data capture (CDC) technology. You can even process the changes with AWS Lambda or Amazon Kinesis Firehose. By using Lambda or Firehose, you can extend the capabilities of AWS services to existing database environments, both those within AWS and in other locations.
Overview of the Example
Let’s consider an example that shows how to bring these services together. We use the AWS Database Migration service to migrate Twitter statistics produced by the City of Seattle, Washington from Amazon RDS for PostgreSQL to Amazon S3 and use AWS Lambda for analysis. We see how DMS handles both the migration of the initial database contents and ongoing change data capture. Here’s a diagram of what we plan to do:
VaporVM will do the following for you:
- Create two datasets containing Twitter account statistics. One initializes a database table. The other one performs subsequent updates to that table, processed by the change data capture feature.
- Create an Amazon RDS for PostgreSQL database instance, define a table, and load the first dataset.
- Create an Amazon S3 bucket to serve as the target.
- Create an AWS Lambda function to analyze S3 objects upon creation.
- Configure and invoke the AWS Database Migration Service to migrate the initial dataset that we just loaded into RDS to the S3 bucket. Doing this demonstrates the initial load phase of DMS and shows AWS Lambda processing the dataset.
- Copy the second dataset into the database. Doing this demonstrates the change data capture feature of DMS and shows AWS Lambda processing the dataset.
- Examine the contents of the Amazon S3 bucket to see the results of both the initial migration and the change data capture.
- View the results of the analysis in Amazon CloudWatch.
Prerequisites and assumptions
You will need the following:
- An AWS account that provides access to the services shown in the diagram.
- Working knowledge of these services.
- A utility that connects to a PostgreSQL database, such aspsql.
Don’t worry if you don’t have these prerequisites, VaporVM can assist you in acquiring these tiny details.
- VaporVM will configure all services in the same VPC and region to simplify networking considerations.
- The VPC must have an S3 endpoint.
VaporVM has an Advanced tier partnership with AWS and our certified team has experience in managing AWS environments for various companies globally especially when it comes to migrating data. Providing tech support 24x7x365, we make sure that your infrastructure is constantly monitored and possible glitches are removed even before they occur.
Consequently, VaporVM improves efficiency of your existing AWS infrastructure & cuts costs by up to 30%, consequently improving the ROI on your AWS investments. Whether you are just getting started on AWS or are looking for AWS management and support for your existing AWS cloud, our team of top-class system administrators, cloud engineers, and network technicians customise solutions to the goals and specific requirements of your company, while meeting enterprise-grade performance, reliability, and security standards.