site-logo

JAWS PANKRATION 2024

site-logo
HomeNewsTimetableCfPCommitteePromotionFollow UpPrivacy Policy

Hurdles in Implementing CDC with Aurora during Operation

Lv400

Lv400

8/24/2024 13:40 (UTC)
Session Info

In this session, I'm going to discuss the hurdles in implementing a CDC (Change Data Capture) solution with Aurora during the operational phase on a specific system.

For instance, I will talk about the compatibility of CDC and Aurora in scenarios such as Aurora blue-green deployment or Aurora failover.

　

The example architecture I will discuss is implemented with Aurora and Datastream, which is the CDC solution provided by Google Cloud.

However, the main focus of this session will be the specifications of the logical replication of Aurora, and I will also examine similar issues in CDC using AWS DMS and Aurora.

Kazuki Maeda

Kazuki Maeda

- AWS Community Builders -

- AWS User Community Leaders -

Session Category
Database
AWS Services
Amazon Aurora (MySQL, PostgreSQL)
AWS Database Migration Service
Session Materials
Session Archive
Session Summary (by Amazon Bedrock)
    The speaker, a community builder and SRE/engineering manager in Japan, discusses system operations and data infrastructure using CDC (Change Data Capture). The main topics are: 1. Designing product and data infrastructure considering both architecture and organization. 2. Separating responsibilities and managing dependencies. The presentation covers: 1. Introduction to product and data infrastructure, organizational structures, and architecture examples. 2. Data transfer methods: batch and streaming. 3. Batch transfer types: snapshot and query-based. 4. Streaming transfer using CDC technology. The speaker explains the challenges of integrating CDC-based data infrastructure with product systems, highlighting: 1. Manual table maintenance 2. Unexpected failovers 3. Database upgrades A detailed example of upgrading an Aurora MySQL database from 5.7 to 8.0 using blue-green deployment is provided, illustrating potential issues with CDC and data synchronization. To address these challenges, the speaker emphasizes: 1. Team collaboration 2. Clear separation of responsibilities 3. Proper management of dependencies Different organizational patterns are discussed, from small teams managing both product and data infrastructure to larger organizations with separate teams for each. The presentation concludes by stressing the importance of: 1. Choosing solutions that fit product characteristics and requirements 2. Properly designing and understanding organizational structure and team responsibilities 3. Maintaining clear dependencies between product and data infrastructure The speaker emphasizes that while there's no one-size-fits-all solution, understanding these concepts is crucial for effective system operations and data management.

Back

©JAWS-UG (AWS User Group - Japan). All rights reserved.