Cloudera → Apache Kafka OSS Migration

Architecture Diagram

App Services_Producers

Microservices_Producers

IoT / Streams_Producers

CDC / DBs_Debezium

Data Layer

Cloudera CDP / HDP

Kafka Brokers_{CM Managed}

ZooKeeper_Quorum

Schema Registry_{Cloudera SR}

Kafka Connect_{CM Connectors}

KSQL / KSQLdb_{CM Managed}

Ranger / Sentry_AuthZ

Atlas_{Lineage / Meta}

Migration Bridge

MirrorMaker 2_{Cross-cluster repl.}

Dual-Write Proxy_{App interceptor}

Consumer Lag_{Monitor / Compare}

Offset Sync_{MM2 Checkpoints}

Apache Kafka OSS

Kafka Brokers_{KRaft Mode}

KRaft Controllers_{No ZooKeeper}

Confluent / Apicurio SR_{Compat. SR}

Kafka Connect_{OSS Connectors}

ksqlDB OSS_Self-managed

OPA / ACLs_AuthZ

OpenMetadata_{Lineage / Meta}

⇄ Dual-Write Window · Consumer Shadowing · Offset Tracking ⇄

Stream Processors_{Flink / Spark}

Analytics / DWH_Consumers

Data Lake Sinks_{S3 / HDFS}

App Services_Consumers

Prometheus + Grafana_Metrics

ELK / Loki_Logs

Cruise Control_Rebalancing

Kafka UI / AKHQ_Management

Terraform / Helm_IaC

Vault / KMS_Secrets

Migration Phases

Phase 1 · Weeks 1–2

Discovery & Baseline

Inventory all topics, partitions, replication factors
Audit consumer groups and current offsets
Capture producer/consumer throughput and SLAs
Document all Schema Registry schemas (Avro/Protobuf/JSON)
Map Cloudera Ranger policies → Kafka ACLs
Identify Kafka Connect connectors and custom SMTs
Profile retention, compaction, and tiered storage needs
Assess KSQL queries and streaming jobs

Phase 2 · Weeks 3-4

Target Cluster Build

Provision OSS Kafka brokers in KRaft mode (3+ controllers)
Apply identical partition counts and replication factors
Deploy Confluent or Apicurio Schema Registry
Migrate all schemas; verify compatibility modes
Re-create ACLs (SASL/TLS + Ranger-equivalent ACLs)
Configure Prometheus JMX exporters + Grafana dashboards
Set up Cruise Control for auto-rebalancing
Deploy Terraform/Helm IaC for reproducibility

Phase 3 · Weeks 5-6

Data Replication & Shadowing

Deploy MirrorMaker 2 for all topic replication
Configure MM2 offset sync and heartbeat topics
Enable consumer group shadow mode on OSS cluster
Run parallel consumers; compare message parity
Validate end-to-end latency and throughput parity
Test failover/failback with MM2 checkpoints
Validate all Connect connectors on OSS cluster
Re-implement KSQL queries in ksqlDB OSS

Phase 4 · Weeks 7-8

Cutover Execution

Migrate consumers cluster-by-cluster (start non-critical)
Use dual-write interceptor for producers during transition
Translate consumer group offsets via MM2 checkpoints
Monitor consumer lag on both clusters in real time
Migrate producers once consumers are fully validated
Disable MirrorMaker 2 after full cutover confirmed
Decommission Cloudera-specific components (ZooKeeper, Ranger)
Update DNS, service discovery, and client configs

Phase 5 · Weeks 9-11

Stabilization & Decommission

Run 2-week post-cutover monitoring period
Tune broker configs (buffer sizes, fetch limits, GC)
Validate Cruise Control auto-rebalancing in prod
Update runbooks and operational documentation
Train ops team on OSS tooling (AKHQ, Kafka UI)
Decommission Cloudera cluster and license revocation
Archive Atlas metadata and export to OpenMetadata
Complete security audit of new ACL policies

Component Mapping

Cloudera Component	OSS Replacement	Migration Notes	Effort
ZooKeeper	KRaft (built-in)	Deploy fresh in KRaft mode. ZK migration path available via `kafka-storage` tool for existing clusters.	Low
Cloudera Manager	Terraform + Helm + AKHQ	Replace GUI-based management with IaC. AKHQ/Kafka UI for topic & consumer group UI. Conduct ops training.	Medium
Cloudera Schema Registry	Confluent SR or Apicurio	Export all schemas via SR REST API. Import into target SR. Validate subject naming strategy and compatibility settings.	Medium
Kafka Connect (CM)	Kafka Connect OSS	Export connector JSON configs. Re-deploy on OSS workers. Validate custom SMTs and converters are available on classpath.	Medium
KSQL (CM)	ksqlDB OSS	Export KSQL DDL statements. Re-run on ksqlDB OSS. Validate persistent queries and UDFs. API is wire-compatible.	Medium
Ranger / Sentry	Kafka ACLs + OPA	Script Ranger policy export to native Kafka ACLs via `kafka-acls.sh`. Consider OPA for fine-grained RBAC.	Medium
Atlas (Metadata)	OpenMetadata / DataHub	Export Atlas entities via REST. Ingest via OpenMetadata Kafka connector. Map lineage graphs manually for complex flows.	High
Cloudera Replication	MirrorMaker 2	Deploy MM2 as standalone or as Connect cluster. Configure active-passive replication per topic. Monitor with JMX metrics.	Low
CM Metrics / Alerts	Prometheus + Grafana	Deploy JMX Exporter sidecar per broker. Import community Kafka Grafana dashboards. Recreate all CM alerts in Alertmanager.	Low

Risk Register

High Risk

Consumer Offset Desync

Translated offsets from MM2 checkpoints may lag, causing message duplication or skipped records during cutover.

▸ Use MM2 __consumer_offsets sync. Validate lag on both clusters before consumer migration. Run in shadow mode for ≥72 hrs.

Schema Compatibility Breaks

High Risk

Schema Registry subject naming strategies may differ between Cloudera SR and Confluent/Apicurio SR.

▸ Export schemas early. Validate naming strategy in target SR. Run producers against both SRs in parallel before cutover.

Medium Risk

ACL Policy Gaps

Ranger RBAC policies don't map 1:1 to native Kafka ACLs; granularity differences may cause auth failures.

▸ Audit all Ranger policies. Script conversion. Test ACLs with canary clients before migrating production traffic.

Medium Risk

Custom SMT / Connector Deps

Proprietary Cloudera Connect plugins or custom transformations may not exist in OSS ecosystem.

▸ Inventory all connectors in Phase 1. Re-implement custom SMTs in OSS. Test in staging with full data volume.

Medium Risk

Performance Regression

OSS Kafka tuning defaults differ from Cloudera-optimized configs, potentially degrading throughput temporarily.

▸ Benchmark target cluster at 120% of peak production load. Tune num.io.threads, socket.send.buffer.bytes, batch sizes before go-live.

Low Risk

KRaft Operational Unfamiliarity

Ops team may be unfamiliar with KRaft mode controller management and metadata quorum operations.

▸ Run KRaft training sessions. Use kafka-metadata-quorum.sh tooling. Document all operational runbooks before go-live.

Go-Live Checklist

Pre-Cutover Validation

All topics replicated with zero lag on MM2

Consumer group offsets translated and verified

Schema Registry parity confirmed (schema IDs match)

All Kafka Connect connectors healthy on target

ksqlDB persistent queries verified in staging

Load test at 120% peak throughput passed

Security & Access

All ACLs re-created and tested with canary clients

TLS certificates deployed and rotated

SASL mechanisms configured (SCRAM-SHA-512 / OAUTHBEARER)

Secrets rotated in Vault / KMS

Network security groups / firewall rules updated

Observability

Prometheus scraping all brokers and ZooKeeper

Grafana dashboards showing broker, topic, consumer metrics

Alertmanager rules for consumer lag and under-replicated partitions

Cruise Control live and monitoring cluster balance

Log aggregation shipping to ELK / Loki

Rollback Readiness

Cloudera cluster left running until D+14

MM2 can reverse-replicate within 15 minutes

Rollback runbook documented and rehearsed

Client bootstrap servers switchable via config map / env var

On-call rotation established for 2-week hypercare period