Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events

Multi-year architectural evolution: Moving from Amazon Kinesis to Managed Streaming for Apache Kafka (MSK), using Kafka Tiered Storage to cut costs, the critical outages that tested our resilience.

Scaling StreamHub: From Kinesis to Kafka and Beyond

At Atlassian, event streaming powers our entire product suite. Most of the clicks, impressions, API calls, CDC events, and telemetry flows through our real time data pipelines.

Not so long ago, our streaming infrastructure handled a respectable 22 billion events per day. As our user base grew and new real time analytics features launched, data volume surged. Today, StreamHub ingests and processes 150 billion events per day, delivers over 225 billion events per day — averaging about 1.68 million events per second with peak traffic exceeding 3.2 million events per second.

Operating at this scale exposes every micro inefficiency in your code, every limitation in cloud providers’ offerings, and every subtle defect in managed services.

In this post, we share our multi year architectural evolution: why and how we moved from Amazon Kinesis to Apache Kafka (via AWS Managed Streaming for Apache Kafka, or MSK), how we used Kafka Tiered Storage to cut costs, the critical outages that tested our resilience, and the design patterns that keep StreamHub robust.

The Starting Line: Why We Outgrew Amazon Kinesis

For years, Amazon Kinesis powered our data ingestion layer. This zero ops service served us well up to 20 30 billion events.

Facing a 5x scale increase, Kinesis revealed three fundamental bottlenecks:

	Bottleneck	Description
1	API reliability	StreamHub API barely supported 99.99% reliability, and Kinesis throttling during spikes affected producers. With increased scale, we aimed for 99.999% reliability.
2	The Cost of High Partitioning	Kinesis scales via “shards,” each with a throughput limit of 1 MB/sec ingress and 2 MB/sec egress. To handle peak loads, we needed thousands of shards. Since Kinesis pricing depends heavily on active shards, our monthly cloud bill grew linearly with traffic. Kinesis autoscaling (on demand) at StreamHub’s scale was prohibitively expensive.
3	Retention Limits & Cost Penalties	Storing data in Kinesis beyond 24 hours required costly extended retention add ons. We aimed to keep event history for up to 7 days to let consumers replay data safely during downstream failures or model retraining.
4	Delivery SLOs & Consumer Throughput Exhaustion	With Kinesis, multiple independent consumer groups share the same shard egress capacity, and Enhanced Fan Out (EFO) carries a steep premium. StreamHub required a local delivery SLO under 2 seconds and cross region delivery latency under 3 seconds. To meet these latency guarantees for two high volume, low latency services, we used EFO to fan out and push events directly to our consumers’ external streaming infrastructures or across regions.

The Decision: Self Managed vs. Managed Apache Kafka

We needed an architecture that supported cost effective multi day retention, scaled without significant cost for additional consumer groups, and could support multi cloud. Apache Kafka was the clear choice.

To reduce operational overhead, we migrated to AWS MSK (Managed Streaming for Apache Kafka). This allowed our small platform team to focus on stream architecture, client libraries, and reliability rather than provisioning zookeepers, managing OS patches, or replacing failed hardware.

Decoupling Storage and Compute: The Magic of Tiered Storage

Our migration to Kafka was more than an API change; it fundamentally altered how we stored and accessed stream data.

Storing billions of events on high performance EBS (Elastic Block Store) volumes would be costly. This made Kafka Tiered Storage our key architectural feature.

Tiered Storage splits the storage of Kafka topic data into two tiers:

Tier	Description	Details
Local Tier (Tier 1)	Utilizes local EBS volumes on the brokers to store “hot” data.	Configured for the last 5 minutes. Ensures real time consumers read directly from fast, low latency storage. Consumers reading beyond this window are not latency sensitive by definition.
Remote Tier (Tier 2)	Asynchronously offloads closed log segments to a highly durable, low cost object store (Amazon S3).	Stores historical data with a 7 day retention configured for our target. Reading from S3 adds a few hundred milliseconds of latency, which is acceptable if the consumer is already over 5 minutes behind.

By moving to Tiered Storage, we reduced infrastructure costs significantly. We stopped over provisioning costly EBS volumes for worst case retention. Historical data reads now come entirely from S3, preventing heavy backfills from starving IOPS for our hot, real time consumers.

Managed Streaming Challenges: What Broke at Scale

Operating at 150 billion events per day means every architectural assumption gets tested. Managed streaming gave us the operational simplicity of a managed Kafka service, but at our scale we also encountered limits that were easy to underestimate: broker networking ceilings, S3 request rate behavior, tiered storage offload throughput, and managed service control plane dependencies.

The following challenges shaped the next phase of StreamHub’s resilience.

Challenge 1: Broker network limits and instance sizing

When teams size Kafka clusters, it is natural to start with CPU and memory. For us, the more important constraint was often broker level throughput. Each broker instance type has practical ingress, egress, EBS, and network limits. A cluster can look healthy on average while a subset of brokers or partitions is already running close to saturation.

Kafka traffic is also amplified. Producer ingress is only one part of the picture; replication, consumer fan out, cross region relay traffic, retries, and tiered storage remote copy all compete for broker resources. A broker shape that is sufficient for steady state writes may still be under provisioned once these secondary workloads are included.

This became especially visible during traffic spikes. Smaller broker instances could accept incoming data faster than they could replicate, serve consumers, and offload closed log segments to S3. As CPU and I/O pressure increased, background Kafka work slowed down, consumer lag grew, and local disk utilization began climbing.

Challenge 2: Tiered storage still needs local headroom

Tiered Storage dramatically reduced our storage cost, but it did not eliminate the need for strong local broker capacity. Kafka writes data locally first and background workers copy closed log segments to S3 async.

During heavy inbound traffic, those background workers compete with producer writes, replication, reads, and broker housekeeping. If the instance is too small, remote copy can fall behind. Once remote copy falls behind, local disks begin to accumulate data that should have been offloaded. At high ingest rates, that disk growth can become linear from an operations perspective: every minute of delay adds more data that the cluster must later catch up on.

The key lesson was simple: tiered storage is not a substitute for broker headroom. It is a cost and retention strategy, not a license to run minimal local capacity.

Challenge 3: S3 request rate limits during retention changes

S3 is effectively unlimited in capacity, but its APIs still have request rate behavior that matters at scale. With thousands of Kafka partitions, Tiered Storage continuously writes segment files, index files, and metadata to S3.

We encountered a painful edge case when temporarily increasing tiered storage retention from 7 days to 21 days for a production investigation and then reverting to 7 days. The retention reduction triggered a large wave of deletes for older remote data. That delete storm competed with regular tiered storage writes and caused remote writes to slow down. As remote writes paused, broker local disk utilization increased quickly.

The issue was not that S3 lacked capacity; it was that a large operational change created a burst of control plane and data plane API activity that interfered with the steady remote copy path.

Challenge 4: Storage scaling cooldowns during incidents

When broker disks fill up, the natural response is to add disk. Storage modification is a managed operation and can introduce a cooldown window before another storage change is allowed.

That matters during an incident. If disks are filling quickly and the first expansion is too small, the cluster may need another expansion before the cooldown expires. In our case, this extended recovery time. We had to wait for the managed service operation window to clear before adding enough storage to recover safely.

The lesson was to avoid small, incremental storage increases during a crisis. If storage must be expanded under active pressure, the safer move is to add enough headroom to survive the full cooldown window and the expected backlog.

Challenge 5: Availability Zone failure and managed control plane behavior

All of our clusters are deployed across multiple Availability Zones so that the Kafka data plane can continue operating when a single zone has problems. During a localized AZ degradation, we expected clients to shift toward healthy brokers and for the cluster to degrade gracefully.

The data plane was not the only dependency, instead the control plane reported inaccurate status, degrading all our services. Due to the zone outage, the cluster was healing and blocked any changes. That meant we could not quickly reboot broker, scale the cluster, increase disk, or complete some automated reassignment actions at the exact moment we needed those controls most.

This changed how we think about managed services. A managed service can reduce day to day operational burden, but critical recovery workflows still need to account for control plane unavailability.

Challenge 6: Scaling is slow, and storage only grows

MSK scaling operations are not instantaneous, and they are not symmetric. Increasing broker storage gives the cluster more breathing room, but that disk allocation cannot be reduced later. Once storage has been expanded, the only practical way to return to a smaller disk footprint is to create a new cluster with the desired size and re route producers and consumers to it.

Broker instance upgrades also require a rolling restart. In practice, this can take roughly 15 minutes per broker, so rolling instance upgrades across a large cluster can take hours, making them unsuitable as incident mitigation. That is useful for planned scaling, but it is rarely fast enough to be the primary mitigation during an active incident.

Scaling out by adding brokers has a similar limitation. New brokers increase theoretical cluster capacity, but they do not immediately help existing hot topics. Until partitions and leaders are rebalanced onto the new brokers, the overloaded brokers continue carrying the same traffic. Because rebalancing adds additional network, disk, and replication load, scaling out during an incident was not a realistic recovery option for our most severe failure modes.

Solutions: How We Hardened StreamHub

After these incidents, we changed both our architecture and our operating model. The goal was not only to make Kafka faster; it was to make failure modes smaller, recovery paths clearer, and capacity boundaries explicit.

Solution 1: Capacity planning around broker level bottlenecks

We now plan capacity using broker level peak ingress, egress, EBS throughput, CPU, partition placement, and remote copy behavior not only aggregate cluster throughput. The cluster average is useful for trend analysis, but incidents usually begin when a few brokers become hot.

In practice, this means:

• Choosing broker instances for network and EBS headroom, not just CPU utilization.
• Keeping sustained CPU below a conservative threshold so background Kafka and tiered storage work can continue during spikes.
• Monitoring per broker BytesIn, BytesOut, disk growth, remote copy lag, partition skew, and consumer lag together.
• Load testing with realistic producer, consumer, replication, retry, and cross region traffic patterns.

One of the most important changes was cultural: we stopped treating unused broker capacity as waste. For a system this critical, headroom is a reliability feature.

Solution 2: Over provisioning network limits intentionally

Over provisioning does not mean guessing high. It means treating documented and observed per broker limits as hard boundaries, then designing the cluster so normal peak traffic uses only a safe fraction of those limits.

Our approach includes four layers:

1. Use larger broker shapes where network is the bottleneck. If CPU looks low but network or EBS is close to saturation, the instance is still too small.
2. Scale out in Availability Zone multiples. Adding brokers increases aggregate network capacity, but it only helps existing hot traffic after partitions and leaders are rebalanced.
3. Balance partitions and leaders continuously. Hot partitions can break a cluster before total capacity is exhausted. We use round robin partition assignment, but keyed partitions require frequent rebalancing.
4. Protect the shared broker budget with quotas. Producer and consumer quotas prevent one workload from consuming all available network bandwidth.

This strategy gives us room to absorb retries, failover traffic, and temporary consumer slowdowns without immediately pushing brokers into resource exhaustion.

Solution 3: Safer tiered storage operations

Tiered Storage remains a major win for StreamHub, but we now treat retention and segment management changes as production risk operations.

We reduced risk by generating fewer, larger segment files where appropriate, staggering operational changes, and watching remote copy metrics closely during any retention adjustment. We size local disks with enough emergency buffer to survive temporary S3 offload delays and configure sufficient tiered storage offload threads to handle peak traffic spikes.

The principle is that remote storage should reduce steady state cost, while local storage should provide operational shock absorption.

Solution 4: Ingress rate limiting and quarantine

We built controls on StreamHub API and the delivery layer to protect Kafka from sudden spikes, malformed traffic, and external exceptions.

The two important controls:

• Rate limiting: We can enforce dynamic limits by tenant, topic, producer, or traffic class before events reach Kafka.
• Quarantine: Traffic that is malformed or repeatedly failing can be diverted to a safe holding path instead of being retried indefinitely through the hot path.

This moved a critical protection boundary closer to producers. Instead of asking Kafka to absorb every spike, we can shed, slow, or quarantine traffic before it threatens shared broker capacity.

Solution 5: Kafka client quotas for noisy neighbors

Inside Kafka, we use client quotas to enforce producer and consumer bandwidth budgets. This protects critical streams from lower priority workloads and prevents a single misconfigured pipeline from monopolizing broker resources.

# Example: enforce producer and consumer bandwidth for a client identity kafka configs.sh bootstrap server $BOOTSTRAP alter \ entity type users entity name analytics consumer group \ add config 'producer_byte_rate=52428800,consumer_byte_rate=104857600'

Quotas are not only a cost control mechanism. At StreamHub scale, they are a blast radius control.

Solution 6: Sharding and Failover clusters for faster failover

We split large clusters into multiple shards to reduce blast radius. During an incident, we quickly route traffic to a healthy cluster.

When a managed control plane is unhealthy, waiting for recovery can extend outages. We created a “failover cluster” runbook: if a primary cluster shows severe sustained degradation, we provision a clean parallel Kafka cluster while continuing incident response.

The runbook has four stages:

Stage	What happens
Shard Fail over	Update routing from affected shard to healthy shard. Adjust rate limits and add quarantine to maintain processing within target shard capacity. (ETA: ~15 mins)
Prepare	Provision a new cluster and apply our infrastructure as code templates for topics, tiered storage, access controls, and client configuration. (ETA: ~2 hours)
Evaluate	Continue mitigating the primary cluster while the clean environment becomes ready in parallel.
Cluster Fail over	If the primary cluster remains unhealthy beyond our decision threshold, update routing so producers and consumers move to the clean cluster.

This gives on call an escape hatch. Even if the primary cluster is stuck behind a managed service operation, recovery no longer depends entirely on that control plane becoming healthy.

Solution 7: Companion regions for compliant regional failover

StreamHub operates globally, and failover must respect data residency requirements. We cannot simply move traffic from one geography to another if that violates customer or regulatory commitments.

To solve this, we established companion regions: pre approved regional zones that stay within the same compliance boundary. Producers and consumers can be configured to fail over to the companion region when the primary region becomes unhealthy, while preserving residency constraints.

This turns disaster recovery from an improvised decision into a pre vetted path.

Lessons for Teams Adopting Managed Streaming at Scale

Our journey reinforced several lessons that are broadly applicable to high throughput streaming platforms:

• Managed does not mean unlimited. Managed streaming services remove a lot of operational work, but broker, network, EBS, S3, and control plane limits still matter.
• Plan for the hottest broker, not the average broker. Partition skew and hot leaders are often where incidents begin.
• Tiered storage reduces cost, not operational responsibility. Remote copy needs CPU, EBS, network, and local disk headroom.
• Do not make small emergency storage increases. If a managed service cooldown applies, add enough capacity to survive the whole window.
• Rate limits and quotas are reliability tools. They protect shared infrastructure and reduce blast radius.
• Have a failover path that does not depend on the failing control plane. Failover clusters and companion regions give operators options when managed operations are unavailable.

Looking Forward

Scaling StreamHub from 22 billion to 150 billion events per day taught us that streaming reliability is built in layers. Kafka gave us the foundation we needed for throughput, retention, and cost efficiency. But the system became truly resilient only after we added traffic control, capacity guardrails, operational playbooks, and compliant failover paths.

The biggest lesson is that reliability at this scale is not one feature or one service. It is the combination of conservative capacity planning, explicit blast radius controls, tested recovery paths, and a willingness to learn from every failure.

We are continuing to evolve StreamHub as Atlassian’s event volumes grow, and we are excited to share more about the next phase of that journey.

The post Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events appeared first on Inside Atlassian.