Organizations generate massive amounts of data across globally distributed services. Atlassian, known for products like Jira and Confluence, recently faced a data challenge: how to give customers powerful analytics and data export capabilities without straining day-to-day operations. The solution was to build a unified a centralized, analytics-optimized repository called customer data lake that could scale with millions of users while respecting regional data regulations. In this blog post, we break down Atlassian’s journey in building a multi-region compliant customer data lake and highlight key lessons for any enterprise tackling similar challenges.

Why Atlassian Needed a Customer Data Lake

Atlassian’s existing cloud architecture stored each product’s information in separate transactional databases, which made comprehensive analytics difficult. Several critical pain points underscored the need for a unified data lake:

Instant analytics expectations: Customers wanted to generate reports and insights from Jira, Confluence, and other products immediately. Without a consolidated analytics store, Atlassian could not provide “instant insights” users lacked a seamless way to analyze data across products in real time.
Data export limitations: In on-premises setups, administrators could query databases directly or dump data into their own warehouses. In Atlassian’s cloud, direct database access isn’t possible, leaving migrating customers without a simple way to export their data. This gap became a blocker for some enterprises moving to the cloud.
Operational databases under strain: Built-in reports and dashboards in Jira were querying production transactional databases. These heavy analytical queries weren’t what those systems were optimized for, causing performance issues. Complex dashboards could slow down the core systems, since the databases were designed for quick transactions rather than large aggregations.

Recognizing these challenges, Atlassian concluded that a dedicated customer data lake was the right path. Unlike transactional databases, a data lake can store massive tables optimized for complex queries and historical analysis. By funneling all product data into this analytics-focused repository, Atlassian aimed to deliver new value and offload queries from operational systems.

Choosing a Logical Replication Strategy

The next hurdle was figuring out how to capture data changes from dozens of cloud services and hundreds of databases in real time. Atlassian’s ecosystem is diverse and it spans old monoliths and modern microservices using PostgreSQL, DynamoDB, MongoDB, and more. They needed a replication method that worked across all these technologies and allowed many autonomous teams to feed data into the lake without centralized bottlenecks. This requirement aligned with data mesh principles, where each product team treats its data as a product and shares it in a standardized way.

Atlassian considered two broad approaches for change data capture (CDC):

Logical replication (application-level CDC): Instrument the application code to emit an event whenever data changes. This approach is agnostic to the underlying database technology. Each service owns its data contract and publishes changes to a shared bus in a consistent format. Logical replication promised stable schemas – since producers define the event structure, they can evolve their database internally without breaking downstream systems, as long as they maintain the contract. The trade-off was the upfront effort: every product team had to add and maintain code to produce these events.
Physical replication (database-level CDC): Tap directly into database transaction logs using tools that automatically capture all inserts, updates, and deletes. This method can cover an entire database quickly with minimal initial coding. However, it depends heavily on database-specific details. With Atlassian’s mix of technologies, a one-size-fits-all CDC tool wasn’t practical. Physical replication also tends to be brittle when schemas change or when teams swap out their storage layer, since the capture is tightly coupled to the internal data structure.

After weighing these options, Atlassian chose the logical replication route despite the greater upfront work. This decision ensured technology independence and long-term flexibility. Each product team became responsible for emitting data change events in a standard format, treating those events as a first-class product. In return, the central data lake pipeline could ingest changes from any system as long as it adhered to the agreed event protocol. It was an investment in a robust foundation that would scale and adapt over time.

Diagram of bootstrap and real-time replication paths from Atlassian product databases to AWS Kinesis and the data lake pipeline.

Implementing a Real-Time Replication Pipeline

To turn logical replication into reality, Atlassian built a Replication Bus a set of APIs and infrastructure that acts as the central highway for data changes. Here’s how it works:

Application-Level Events: Whenever a record is created, updated, or deleted in any Atlassian cloud product, the application emits a structured event describing that change. For example, if a new issue is created in Jira or a page is edited in Confluence, an event is produced capturing the key details. All teams adhere to a unified schema for these events to ensure consistency across products.
Replication Bus Intake: A central service receives the incoming events and immediately publishes them to an AWS Kinesis streaming channel dedicated to that domain or product. This stream-of-events design allows downstream consumers to tap in and consume changes in real time without affecting the source applications.
Bootstrapping via Bulk Export: When initially populating the data lake for a product or migrating a very large customer dataset, streaming every single historical record through Kinesis could be inefficient. Atlassian handles these cases by doing one-time bulk exports of the current database state to Amazon S3. The data lake pipeline can load this snapshot from S3 as a starting point, then switch to consuming incremental events from the stream. This two-pronged approach avoids overloading the real-time channels during big migrations.

Central to this pipeline is a standard event schema using Protocol Buffers. Every change event follows a common envelope format that includes metadata like the entity type, a unique entity ID, the operation type, and a version number. The version is especially important, Atlassian designed the system with a “last write wins” strategy. Each new change increases the version, so if events ever arrive out of order, consumers can use the highest version and safely ignore older updates. This guarantees that the data lake eventually reflects the latest state of each item without requiring perfect ordering of events.

The event payload is also embedded in the Protobuf message and can be different for each product. By using a flexible schema type, each team defines the structure that makes sense for its data while conforming to the overall envelope. If an event’s payload is very large, the producer will offload that content to S3 and include only a reference link in the event. The consuming pipeline detects this and retrieves the full data from S3 as needed. This mechanism keeps the real-time streams flowing smoothly by handling large blobs outside of Kinesis.

With this replication bus in place Atlassian had the raw ingredients for the data lake. The next step was to transform those raw events into queryable tables without delay.

Streaming Data Pipeline to the Analytics Lake

Atlassian implemented a continuous data pipeline to process incoming events and populate the Atlassian Data Lake in near real time. They used Apache Flink to build a streaming ETL (Extract, Transform, Load) job. The pipeline reads events from the Kinesis streams, applies transformation logic, and writes the results out to Delta Lake tables stored in Amazon S3. Delta Lake was chosen because it supports ACID transactions and efficient upserts, which are crucial for maintaining accurate, up-to-date analytics tables.

The data lake itself is organized into two layers of tables:

Raw tables (immutable logs): For every entity type – such as a Jira issue or a Confluence page – there is a raw table that acts as an append-only log of all events received for that entity. Every change event is recorded as a new row in the raw table, capturing an audit trail of how an item evolved over time. These raw tables aren’t exposed to end users; they serve internal purposes like debugging, reprocessing, or recovering from errors. If Atlassian ever needs to rederive a customer’s data or backfill something, they can replay the sequence of raw events.
Refined tables (analytics models): These are the polished, customer-facing tables optimized for querying and analysis. A refined table represents the latest state of a certain data object. The streaming pipeline transforms raw events into these refined records by applying business logic and data modeling rules. In many cases, this means filtering or renaming fields, computing derived metrics, and joining in relevant context so that the data is intuitive and ready for analysis.

Every time a new event comes through, the Flink job first appends it to the appropriate raw table and then determines which refined table(s) need updating. Atlassian developed a set of transformation rules corresponding to each event type. When, for example, an “Issue Updated” event arrives, the pipeline knows how to map that to the Jira Issues refined table and any related aggregate tables. It then performs an upsert on those Delta tables, updating the existing row for that issue with the new data, or inserting a new row if it’s a new record. Delta Lake’s merge capabilities make these upsert operations efficient by handling them in batches. The result is that each refined table always reflects the current state of the data as changes stream in, rather than storing every change multiple times.

To maintain consistency between what events produce and how they are transformed, Atlassian keeps the Protobuf schemas and transformation logic in sync. Both the event definitions and the corresponding SQL for transforming them live in a shared configuration repository. Whenever a product team wants to add or change a field in the data they send, they collaborate with the data lake team to update the transformation rules in tandem. This co-development process ensures that when new data fields start flowing, the refined tables are ready to handle them without breaking queries.

Finally, to ensure each customer only sees their own data, Atlassian implemented row-level security by design. All data in the lake is partitioned or tagged by customer organization. When customers access Atlassian Analytics, they are querying views that automatically filter to their organization’s slice of the data. This way, a single multi-tenant data lake can securely serve many customers, and each one experiences it as if it were their exclusive database.

Atlassian customer data lake pipeline architecture using Protobuf, dbt, Flink jobs, Delta Lake, and Databricks for refined analytics.

Enhancing Analytics with Pre-Joined Data

The real-time streaming pipeline keeps each data entity up to date in its own table, but many business questions require combining data from multiple sources. For example, reporting on Jira issues alongside user details, or linking Jira and Confluence data in one view. Atlassian found that performing these joins on the fly for every query could become expensive, especially if numerous customers were independently running the same complex joins. To tackle this, they introduced Enhanced Views and Tables as a supplementary approach for complex analytics:

Enhanced Views: These are predefined SQL views that join two or more refined tables dynamically. They don’t store any data themselves; instead, when a user runs a query on an enhanced view, it pulls the latest data from the underlying tables and combines it on demand. Views are a lightweight way to simplify analytics for common questions. As long as the ad-hoc join performance is acceptable, a view can be used to transparently handle the relationship.
Enhanced Tables: If a particular join or aggregation is too slow to compute on the fly, and many users need it regularly, Atlassian creates a materialized table that holds the result of that join. These enhanced tables are populated by periodic batch jobs which run the expensive query in the background and store the results as a ready-to-query table. Essentially, it’s a cached dataset for heavy analytic workflows. Using enhanced tables can drastically speed up customer queries at the cost of a slight lag in how up-to-date the data is.

By using a mix of real-time streaming for most data and selective batch processing for complex derived data, Atlassian ensures customers get a responsive analytics experience. The frequently accessed simple data is always real-time, while the complicated cross-product metrics are available through these pre-computed tables or views, keeping query times fast and predictable.

Multi-Region Architecture for Data Residency

From the outset, Atlassian designed the customer data lake to operate across multiple regions worldwide. Many enterprise customers have strict data residency requirements or preferences for example, European organizations might require that all their data stay within EU data centers. Beyond compliance, multi-region data deployments also offer performance and reliability benefits. Key advantages of Atlassian’s multi-region data lake include:

Customer choice and compliance: Customers can choose where their analytics data resides to meet legal requirements and internal policies. If a company needs all data stored in Europe, Atlassian can maintain that customer’s data partitions solely in an EU-region data center.
Lower latency: By hosting data and running analytics queries in the region closest to the customer, query responses are faster. Users get quicker dashboards and reports because the data doesn’t need to travel across the world for processing.
Scalability through sharding: Instead of one giant global dataset for, say, all Jira issues, Atlassian splits the data by region. Each region’s stack handles a subset of customers. This means each analytics job works on a smaller chunk of data, improving throughput and manageability. Essentially, each region acts as a shard of the whole data lake, which makes the system more scalable as it grows.
Fault isolation: Regional independence acts as a safety net. If one region experiences an outage or a critical issue, it only affects the customers in that region. Users elsewhere continue unaffected. This containment significantly reduces the blast radius of incidents and contributes to overall reliability.

Technically, Atlassian achieved multi-region support by leveraging infrastructure-as-code and automation. The entire data lake platform is defined in reusable templates. To deploy a new region, the team doesn’t rewrite the pipeline; they simply instantiate a new set of infrastructure with region-specific parameters. The same Flink jobs and transformation code run in each region, pointing to local resources. This approach allowed Atlassian to bring up additional regions with minimal effort, ensuring consistent behavior everywhere.

Multi-region architecture diagram of Atlassian Data Lake with AWS Kinesis streams and Databricks endpoints for Atlassian Analytics.

Migrating Data Between Regions

Operating in multiple regions also meant Atlassian needed a way to move customer data if a client chose a different residency or if data had to be purged entirely. For example, a European customer might later request to have their site and data migrated to the United States, or vice versa. Additionally, regulations like GDPR require that when a customer leaves the service, all their personal data can be deleted on request. Atlassian addressed these scenarios with a migration and deletion workflow built on top of the data lake.

When a customer’s site needs to move from Region A to Region B, Atlassian essentially replays that customer’s event history into the new region’s pipeline. The steps are roughly as follows:

Migration trigger: Atlassian’s control plane signals that a specific customer should be transferred to a target region. This provides the context for source and destination.
Export from source: In the original region, a migration job reads all events from the raw tables that belong to the given customer. Because the raw tables contain every change event, the job can collect the entire history of that customer’s data.
Stream to destination: Instead of transforming these events in Region A, the system feeds them into the Kinesis streams of Region B as new events. Essentially, Region B “sees” a flood of events as if the customer were quickly creating all their data from scratch. The pipeline in Region B then processes these events normally rebuilding the customer’s dataset in that region.
Throttling and monitoring: The migration is done in a controlled manner so as not to overwhelm the destination. Atlassian can adjust the rate of events being injected and ensure that live, incoming data in Region B continues smoothly. They isolate migration events in a separate queue and merge them gradually, giving priority to any real-time activity.
Delete from source: Once Region B has the full, up-to-date copy of the customer’s data and everything is verified, Atlassian removes the data from Region A. They insert special tombstone events for all of that customer’s records in Region A, which the pipeline there processes as deletions. This clean-up ensures no personal or residual data remains in the old location, fulfilling data residency commitments and privacy obligations.

The same mechanism that moves data can also be used to delete data completely. If a customer decides to leave Atlassian Cloud and requests erasure of their data, the system can generate tombstone events for all their records just as in a migration source deletion. Because each customer’s data is partitioned and indexed, Atlassian can precisely target and wipe the data without affecting others. This level of control was built in from the start to comply with strict privacy laws.

Delivering Value Through Analytics and Exports

All these engineering efforts led to a powerful platform that unlocked new capabilities for Atlassian’s cloud customers:

Zero-effort analytics: Atlassian launched Atlassian Analytics, a built-in analytics interface that sits on top of the new data lake. Customers can log in and immediately analyze their Jira, Confluence, and other product data through interactive dashboards or SQL queries. Because the data lake is pre-populated and continuously updated, even a brand-new cloud customer can get insights from their data on day one. This fulfilled the promise of instant analytics by providing a web-based BI tool powered entirely by the unified data lake.
Secure data exports via Delta Sharing: For organizations that prefer to work in their own business intelligence environment, Atlassian introduced a safe data export feature. Using the open Delta Sharing protocol, Atlassian can share a customer's portion of the data lake with them, without making separate database dumps. Customers who opt in get a Delta Sharing endpoint and credentials, allowing their analysts to connect with tools like Databricks or Snowflake and query their Atlassian data directly from Atlassian’s cloud storage. This means they receive up-to-date data increments in a controlled, secure manner, solving the previous cloud export limitation. It’s like having a dedicated data warehouse for each customer’s Atlassian data, without duplicating data or compromising on security.
Improved in-product dashboards: With the heavy lifting done by the data lake, Atlassian’s products themselves benefited. Jira and Confluence could offer richer built-in reports and dashboards by querying the data lake instead of the production database. For example, Jira dashboards can now show historical trends and cross-project statistics that were formerly impractical to compute live. Offloading reporting queries to the read-optimized data lake makes the apps more responsive for end users and ensures the transactional databases stay focused on their primary job. As a bonus, Atlassian can innovate with more advanced analytics in the products knowing the performance impact on the core system is minimal.

Diagram showing Atlassian Data Lake feeding enriched, modeled, cross-product data into Atlassian Analytics dashboards.

Equally important, Atlassian delivered these features in a way that meets enterprise expectations for security and compliance. The platform was designed from day one with multi-region data residency, encryption of data at rest and in transit, and robust deletion workflows. Building those capabilities into the foundation saved a lot of future headaches and made the data lake solution viable for even the most regulated customers.

A Platform Built for Scale and Compliance

Atlassian’s journey to build a multi-region compliant customer data lake shows the power of long-term architectural thinking. By investing in a flexible logical replication model and a real-time streaming pipeline, Atlassian created a platform that not only solved immediate problems but also provided a launchpad for future data-driven features. The decision to tackle the harder engineering work up front, instrumenting applications to emit events, designing a stable schema contract, and automating infrastructure across regions has paid off in a resilient, scalable system that grows with their customers’ needs.

For other organizations facing similar data challenges, Atlassian’s approach offers valuable insights. A well-designed data lake can become the single source of truth for analytics, enabling new products and better customer experiences without compromising on compliance. By treating data as a product and empowering domain teams, you can build a solution that is both agile and robust against change.

Interested in learning more? Contact Atlas Bench to book a complimentary consultation with our Certified Atlassian Experts.

Start analyzing your Jira and Confluence data instantly with Atlassian Analytics, sign up today!

undefined

Atlas Bench is a Platinum Atlassian Solution Partner with deep Cloud Specialization, helping Data Center customers move beyond migration and fully transform on the Atlassian Cloud Platform. We deliver Atlassian’s System of Work end to end through four collections: Strategy, Teamwork, Service, and Software, connecting planning to execution, aligning teams around outcomes, modernizing service delivery, and accelerating software development.

We are also recognized as an Atlassian World Class Software Development Partner of the Year (Americas), and we bring AI leadership to every engagement. By leveraging Rovo for automation and purpose built agents, we reduce manual work, unlock knowledge across tools, and speed up decision making so organizations realize measurable value faster in cloud.

Building a Multi‑Region Compliant Customer Data Lake at Scale