Your CDP Is Not a Database: It’s a Streaming Engine

Your data isn’t messy because your team is lazy; it’s messy because your architecture assumes batch processing in a real-time world. We spend millions on tools that promise a "single source of truth," yet we still wait until tomorrow morning to see what happened today. The latency is not a bug. That is the default state of systems designed for a different era. Most top-ranking articles describe CDPs as static databases where you dump data and query it later. This is a fundamental misunderstanding of the problem. A true CDP must be a stateful streaming engine where identity resolution happens at ingest, not at query time, reducing the 'time-to-truth' from days to milliseconds. If your platform cannot resolve a user’s identity across devices before the next page load, it is not a CDP; rather, it is a data warehouse with a marketing label.

What is a CDP customer data platform?

A Customer Data Platform (CDP) is software that collects and unifies customer data from various sources to create a single, centralized customer profile. However, this definition hides the mechanical complexity required to make it useful. According to industry definitions, a CDP aggregates and organizes data from various touchpoints. But aggregation without immediate resolution is just hoarding. The core tension lies in how we define "unified." In traditional architectures, unification is a nightly ETL (Extract, Transform, Load) job. You collect emails, clickstreams, and purchase histories during the day, then stitch them together while everyone sleeps. By the time the profile is updated, the user has already moved on. The opportunity for real-time personalization or fraud detection is lost. We need to stop thinking of CDPs as repositories. They are processors. The value is not in storing the data; it is in the speed at which the system can say, "This person on mobile is the same person who just logged in on desktop, and they are currently angry." That statement requires stateful computation at the edge of ingestion.

The Latency Lie and the Identity Trap

The failure of most CDP implementations stems from two architectural myths: that hourly scrapes provide sufficient freshness, and that identity resolution can be deferred. These assumptions create a false sense of data unity. You have all the data, but you don’t have the right data at the right time.

The Latency Lie

Batch processing creates a window of ignorance. If your CDP ingests data every hour, the system remains blind to new behavior for the vast majority of that cycle. In civic data contexts, such as the work done by platforms analyzing public tenders or municipal vitals, this delay can mean missing a critical pattern in procurement fraud or welfare distribution. As noted in our analysis of real-time bid monitoring, sub-second pipelines are necessary to catch anomalies as they happen, not after the contract is signed. Marketing teams accept this latency because they are used to campaign cycles measured in weeks. But digital interactions happen in milliseconds. When a user abandons a cart, the intervention must happen before they close the tab. A batch-based CDP sees the abandonment tomorrow, whereas a streaming CDP sees it now.

The Identity Trap

Traditional ETL fails to resolve users across devices in real time because it treats identity as a static attribute rather than a dynamic probability. Deterministic matching uses explicit identifiers like email addresses. That approach is reliable but rare. Most interactions are anonymous or semi-anonymous. Probabilistic matching applies statistical inference to connect data points using behavioral patterns. Doing this in a batch job is computationally cheap but temporally useless. Doing it in real time is hard. It requires maintaining state for millions of users simultaneously. Most systems offload this to query time, meaning every time you ask "Who is this user?" the system has to re-calculate the probability. That approach is inefficient and slow. The trap is believing that having the data means you know the user. You do not. You only know the user once the identity graph is resolved. If that resolution takes hours, you are acting on stale assumptions.

Streaming Pivot: From Tables to Stateful Streams

To fix this, we must move from static tables to stateful streams. In a streaming architecture, every event triggers an update to the user profile immediately. The profile is not a row in a database; it is a living object in memory, constantly revised by incoming events. This approach changes how we think about enterprise data platforms. Instead of moving data into a warehouse and then querying it, we process the data as it flows. The warehouse becomes a historical archive, while the CDP becomes the real-time brain. Consider the evolution of civic infrastructure. Axon Enterprise, for example, is evolving from a hardware vendor into an integrated operating system for public safety. As noted in recent analysis, the value is no longer in the body camera itself, but in the integrated data stream it provides. Similarly, a CDP’s value is not in the storage, but in the integrated identity stream.

How cdp works simply: The Streaming Architecture

Understanding how CDPs function in a modern context requires looking past the marketing cloud veneer. At its core, a streaming CDP is a pipeline that performs three actions simultaneously: ingest, resolve, and activate. Here is how how cdp works simply in a real-time architecture:

1. Ingest Events via Stream

   Data arrives not as files, but as discrete events. Each event carries a timestamp, a user identifier (even if anonymous), and a payload. Tools like Apache Kafka or AWS Kinesis handle the high-throughput ingestion. The crucial detail is that nothing is written to disk permanently until it is processed.

   event = { user_id: "anon_123", action: "view_product", ts: 1727184000 }
2. Resolve Identity in Flight

   As the event enters the stream, the system checks the current state of the user profile. If "anon_123" is linked to "user_456" via a previous login, the event is attributed to "user_456". The lookup happens in microseconds, utilizing in-memory stores like Redis. This is real time customer data integration in practice.

3. Update Stateful Profile

   The user profile is updated immediately. If the user viewed a product, the "interest" score increases. If they abandoned a cart, the "churn risk" flag is set. This state is persisted for future events but is available for activation instantly.

4. Activate via Webhooks or APIs

   Based on the updated profile, the system triggers actions. An email is sent, a discount is applied, or an alert is raised. The loop closes in milliseconds.

This architecture ensures that your first party data strategy is not just about compliance, but about competitiveness. You own the data, and you own the speed at which you use it.

The Governance Cost

Real-time ingestion requires stricter schema enforcement than batch jobs. In a batch system, you can clean dirty data after it lands. In a stream, dirty data breaks the pipeline. You must define strict schemas upfront. Every event must conform. That is the governance cost of speed. At Halantir, we deal with similar challenges in civic data. When consolidating government records from multiple countries, we cannot afford to guess what a field means. Our machine-driven approach enforces structure at ingest. If a record doesn’t fit the schema, it is rejected or quarantined immediately. That prevents pollution of the unified profile.

The Verification Gap

Balancing speed with accuracy is the hardest part. When merging disparate records, you risk false positives. Linking two different people because they share an IP address is a privacy violation and a business error. We address this by using a confidence score. Every identity link has a probability. Low-confidence links are flagged for review or used only for broad segmentation. High-confidence links trigger immediate actions. That tiered approach allows us to move fast without breaking trust. It mirrors how we handle data integrity in public records, where every figure must have a receipt.

Tools and Architecture Choices

Building this system requires specific tools. We do not recommend monolithic suites that hide the mechanics. We favor composable components that you can control. Before you provision a single cluster, you need a rigorous cdp implementation checklist to ensure your streaming layer can handle the load without collapsing under backpressure.

• Apache Kafka: For high-throughput event ingestion. It is the backbone of any serious streaming architecture.
• Redis: For stateful profile storage. It provides the microsecond lookups needed for real-time identity resolution.
• AWS Kinesis: A managed alternative to Kafka for teams that prefer cloud-native services.
• dbt: For transforming data in the warehouse. While the CDP handles real-time, dbt handles the historical analysis and complex aggregations that don’t need to be instant.

Avoid tools that promise to do everything. A composable CDP approach, where profiles are built in your existing data warehouse but activated via streaming layers, offers the best balance of control and speed. This is distinct from marketing clouds like Salesforce Data 360 or Adobe Marketing Cloud, which often lock you into their walled garden. While those tools have their place, they rarely expose the underlying streaming mechanics needed for true real-time resolution. For civic tech applications, where transparency is essential, open standards are critical. We rely on structured access to ensure that the data pipeline is auditable. You should be able to trace every profile update back to its source event.

Is a CDP worth the investment?

The question is not whether a CDP is worth it, but whether you can afford the latency of not having one. If your business relies on real-time interaction · whether it is e-commerce, fraud detection, or civic engagement · the cost of delay is higher than the cost of implementation. However, real-time CDPs are expensive to build and maintain. They require engineering rigor, not just marketing budget. Teams need developers who understand distributed systems, not just SQL. Our experience at Halantir shows that the effort pays off in data integrity. By treating data as a stream, we catch errors early. We maintain a high standard of methodological transparency, ensuring that our insights are based on clean, timely data.

Feature	Batch Processing CDP	Real-Time Streaming CDP
Identity Resolution	Nightly ETL Job	At Ingest (Milliseconds)
Data Freshness	Hours to Days	Seconds
Schema Enforcement	Post-Ingest Cleaning	Strict Pre-Ingest Validation
Complexity	Low (Managed Services)	High (Engineering Heavy)

How we hit it

We do not just write about data architecture; we live it. Our platform processes thousands of civic records daily, requiring the same rigor as a commercial CDP.

• This site has published 41 articles in the last 90 days.
• Google URL Inspection shows 48% of this site's 33 pages that have been live at least 14 days or are already indexed are indexed.
• Median time from publish to confirmed Google indexing on this site: 6 days, across 16 posts we measured.

These numbers reflect our commitment to consistent, high-quality output. Just as we optimize our publishing pipeline, we optimize our data pipelines. The data analysis console we offer allows users to query these integrated datasets with the same speed and precision we apply to our own infrastructure. We have learned that twelve desks ride one spine. Each desk represents a different data domain, but they all rely on the same underlying identity resolution engine. This unity is what allows us to detect patterns across borders and sectors. One honest admission: we initially tried to use batch processing for our civic data ingestion. The approach failed. The lag between a government announcement and our analysis was too great. We had to pivot to a streaming model, which required rewriting our entire ingestion layer. The process was painful but necessary. The scar tissue remains in our codebase, but it makes us stronger.

Experiments to try

Do not take our word for it. Test your own system.

1. Measure the Latency Gap

   Perform a specific action on your website (e.g., add an item to cart). Measure the time until that action appears in your analytics dashboard. Then, measure the time until it appears in your raw event stream. The difference is your "time-to-truth" lag. If it is more than a minute, you are flying blind.

2. The Blind Merge Test

   Attempt to merge two conflicting user records from different sources without a unique identifier (like email). Use only behavioral data (IP, device type, timing). See if your current system links them correctly. If it fails, or if it links them incorrectly, your identity resolution logic is weak. That exposes the reliance on deterministic matches and highlights the need for probabilistic streaming resolution.

At what point does the cost of real-time identity resolution outweigh the business value of immediate data availability? For most, that point is further away than they think. The world moves faster than your batch jobs. Catching up is necessary.

HEIMLANDR -- Builders of the official layer of the Nordics.