Real-Time Data Reporting Platform

E-commerce Recommendation System

A unified platform to collect, process, analyze recommendation events and power business + engineering dashboards at 2B+ events/day scale.

1. System Overview

Core Objectives

✅ Ingest and process 2B+ clickstream events per day from recommendation surfaces
✅ Deliver real-time CTR / CVR / revenue metrics with sub-second freshness
✅ Support both streaming (Flink/Kafka) and batch (Spark) pipelines
✅ Detect corrupted or missing fields with automated data quality checks
✅ Fire data quality alerts when SLAs are breached
✅ Power BI dashboards for business stakeholders (GMV, CTR, CVR)
✅ Enable historical analysis with a queryable data warehouse
✅ Monitor ML feature freshness for recommendation model health

2. High-Level Architecture

flowchart TD A[User Clicks / Impressions] --> B[Event Collector SDK] B --> C[Kafka — Raw Events] C --> D[Flink Stream Processor] C --> E[Spark Batch Processor] D --> F[Real-Time Metrics Store\nRedis / Druid] E --> G[Data Warehouse\nSnowflake / BigQuery] F --> H[Engineering Dashboard\nGrafana] G --> I[BI Dashboard\nTableau / Superset] D --> J[Data Quality Engine] E --> J J --> K[Alert Manager\nPagerDuty / Slack] G --> L[ML Feature Store\nFeast / Tecton] L --> M[Recommendation Model]

3. Data Ingestion Layer

3.1 Event Schema

Recommendation Event — Required Fields

Field	Type	Description	Required
event_id	UUID	Unique event identifier	Yes
user_id	STRING	Hashed user identifier	Yes
item_id	STRING	Recommended item SKU	Yes
event_type	ENUM	impression / click / add_to_cart / purchase	Yes
timestamp	INT64	Unix epoch ms	Yes
session_id	STRING	Browser/app session	Yes
rec_model_ver	STRING	Model version that served the result	Yes
position	INT	Rank position in recommendation list	Yes
revenue	FLOAT	Transaction value (purchase only)	No
page_context	STRING	Homepage / PDP / Cart	No

3.2 Ingestion Pipeline

flowchart LR A[Mobile SDK] --> D[API Gateway\nKong] B[Web SDK] --> D C[Server Events] --> D D --> E[Kafka\n32 partitions\n3x replication] E --> F[Schema Registry\nAvro]

Kafka Config

Parameter	Value
Topics	raw.events / validated.events / dlq.events
Partitions	32 per topic
Retention	7 days
Throughput target	2.5M events/min peak
Replication	3x

Throughput Estimate

Peak Events/sec

28000

Avg Event Size

~800B

Daily Volume

~1.8TB raw

Kafka Capacity Used (peak)68%

4. Stream Processing — Real-Time Pipeline

flowchart LR A[Kafka\nraw.events] --> B[Flink Job 1\nValidation & Enrichment] B --> C[Kafka\nvalidated.events] B --> D[Kafka\ndlq.events] C --> E[Flink Job 2\nMetric Aggregation] E --> F[Druid\nOLAP Store] E --> G[Redis\nLive Counters]

4.1 Flink Job: Validation & Enrichment

Validation Rules Applied Per Event

✅ Check all required fields are non-null and correctly typed
✅ Validate event_type against allowed enum values
✅ Reject events with timestamp drift > 10 minutes from server time
✅ Enrich user_id with user segment from Redis lookup (age group, tier)
✅ Enrich item_id with category and brand from item catalog cache
✅ Route invalid events to dlq.events topic with failure reason tag

4.2 Flink Job: Real-Time Metric Aggregation

Metric	Window	Sink	Use Case
CTR (clicks/impressions)	1min / 5min tumbling	Druid + Redis	Live dashboard
CVR (purchases/clicks)	5min / 15min tumbling	Druid	Conversion monitoring
Revenue per model	5min tumbling	Druid	A/B experiment tracking
DLQ rate	1min tumbling	Redis + Alert	Data quality alerting
P99 event latency	30s sliding	Redis	Eng health monitoring
Feature freshness lag	1min tumbling	Redis	ML model health

5. Batch Processing — Historical Pipeline

flowchart LR A[Kafka\nraw.events] --> B[S3 / GCS\nRaw Data Lake\nParquet/day] B --> C[Spark ETL\nHourly + Daily] C --> D[Data Warehouse\nSnowflake / BigQuery] D --> E[dbt Models\nBusiness Layer] E --> F[BI Dashboard\nTableau / Superset] E --> G[ML Feature Store\nFeast / Tecton]

5.1 Spark Batch Jobs Schedule

T+00 — 05: Hourly micro-batch — Spark reads S3 partition, validates schema, writes to DW staging

T+01 — 00: Hourly aggregation — CTR/CVR/revenue rolled up per hour per model version

T+06 — 00: Daily full refresh — Recompute 30-day cohort metrics, funnel analysis

T+07 — 00: ML feature batch — Recompute collaborative-filtering features, write to Feature Store

T+08 — 00: dbt run — Refresh business-layer models for BI dashboards

5.2 Data Warehouse Schema (Star Schema)

Core Tables

Table	Type	Grain	Retention
fact_rec_events	Fact	1 row per event	2 years
fact_rec_daily	Fact (agg)	model × date × category	3 years
dim_item	Dimension	1 row per item	Current
dim_user_segment	Dimension	1 row per user tier	Current
dim_model_version	Dimension	1 row per model deploy	Permanent
dim_date	Dimension	1 row per date	Permanent

6. Data Quality & Alerting

6.1 Data Quality Engine

flowchart TD A[Validated Events Stream] --> B[Quality Rule Engine\nFlink / Great Expectations] B --> C{Pass?} C -- Yes --> D[Metrics Store] C -- No --> E[DLQ + Quality Event] E --> F[Alert Manager] F --> G[PagerDuty\nP1 — Data Outage] F --> H[Slack\nP2 — Quality Degradation] F --> I[Dashboard Warning\nP3 — Anomaly]

6.2 Quality Rules Catalog

Rule	Threshold	Action
Null event_id rate	>0.01%	P1 Alert + pause ingestion
Null user_id rate	>0.5%	P2 Alert
Invalid event_type	>0.1%	P2 Alert + route to DLQ
Timestamp drift >10min	>1%	P2 Alert
Missing revenue on purchase	>0.01%	P1 Alert

Rule	Threshold	Action
Events/min drop >50%	vs 7-day avg	P1 Alert
Events/min drop >20%	vs 7-day avg	P2 Alert
CTR anomaly (z-score)	>3 sigma	P2 Alert
Zero events >2min	Any topic	P1 Alert
DLQ rate spike	>0.5%	P2 Alert

Rule	Threshold	Action
Stream lag behind wall clock	>5min	P1 Alert
Hourly batch delay	>30min	P2 Alert
Daily batch delay	>2hr	P2 Alert
Feature store staleness	>1hr	P2 Alert
DW table not updated	>2hr	P2 Alert

⚠️

Alert Routing Policy

P1 alerts page on-call engineer immediately via PagerDuty. P2 alerts post to #data-quality Slack channel. P3 anomalies surface as dashboard warning banners only. All alerts auto-attach a runbook link.

7. Dashboards

7.1 Business Dashboard (BI — Tableau / Superset)

Key Business KPIs

Overall CTR

4.2%▲ +0.3%

Target: 4.5%

CVR

2.1%▲ +0.1%

Target: 2.5%

Revenue via Rec

$1.8M/day▲ +8%

Target: $2M/day

Rec Coverage

72%▲ +2%

Target: 80%

Model Performance

7.2 Engineering Dashboard (Grafana)

Pipeline Health

Kafka Consumer Lag

<5K msgs

Flink Checkpoint Interval

30s

Spark Job P99 Duration

22min

DLQ Rate (live)

0.04%

Infrastructure

Kafka Disk Usage61%

Flink TM Memory74%

Druid Query Cache Hit88%

Redis Memory55%

8. ML Feature Freshness Monitoring

Feature Store Design

flowchart LR A[Batch Features\nSpark — daily] --> C[Feature Store\nFeast / Tecton] B[Stream Features\nFlink — 1min] --> C C --> D[Online Store\nRedis — serving] C --> E[Offline Store\nS3 + DW — training] D --> F[Recommendation\nModel Server]

8.1 Feature Freshness SLAs

Feature Group	Update Freq	Max Allowed Staleness	Alert Threshold
User interaction history	1 min stream	5 min	>3 min lag
Item popularity scores	5 min stream	15 min	>10 min lag
User-item affinity matrix	Hourly batch	2 hrs	>90 min lag
Collaborative filter emb	Daily batch	26 hrs	>25 hr lag
User demographic features	Daily batch	26 hrs	>25 hr lag

ℹ️ Feature freshness metrics are published to Grafana and compared against a rolling 7-day baseline. Any staleness breach automatically blocks model promotion in the CI/CD pipeline.

9. Technology Stack

Component	Technology	Purpose
Event SDK	Custom JS/iOS/Android	Client-side event capture
API Gateway	Kong	Rate limiting, auth, routing
Message Broker	Apache Kafka	Durable event streaming
Schema Registry	Confluent SR (Avro)	Schema enforcement and evolution

Component	Technology	Purpose
Stream Processing	Apache Flink 1.18	Real-time metrics and validation
Batch Processing	Apache Spark 3.5	Historical aggregation and ETL
Orchestration	Apache Airflow	DAG scheduling for batch jobs
Transformation	dbt	Business-layer SQL models

Component	Technology	Purpose
Raw Data Lake	S3 / GCS (Parquet)	Immutable event archive
OLAP Store	Apache Druid	Sub-second real-time queries
Live Counters	Redis Cluster	Real-time metric serving
Data Warehouse	Snowflake / BigQuery	Historical analytics and BI
Feature Store	Feast + Tecton	ML feature serving and training

Component	Technology	Purpose
Engineering Board	Grafana	Pipeline health, infra metrics
BI Dashboard	Tableau / Superset	Business KPIs and reporting
Alerting	PagerDuty + Slack	Incident and quality alerts
Data Quality	Great Expectations	Rule-based data validation
Tracing	OpenTelemetry + Jaeger	Distributed pipeline tracing

10. Scalability & SLA Summary

Throughput Targets

Daily Events

2B+▲ +5%

Target: 2B

Peak Throughput

28K/sec▲ +8%

Target: 30K/sec

Stream End-to-End

<500ms▼ 0%

Target: <500ms

Batch SLA

<30min▼ -5%

Target: <30min

Reliability Targets

Ingestion Uptime

99.99%▼ 0%

Target: 99.99%

Data Completeness

>99.9%▲ +0.02%

Target: 99.9%

DLQ Rate

<0.1%▼ -0.02%

Target: <0.1%

Feature Freshness SLA

99.7%▲ +0.2%

Target: 99.5%

Key Design Decisions

Lambda Architecture — Flink for speed layer (real-time), Spark for batch layer (accuracy), Druid/DW as serving layer satisfies both latency and historical query needs
Schema Registry with Avro — Enforces schema evolution contracts at ingestion time, preventing downstream breakage from producer changes
DLQ-first validation — Invalid events are never silently dropped; they're routed to dlq.events for replay after fixes, ensuring no data loss
Feature freshness gating — Model promotion in CI/CD is blocked if any feature group breaches its freshness SLA, preventing stale-feature degradation in production
Tiered alerting (P1/P2/P3) — Reduces alert fatigue by separating actionable outages (P1) from quality degradation (P2) and monitoring anomalies (P3)

🚀

Next Steps

Phase 1: Deploy Kafka + Flink stream pipeline with validation. Phase 2: Build Spark batch ETL and DW models. Phase 3: Launch BI + Grafana dashboards. Phase 4: Integrate Feature Store with ML model CI/CD.