Public architecture showcase of a private production-oriented Data Engineering proof of concept. Source code, internal infrastructure, credentials, proprietary data, operational identifiers and company-specific integrations are intentionally not published.
This repository documents the architecture, engineering decisions and lessons learned from building a Lakehouse-style data platform using a Medallion Architecture (Bronze, Silver and Gold).
The goal of the original initiative was to validate a scalable data-engineering approach for operational analytics, decoupling raw analytical workloads from transactional systems while preserving historical data, improving reproducibility and creating a clean serving layer for BI.
- End-to-end Data Engineering architecture
- Medallion Architecture with Bronze, Silver and Gold layers
- Distributed processing with Apache Spark / PySpark
- ACID data storage with Delta Lake
- S3-compatible object storage with MinIO
- Batch ingestion and workflow integration with Apache Airflow
- JDBC ingestion from relational databases
- Data transformation, cleansing, aggregation and historical preservation
- Dedicated Gold serving layer for BI consumption
- Pragmatic architecture decisions based on operational constraints
- Production-oriented concerns such as memory pressure, fault isolation, schema evolution and data quality
flowchart TB
SRC["Operational Data Sources"]
ORCH["Airflow / Scheduled Ingestion"]
DB[("Relational Databases")]
subgraph LAKE["Lakehouse Platform"]
BRONZE[("Bronze Layer\nRaw / Historical")]
SPARK["PySpark + Delta Lake"]
SILVER[("Silver Layer\nCleaned / Standardized")]
GOLD[("Gold Layer\nAggregated / Business Ready")]
end
SERVING[("Gold Serving Database")]
BI["BI / Analytics"]
SRC --> ORCH
ORCH --> BRONZE
DB --> SPARK
BRONZE --> SPARK
SPARK --> SILVER
SILVER --> SPARK
SPARK --> GOLD
GOLD --> SERVING
SERVING --> BI
The architecture intentionally separates raw historical storage, data processing, business-ready analytical models and BI serving.
| Area | Technologies |
|---|---|
| Processing | Apache Spark, PySpark |
| Lakehouse storage | Delta Lake |
| Object storage | MinIO / S3-compatible storage |
| Orchestration | Apache Airflow |
| Relational integration | PostgreSQL, JDBC |
| Transformation | Python, SQL, dbt concepts/workflows |
| BI serving | PostgreSQL |
| Analytics | Metabase / BI tools |
| Infrastructure | Docker, Docker Compose, Linux |
| Development | JupyterLab |
The Bronze layer preserves raw or minimally transformed data by execution period rather than continuously overwriting the latest state.
This provides:
- historical traceability;
- reproducible processing;
- safer reprocessing;
- comparison between snapshots;
- separation between source extraction and downstream analytics.
Data can arrive from scheduled file ingestion or directly from operational relational systems.
The Silver layer applies data engineering rules such as:
- schema normalization;
- type conversion;
- deduplication;
- business-key normalization;
- joins between reference dimensions;
- data-quality rules;
- technical metadata;
- historical consistency.
Delta Lake is used to provide transactional guarantees, schema enforcement and a stronger foundation for incremental evolution than plain files alone.
The Gold layer contains aggregated and business-oriented datasets optimized for analytical consumption.
Instead of coupling BI directly to the transformation engine, the architecture also publishes selected Gold datasets to a lightweight relational serving database.
This creates a clean separation:
Lakehouse = analytical source of truth + history
Serving DB = simplified access layer for BI
It also avoids requiring every BI tool to understand Delta Lake or object-storage formats directly.
Two primary ingestion patterns were validated:
Source System
↓
Scheduled Extraction
↓
Airflow
↓
Object Storage / Bronze
↓
PySpark
↓
Silver → Gold
Operational Database
↓
JDBC
↓
PySpark
↓
Bronze / Silver
↓
Gold
For large relational tables, JDBC reads were configured to use controlled fetch sizes rather than loading an entire result set into driver memory.
This was an important reliability improvement for large-volume ingestion.
Bronze data is partitioned by execution period so previous states remain available for reprocessing, auditing and change detection.
Silver and Gold datasets benefit from ACID semantics, schema enforcement, versioning capabilities and reliable updates.
A relational Gold serving database was selected for BI access rather than introducing a heavier distributed SQL query layer during the proof-of-concept phase.
The decision favored:
- lower memory footprint;
- reduced operational complexity;
- native BI compatibility;
- faster validation of the architecture.
The heavier query-engine alternative remains a valid evolution when infrastructure capacity and workload requirements justify it.
New data-lake ingestion was introduced alongside existing operational flows rather than immediately replacing critical production paths.
This reduced migration risk and allowed the new architecture to be validated before becoming a dependency of operational systems.
Large database reads can exhaust driver memory when the JDBC driver buffers too much data. Using an explicit fetchsize enables streamed retrieval in manageable batches.
Real operational datasets frequently contain sentinel or historically old date values. Spark date-rebase behavior must be configured deliberately when writing Parquet-compatible formats.
When integrating Spark/Hadoop S3A with non-AWS object storage, endpoint, region and path-style configuration need to be explicit.
A technically elegant architecture is not always the operationally correct architecture. Resource consumption, available memory, deployment complexity and supportability were treated as first-class architectural constraints.
The project applied several principles commonly required in operational data platforms:
- stable technical keys instead of relying exclusively on reusable business identifiers;
- snapshot-based historical analysis;
- separation of raw source data from curated dimensions;
- aggregated Gold models for consumption;
- explicit handling of incomplete or sparse attributes;
- validation of totals and joins after source consolidation;
- preservation of auditability across pipeline stages.
The POC was designed to test whether analytical workloads could evolve from a relational-database-centric approach toward an architecture where:
Operational Systems
↓
Raw Historical Storage
↓
Distributed Processing
↓
Curated Analytical Data
↓
Serving / BI
This approach reduces coupling between ingestion, transformation and visualization while providing a clearer path toward larger-scale Data & AI workloads.
I was responsible for the technical design and implementation of the proof of concept, including:
- architecture definition;
- containerized data platform setup;
- PySpark pipelines;
- Delta Lake integration;
- object storage integration;
- JDBC ingestion;
- Bronze / Silver / Gold modeling;
- BI serving architecture;
- troubleshooting and performance analysis;
- integration strategy with existing workflows;
- evaluation of architectural alternatives.
This repository does not contain:
- production source code;
- internal IP addresses or hostnames;
- credentials, tokens or connection strings;
- production database names;
- internal system names;
- proprietary business rules;
- customer or operational data;
- internal directory structures;
- company-specific Airflow DAG names;
- infrastructure topology;
- production runbooks;
- private datasets;
- company-specific metrics or identifiers.
The purpose of this repository is to demonstrate engineering capability and architectural reasoning without exposing proprietary implementation details.
This project connects directly with:
Data Engineering · Lakehouse Architecture · PySpark · Delta Lake · ETL/ELT · Data Modeling · Data Quality · Airflow · PostgreSQL · Docker · BI · Distributed Processing · Analytics Engineering
Jailson Bezerra
Software Engineering · Automation · Data · AI · Technical Leadership
- LinkedIn: https://linkedin.com/in/jailsonsb
- Portfolio: https://jailson.es
Private implementation. Public architecture documentation provided for portfolio purposes only.