Architecture Lab

Interactive code modules detailing scalable engineering patterns. Hit Run to see simulated output.

Engineering Patterns

dbt Incremental Strategy

Composite key approach for unique_key to manage upserts efficiently.

Schema Definition Standards

Strict adherence to data typing standards defining all string columns as unbounded varchar.

Column Normalization Macro

Jinja/SQL macro that standardizes incoming raw data columns by replacing spaces with underscores.

Python S3 Ingestion Pipeline

Boto3-powered incremental ingestion from S3 with schema validation and Athena partition registration.

Dynamic Airflow DAG Factory

Programmatically generates DAGs from a config file — one pattern drives all pipeline schedules.

model.sql
1{{
2  config(
3    materialized='incremental',
4    unique_key=['user_id', 'event_timestamp'],
5    incremental_strategy='merge'
6  )
7}}
8 
9SELECT * FROM {{ ref('stg_events') }}
10{% if is_incremental() %}
11  WHERE event_timestamp > (SELECT max(event_timestamp) FROM {{ this }})
12{% endif %}