Skip to main content
Duspat

Service

Data Engineering & Integration

Batch, streaming and event-driven pipelines that hold up in production — with the data contracts, quality checks and observability that keep them holding up.

Who it is for

  • A team whose pipelines fail silently, and who find out from the business rather than an alert
  • An organisation integrating systems that were never designed to talk to each other
  • A platform team drowning in bespoke one-off jobs, each with exactly one person who understands it

Problems we solve

What tends to be going wrong.

No contract, no test, no owner
A pipeline that works until an upstream schema changes on a Friday. Nothing declared the shape it expected, so nothing failed loudly — it just produced wrong numbers.
Every integration written from scratch
The fifth source system means the fifth bespoke job, sharing no pattern with the previous four. Cost scales linearly with sources, and so does the fragility.
Failures found by the business
The first signal that a pipeline broke is somebody asking why the dashboard looks wrong. By then it has been wrong for three days.

How we help

What Duspat delivers.

01
ETL/ELT and batch pipelines
Orchestrated, tested, idempotent pipelines with declared contracts — built with Airflow, DBT, PySpark and Glue, and designed to fail loudly rather than quietly.
02
Streaming and event-driven architecture
Kafka, Kinesis and event-driven integration where latency genuinely matters — and an honest conversation about batch where it does not.
03
APIs and integration frameworks
API-led integration and reusable, configuration-driven frameworks rather than bespoke jobs, so the sixth source system costs a fraction of the first.
04
Data quality, monitoring and remediation
Quality checks with named owners, monitoring that alerts a human who can act, and a remediation path defined before it is needed at 2am.

Typical deliverables

Things you can hold, review and hand to a team.

Design-and-build, or a remediation engagement on pipelines that already exist and can no longer be trusted.

  • Batch and streaming pipeline design and build
  • Event-driven integration architecture
  • API-led integration and API design
  • Reusable, configuration-driven integration frameworks rather than bespoke jobs
  • Data contracts and schema-change handling
  • Data quality checks, monitoring, alerting and a remediation path
  • Metadata cataloguing and end-to-end lineage
  • Migration and change-data-capture patterns

Technology fit

What we build this on.

We are independent: no reseller agreements and no partner quotas. If your estate points a different way, we will say so.

Orchestration & transformation
  • Apache Airflow
  • DBT
  • PySpark
Streaming
  • Apache Kafka
  • Kinesis
  • Spark Streaming
  • Snowpipe
AWS
  • AWS Glue
  • Lambda
  • EventBridge
  • SQS
  • SNS
  • DMS
Targets
  • Snowflake
  • Redshift
  • Databricks
  • BigQuery
  • PostgreSQL

Governance & production readiness

Designed in, not added later.

A pipeline is a governance surface whether or not you treat it as one. Lineage, quality and access are properties of the design, not a dashboard added afterwards.

How we run an engagement

  • Data contracts, so a schema change is a failed build rather than a wrong number
  • Automated data quality checks with named owners
  • End-to-end lineage and metadata cataloguing
  • Monitoring and alerting that reaches a human who can act on it
  • GDPR-related controls, retention and auditability applied in the pipeline

Example outcomes

Evidence, not testimonials.

Anonymised and sector-level. Client names are withheld by design — the constraint and the architecture are the parts that carry any information.

  • Aviation

    Transaction data for reconciliation and billing

    The constraint. Fuel transaction data feeding reconciliation, billing and reporting — where an error is a financial error, and must be traceable.

    Batch and streaming pipelines with lineage, data quality checks and monitoring, replacing a reconciliation process that had been running on spreadsheets and trust.

    • Airflow
    • PySpark
    • DBT
    • BigQuery
    • Kubernetes
    • Superset
  • Regulated data

    Data governance in production, not on paper

    The constraint. Governance that had been written down but never enforced, across an estate holding personal data.

    Metadata cataloguing, lineage, RBAC, retention and automated data quality checks with named owners and a remediation path — so that a governance claim became something a regulator could be shown rather than told.

    • Data quality
    • Lineage
    • RBAC
    • GDPR controls
    • Monitoring

Questions

Common questions about data engineering.

What is the difference between ETL and ELT?

ETL transforms data before loading it; ELT loads it first and transforms it inside the warehouse. ELT is the modern default because cloud warehouses make transformation cheap and it keeps the raw data available when a definition changes — which it will. ETL still wins where data must be filtered or masked before it lands.

When should we use streaming rather than batch?

When a decision genuinely depends on freshness measured in seconds. Streaming is materially harder to build, test and operate, and most reporting requirements described as real-time turn out to mean hourly. We will ask what actually changes if the data is an hour old, and if the answer is nothing, we will recommend batch.

How do you ensure data quality in a pipeline?

With declared data contracts and automated checks that fail the build rather than warn quietly, each with a named owner and a remediation path. A quality check nobody owns is a check nobody fixes, and a pipeline that degrades silently is worse than one that stops.

What is event-driven architecture, and when does it help?

Systems communicate by emitting events rather than calling each other directly, so producers and consumers can change independently. It helps when you have many integrations, variable load, or a need for auditability — every state change leaves a record. It hurts when a simple synchronous call would have done.

Talk through a pipeline you no longer trust.

A first call is technical, not a sales call. Bring a problem, an architecture, or a stalled initiative.

30 minutes, no pitch deck.