Every data team eventually hits the same wall. Raw tables pile up in the warehouse. Analysts write CTEs in Jupyter notebooks that nobody else can read or trust. Someone builds a revenue dashboard, then another analyst builds a slightly different one with a different filter, and now leadership is asking why two reports give different numbers in the same board meeting. dbt — the data build tool — was built specifically to dismantle that wall. It brings software engineering discipline (version control, testing, documentation) to SQL transforms, and it does it without asking you to learn a new programming language.
What Problem dbt Solves
Before dbt became mainstream, the analytics workflow at most companies looked roughly like this: raw data lands in a warehouse via some ETL pipeline, a data analyst opens a notebook, writes a long chain of SQL CTEs to clean and reshape it, and saves the result somewhere — sometimes as a materialised view, sometimes just as a saved query that gets copy-pasted into the next notebook. There is no version control. There are no tests. If a source table changes its column name or drops a field, nobody finds out until a dashboard goes blank at 9 AM on a Monday.
dbt replaces that scattered workflow with a project-based structure where every transformation is a .sql file that lives in a Git repository. Each model is tested every time you deploy. Documentation is generated automatically from the code you have already written. The role that emerged alongside this tooling is the analytics engineer — someone who sits between the data engineer who builds pipelines and the analyst who interprets numbers. Analytics engineers own the transformation layer: they make sure the data the business sees is clean, documented, and reliable.
Installing dbt Core in 10 Minutes
dbt Core is open-source and distributed as a Python package, but that does not mean you need Python experience. The installation is a single terminal command. For BigQuery:
pip install dbt-bigqueryFor Redshift:
pip install dbt-redshiftOnce installed, dbt needs a profiles.yml file — typically stored at ~/.dbt/profiles.yml — that tells it how to connect to your warehouse. A BigQuery profile using a service account JSON key looks like this:
my_project:
target: dev
outputs:
dev:
type: bigquery
method: service-account
project: your-gcp-project-id
dataset: analytics_dev
keyfile: /path/to/service-account.json
location: asia-south1
threads: 4Run dbt debug to verify the connection. If you see green checkmarks, you are ready. The entire setup runs comfortably on a ₹800/month DigitalOcean droplet or simply on your laptop — there is no server requirement for dbt Core itself.
Your First dbt Model
A dbt model is just a SQL file. Place it under models/staging/stg_orders.sql and write a SELECT statement that cleans your raw source data:
-- models/staging/stg_orders.sql
with source as (
select * from {{ source('ecommerce', 'raw_orders') }}
),
renamed as (
select
order_id,
customer_id,
cast(order_date as date) as order_date,
cast(total_amount as numeric) as order_total_inr,
lower(status) as order_status
from source
)
select * from renamedRun dbt run and dbt compiles that file, sends it to your warehouse, and creates a view called stg_orders in your target dataset. No dashboard breaks. No manual queries to execute. One command.
The {{ ref() }} function is how you chain models together. If you have a downstream model that needs the cleaned orders data, you reference it like this:
select * from {{ ref('stg_orders') }}dbt parses all the ref() calls in your project and builds a directed acyclic graph (DAG) of dependencies. When you run dbt run, it executes models in the correct order automatically — no need to manually manage execution sequence.
dbt Tests — Data Quality You Can Enforce
Tests are where dbt earns its place in a production analytics stack. You define them in YAML schema files alongside your models:
# models/staging/schema.yml
version: 2
models:
- name: stg_orders
columns:
- name: order_id
tests:
- not_null
- unique
- name: order_status
tests:
- accepted_values:
values: ['pending', 'shipped', 'delivered', 'cancelled']
- name: customer_id
tests:
- not_null
- relationships:
to: ref('stg_customers')
field: customer_idRun dbt test and dbt generates SQL to verify each of these assertions against your warehouse. If an order_id turns up duplicated — perhaps because an upstream pipeline ran twice — the test fails. In a CI/CD pipeline, that failure blocks the deployment before bad data reaches your dashboards. These are not log messages you might catch later; they are hard blockers that protect data consumers downstream.
For logic that the built-in tests cannot cover, you can write custom SQL tests as plain .sql files. Any query that returns rows is treated as a failure — a clean, intuitive convention that any SQL writer can pick up immediately.
Sources and the Lineage Graph
Raw tables that arrive from external pipelines — Fivetran syncs, Airbyte extracts, application database replicas — are declared in a sources.yml file:
# models/sources.yml
version: 2
sources:
- name: ecommerce
database: your-gcp-project-id
schema: raw_data
tables:
- name: raw_orders
freshness:
warn_after: {count: 12, period: hour}
error_after: {count: 24, period: hour}
- name: raw_customersDeclaring sources does two things. First, it lets dbt track the lineage from raw table all the way through to the final reporting model — you can see exactly which raw tables feed which dashboards. Second, the freshness block lets you run dbt source freshness to detect when an upstream load has gone stale.
Run dbt docs generate followed by dbt docs serve and you get an HTML site with a fully interactive lineage graph. When a dashboard breaks and someone asks "which raw table is feeding this metric?", you have a one-click answer. For teams where this question used to involve a two-hour Slack thread, that alone justifies the tooling.
dbt Macros and Jinja Templating
SQL is notoriously bad at reuse. If you have a fiscal year calculation that needs to appear in fifteen different models, you either copy-paste the same CASE WHEN block everywhere or you hope someone remembers to update all fifteen files when the fiscal year boundary changes. dbt solves this with Jinja templating and macros.
A macro is a reusable SQL snippet defined in a macros/ folder:
-- macros/fiscal_year.sql
{% macro fiscal_year(date_column) %}
case
when extract(month from {{ date_column }}) >= 4
then extract(year from {{ date_column }})
else extract(year from {{ date_column }}) - 1
end
{% endmacro %}Use it in any model:
select
order_id,
{{ fiscal_year('order_date') }} as fiscal_year
from {{ ref('stg_orders') }}The dbt_utils package from dbt Hub adds over 50 pre-built macros — date spine generation, pivot functions, surrogate key generation — so you rarely need to write a macro from scratch for common patterns. Add packages to your project with a packages.yml file and run dbt deps to install them.
Deploying in Production
For individual contributors and small teams, two paths make sense.
dbt Cloud free tier gives you one developer seat and one scheduled job. You connect your Git repository, configure a job to run on a schedule, and dbt Cloud handles execution and sends email alerts on failures. This is genuinely useful for getting started without any infrastructure work.
GitHub Actions with dbt Core is more flexible and costs almost nothing. A typical workflow runs dbt test on every pull request (catching bad transforms before they merge) and dbt run on every merge to main (pushing updated models to the production warehouse). The workflow YAML is about 30 lines and reuses the same profiles.yml structure you already know.
dbt Cloud's paid tier starts around $50/month and adds a browser-based IDE, shared lineage access for the whole team, and more granular scheduling options. For solo analysts or teams of two or three, dbt Core with GitHub Actions is the cost-effective path. At five or more collaborators who need simultaneous access to a shared environment, the Cloud tier starts justifying the spend.
When to Use dbt — and When Not To
dbt operates entirely inside your data warehouse. It is the T in ELT — it transforms data that is already loaded. It does not move data between systems, it does not train machine learning models, and it does not process streaming events in real time. Those jobs belong to other tools: Fivetran or Airbyte for extraction and loading, Python or R for statistical modelling, Apache Flink or Kafka Streams for real-time pipelines.
The clearest indicator that dbt is the right tool: you have analysts who know SQL, data that arrives in a warehouse on a schedule, and reporting that depends on multi-step transformations. If your transformation logic currently lives in a mix of stored procedures, notebook cells, and view definitions scattered across different schemas, dbt gives you a single place to own all of it — with version history, tests, and documentation included.
One practical note for Indian teams evaluating the tooling: both BigQuery's ap-south1 (Delhi) region and AWS Redshift's ap-south-1 (Mumbai) region are fully supported by their respective dbt adapters. Choosing an India-hosted warehouse and connecting it to dbt Core satisfies the data localisation considerations that are becoming more relevant as DPDPA 2023 implementation discussions mature.
Frequently Asked Questions
Do I need to know Python to use dbt?
No. dbt is entirely SQL and YAML. The only Python you touch is the pip install command to get dbt Core onto your machine. After that, everything — models, tests, documentation, macros — is written in SQL or YAML config files. An analyst who is comfortable writing SELECT statements can be productive in dbt within a single day. Python only becomes relevant if you want to build custom materializations from scratch, which is an advanced use case most teams never need.
What is the difference between dbt Core and dbt Cloud?
dbt Core is the open-source CLI tool — free forever, but you manage your own scheduling, hosting, and collaboration setup. dbt Cloud is the managed platform from dbt Labs: it adds a browser IDE, a built-in scheduler, shared lineage graphs, and team access controls, starting at around $50/month. For a solo analyst or a small team already comfortable with GitHub Actions, dbt Core is perfectly adequate and effectively zero-cost. dbt Cloud starts earning its fee at five or more collaborators who need simultaneous access to a shared environment.
Does dbt work with data warehouses hosted in India, like AWS Redshift in the Mumbai region?
Yes. dbt supports adapters for more than 30 warehouses, and AWS Redshift (ap-south-1 Mumbai) and Google BigQuery (ap-south1 Delhi) are both fully supported. Keeping your warehouse in an Indian region is increasingly relevant given DPDPA 2023 discussions around data localisation for personal data. If your organisation is evaluating where to store customer data, choosing an Indian-region warehouse and connecting dbt Core to it satisfies both the engineering and compliance angles without any special configuration.