Case Study
How a Property Services Business Grew Sales Revenue 20% After Rebuilding Its Data.
A property services business needed current location, property, and weather data to focus its sales outreach. We rebuilt the enrichment pipelines, cutting processing from 5–6 hours to 30 minutes. The project owner reports a 20% increase in sales revenue following the work.
- Pipeline processing, down from 5–6 hours
- 30 Min
- Sales revenue increase reported by the project owner
- 20%
- Property attributes added per record
- ~60
The Challenge
The Sales Opportunity Needed More Than an Address.
Give sales teams current, richer context about the places they serve, supported by a data process the business can run regularly.
The Business Context
Sales and marketing used service-location information and external data to identify where their services could be relevant. Property characteristics helped explain the locations in their market; weather and air-quality information added context for when to approach those customers.
Regular refreshes depended on pulling millions of records from third-party sources, with pipeline runs taking 5–6 hours. Raw address strings also needed to become usable locations with coordinates and property details. The work had to move faster while respecting provider request limits and handling incomplete or unmatched records.
- Data refreshes took hours before updated information was available for analysis and outreach.
- Inconsistent addresses needed reliable coordinates and property matches.
- More parallel requests could trigger third-party throttling instead of speeding up the work.
- A failed lookup or malformed record needed attention without holding up the remaining data.
What We Delivered
We built scheduled pipelines for property, weather, and air-quality enrichment. The property flow geocodes service locations, retrieves property attributes, and loads the results into a spatially searchable database. Apache Airflow coordinates pipeline schedules and dependencies. Within each run, a shared execution pattern controls API-bound parallel work, isolates failures, and supports repeatable deployment.
The Solution
Four Problems, and What We Built for Each.
01 / 04
A service address
An address, and little else to go on The Problem
Sales Had an Address, and Little Else
Sales and marketing knew where customers were, but not enough about those places to decide where and when to reach out.
A service address
- Location
- Building age
- Building size
- Construction
- Weather
- Air quality
The same address, with around 60 property details What We Built
Every Address Becomes a Place Sales Can Understand
Each location gains coordinates, property details, and weather context. The project owner reports a 20% increase in sales revenue following the work.
How We Built It
Addresses Turned into LocationsPythonEach service address gets coordinates it can be mapped with
- 02
Property Details AddedAround 60 attributes per record, such as building age, size, and construction
Searchable by PlaceAzure Database for PostgreSQLStored so teams can search and group locations on a map
Weather and air-quality feeds add context for when an outreach conversation may be relevant.
02 / 04
A data refresh for sales
5–6 hours before sales could use it
Bars drawn to scale: five and a half hours beside thirty minutes The Problem
Fresh Data Took Most of a Working Day
Each refresh pulled millions of records from outside sources, and a run took 5–6 hours before sales could use the result.
A data refresh for sales
Ready in 30 minutes
Bars drawn to scale: five and a half hours beside thirty minutes What We Built
A Refresh in 30 Minutes
The rebuilt pipelines run on a schedule, skip what is already done, and work in parallel. Processing fell from 5–6 hours to 30 minutes.
How We Built It
Scheduled, in the Right OrderApache AirflowProperty, weather, and air-quality runs are coordinated together
- 02
Only What Is NewLocations that are already enriched are not looked up again
- 03
Many Lookups at OnceWork is done in parallel instead of one record at a time
Runs on Azure, with the infrastructure written as code in Terraform.
03 / 04
What the data provider allowsEverything asked for at once, then slowed down The Problem
Asking Faster Only Made the Data Providers Slow Down
Sending more requests at once could trigger the providers’ limits, which slowed the work instead of speeding it up.
What the data provider allowsSteady batches, kept within the limit What We Built
Steady Batches, Paced to Each Provider
Lookups are split into batches sized to what each provider allows, and the next batch starts only when it can be taken.
How We Built It
- 01
Work Split into BatchesSized to each data provider’s limits
- 02
One Overall PaceThe next batch begins only when the provider can take it
Parallel within the LimitPythonWorkers handle different records at the same time, under that pace
04 / 04
One run, and one bad record can hold it up
- Address matched
- Address not found
- Address matched
The Problem
One Bad Address Could Hold Up All the Others
A failed lookup or a malformed record needed attention, without stopping the rest of the data.
- Address matchedContinues
- Address not foundSet aside to retry
- Address matchedContinues
What We Built
Problems Set Aside, and the Rest Carries On
Unmatched addresses and failed records are separated for investigation, with what is needed to retry them. Successful work continues.
How We Built It
- 01
Failures SeparatedKept with their inputs and the reason, ready to retry
- 02
The Rest ContinuesSuccessful records are not held up
Visible at Every StageDatadogThe team can inspect row counts and failures
Changes to the pipelines are deployed through GitHub Actions.
How a Refresh Runs Now
Before: Long Refresh Cycles
- Service Locations + Third-Party Data
Millions of Records Pulled Each Time
Addresses without Coordinates or Details
Hours before Sales Could Use the Result
Regular data requests involved millions of records. Addresses also needed coordinates and property details before they could support useful segmentation.
Data Refresh5–6 Hours
After: Scheduled, Coordinated Pipelines
01 · Scheduled Run
Select and Stage
Extract eligible records into staged data files
- 02
The Pipeline
Plan and Pace
Split work into batches aligned with provider limits
03 · The Pipeline
Enrich and Validate
Run lookups in parallel and isolate failed records
04 · Sales & Marketing
Load and Use
Make enriched data available for analysis and outreach
Apache Airflow coordinates the property, weather, and air-quality workflows. Each keeps its own source clients, schemas, and pacing settings.
Data Refresh30 Minutes
Technologies Used
Python
Data Processing & API Integration
Apache Airflow
Pipeline Scheduling & Coordination
Microsoft Azure
Pipeline Orchestration, Compute & Storage
Azure Database for PostgreSQL
Property Data & Spatial Analysis with PostGIS
Terraform
Repeatable Cloud Infrastructure
GitHub Actions
Version Control & Deployment
Datadog
Pipeline Monitoring
Azure Implementation Approach
- 01
Turn Addresses into Useful Property Context
The property pipeline adds coordinates and attributes such as building age, size, and construction details. That gives teams a more useful basis for grouping locations and understanding the properties they could serve.
Technical Detail: Turn Addresses into Useful Property Context
ArcGIS supplies geocoding and Realie supplies property records. In the Azure adaptation, Azure Database for PostgreSQL with PostGIS stores the enriched locations and geometry. The flow rejects unsuitable address matches and checks that a unit identifier is not lost when the source address contains one.
- 02
Avoid Repeating Completed Property Lookups
The property extraction stage selects locations that have not yet been enriched. Completed records do not need the same lookup again on every scheduled run.
Technical Detail: Avoid Repeating Completed Property Lookups
The Azure adaptation stages extracted database pages as Parquet files in Azure Blob Storage. This selection rule applies to the property pipeline; it should not be interpreted as skipping necessary weather or air-quality refreshes. Those pipelines have separate inputs and refresh requirements.
- 03
Match Parallel Work to the Provider’s Capacity
The system divides lookups into batches and controls when the next batch can begin. Workers can process different records at the same time while the orchestrator paces the overall request volume.
Technical Detail: Match Parallel Work to the Provider’s Capacity
The Azure design uses Durable Functions to coordinate the stages, with bounded fan-out to activity functions for API lookups. Durable timers pace successive batches. Azure Container Apps Jobs handle longer extraction and loading tasks. Batch size and concurrency must be tuned for the Azure runtime and each provider; the original worker settings are not assumed to transfer unchanged.
- 04
Keep Failed Records Visible and Recoverable
Unmatched addresses and failed records are separated for investigation. Successful work can continue, and operators retain the inputs and error context needed to retry the affected data.
Technical Detail: Keep Failed Records Visible and Recoverable
In the Azure design, stages write date-partitioned Parquet files to Azure Blob Storage, including failure records. Loading uses bulk upserts with a row-by-row fallback when a chunk fails. Provider keys and dependency checks help maintain record relationships. Empty input is handled as a normal completion.
- 05
Bring Weather Context into Sales Planning
The same pipeline approach supports weather and air-quality enrichment. Together with property and service-location information, these feeds help sales and marketing decide where and when an outreach conversation may be relevant.
Technical Detail: Bring Weather Context into Sales Planning
The supplied account describes separate Google Weather and Air Quality API pipelines using the same staged-file and batch-planning pattern. Their clients, output schemas, and API pacing differ. No weather prediction model, automatic campaign execution, or precise equipment inventory inferred from property metadata is claimed.
- 06
Make the Refresh Process Operable
Apache Airflow schedules the workflows and coordinates their dependencies. Versioned infrastructure and stage-level monitoring make the pipelines a repeatable operating process. Teams can inspect row counts and failures rather than reconstructing a run from scattered logs.
Technical Detail: Make the Refresh Process Operable
Airflow DAGs define schedules, task dependencies, and run status, initiating the Azure execution workflows. Durable Functions handle the per-run parallel activities and batch pacing. Terraform provisions resources, and GitHub Actions deploys the functions and jobs, with job images stored in Azure Container Registry. Azure Key Vault holds API credentials. Azure Monitor and Application Insights provide runtime telemetry, with Datadog used for centralized monitoring. Non-production lookup clients remain mocked. Hosting plans and job settings determine idle and execution costs.
The Results
Faster Refreshes. Richer Data for Sales.
Data Pipeline Processing Time
- Previous Pipeline Runs
- 5–6 Hrs
- After the Pipeline Changes
- 30 Min
Updated Data Available Earlier
The project owner reports the shorter runtime alongside richer property data and a 20% increase in sales revenue.
What Was Delivered
Scheduled property, weather, and air-quality pipelines, with provider-aware batching, validated address matching, staged data files, recoverable failures, and repeatable deployment. Property enrichment adds approximately 60 attributes per record.
20% Sales Revenue Increase
The project owner reports a 20% revenue increase following the work. Sales and marketing used enriched location information and weather context to focus outreach around relevant service needs. Pipeline runs fell from 5–6 hours to 30 minutes.
Runtime and revenue figures were supplied by the project owner. No measurement period, comparison cohort, or revenue attribution method was provided, so the revenue change is reported alongside the work rather than established as its sole effect. The attribute count comes from the supplied engineering account. Azure service mappings are an architecture adaptation, not evidence that these results were measured on Azure.
What Changed for the People Doing the Work
- Sales
- Use richer property and location context to focus conversations on relevant service opportunities.
- Marketing
- Bring weather and air-quality context into the timing and geographic focus of outreach.
- Data Teams
- Run a common enrichment pattern across multiple sources, with failures available for investigation.
- Operations
- Monitor scheduled processing and recover affected records without losing the surrounding run context.
What This Project Reinforced
Freshness and Context Belong Together
A faster pipeline is useful when it delivers the attributes and current information the business needs to act on.
Design Around the Source Limits
Provider quotas shape useful throughput. More workers alone do not make the entire process faster.
A Rejected Match Is Worth Investigating
An incomplete or incorrect property match can mislead downstream analysis. Keep the source and reason available for review.
Reuse the Pattern, Respect the Differences
The orchestration approach can carry across data domains while source requirements, refresh rules, and validation remain specific to each one.