Inside a Flat Market
A Magnifier on Melbourne

A data tale from AURIN, read through the Domain residential listings endpoint.

📖 Story

Property data platform · accessed by AURIN · Request access (EOI)

Melbourne’s median house price barely moved for years. Case closed?

Not quite. A city-level median is an average, and an average hides as much as it shows. This is the second of two short data tales built on the property data platform accessed by AURIN. The companion tale, “The Great Divergence”, used the suburb performance statistics endpoint to map how Australia’s cities pulled apart. Here we answer the question that endpoint could not: is Melbourne’s flatness real, or is something more interesting happening inside it?

To find out we switch to the residential listings endpoint, which returns individual property data - each record carrying attributes such as bedroom count and coordinates. That granularity lets us drop below the city average and compare two Melbourne suburbs street by street: Hawthorn (inner, premium) and Frankston (outer, affordable).

A tale of two suburbs

The Domain API also gives us a more fine-grained lens over individual cities. Consider a market where expensive suburbs are going nowhere while affordable ones climb hard. The city-level median would look unremarkable. But the story underneath would be a market pulling apart at both ends. That is Melbourne.

The suburb performance statistics endpoint used in the companion tale is powerful precisely because of its breadth: a handful of API calls return a decade of data across every major city. Its limitation, however, is fundamental. It reports what a whole suburb did on average, aggregated over time. It cannot reveal what is happening at different ends of the price spectrum within that suburb, or whether those ends are moving in opposite directions across the same period.

The statistics endpoint told us Melbourne is flat. What it cannot tell us is that “flat” may be the average of two opposite movements, one at the expensive end and one at the affordable end, cancelling each other out.

To look inside Melbourne’s flatness, we shift to a different kind of data: individual property data. Every record carries its suburb, property type and bedroom count. This is far more granular than suburb-level aggregates, and far more expensive to collect, but it is the only way to test whether Melbourne’s flat median represents a genuine pause or an artefact of averaging two very different markets into one number.

A note on the data

These charts read the property data Domain recorded in Hawthorn and Frankston since 2018 (and, for days on market, the suburb performance statistics endpoint). What each chart includes differs, so it is worth stating up front:

• the property map and the auction- and bedroom-mix charts use all sold listings, of any property type;
• the like-for-like growth chart restricts to 4-bedroom houses and townhouses, so the two suburbs are compared on the same kind of home;
• days on market comes from the statistics endpoint, houses only;
• the suburb medians quoted in the text, where a single figure describes a whole suburb, also come from the statistics endpoint (House, all bedroom counts), so the two suburbs are quoted on one consistent measure.

Because we are now working suburb by suburb rather than city-wide, some of the finer cuts rest on only a handful of sales per quarter - where that matters, it is flagged at the chart.

Definitions: what counts as a “house”?

• House - Domain’s category, and broader than the word suggests. Alongside detached houses it counts townhouses, terraces, duplexes and semi-detached homes.
• Unit - apartments, flats, studios and villas. Vacant land is excluded from both.
• Median sold price - the middle sale in a suburb for that quarter, not the average. Every bedroom count is pooled, so a two-bedroom and a five-bedroom sale count equally.
• The city line - each suburb gives its own median for the quarter, and the city line is the middle value of those. A middle of middles.
• % change since Q1 2018 - each city indexed to its own Q1 2018 price, which counts as 0%, so cities are compared on growth rather than price level.
• Withheld prices - Domain returns some sold listings with no price. Through 2018-2024 this affected about 6% of Hawthorn sales and 1% of Frankston’s; by 2026 it reaches 29% in Hawthorn. Those sales are missing from every median here, and withholding is probably not random, since a disappointing result is the more likely one to be held back. Treat the most recent quarters as provisional.

The two suburbs · Like for like · Days on market · How you sell · Zoom further · Get the data

Hawthorn and Frankston: same city, 37 kilometres apart, opposite stories The map on the right plots property data from our dataset for two Melbourne suburbs: Hawthorn (pink-purple dots) and Frankston (blue dots). Each dot is one property, and darker colours mean higher prices within that suburb’s own range.Notice how differently the two suburbs sit on the map. Hawthorn’s properties cluster tightly in a small, dense patch of inner Melbourne. Frankston’s spread across a much wider footprint at the city’s edge. The geography itself signals different buyer types: professionals competing for scarce inner-city homes in one case, families and first-home buyers seeking space and affordability in the other.

Hawthorn sits 6km east of the Melbourne CBD. Leafy streets, Victorian terraces, elite schools. It reads as a high-status, established heritage enclave, and buyers are established professionals and upsizers competing for a constrained supply of character homes. The median house price was $2.37M in Q1 2018. This is a market shaped by sentiment and interest rates, and it has spent eight years going nowhere in particular: up around 13%, while the city around it moved.

Frankston sits 40km south-east, at the end of the Frankston line. The suburb carries a mixed reputation, historically freighted with outer-suburban stereotypes and a perception of higher crime. Buyers are generally first-home owners, sea-changers, and families priced out of inner Melbourne. The median was $603K in Q1 2018. This market is driven by structural demand: population spillover, affordability relative to the city, and the pull of a coastal lifestyle at an accessible price. It is up around 43% since Q1 2018.

Same property. Same bedroom count. Opposite directions Hawthorn and Frankston represent two different relationships with interest rates, income levels and borrowing capacity. A like-for-like comparison on 4-bedroom houses and townhouses strips out most dwelling-type noise and asks a cleaner question: when the same macro forces hit the same product in two very different price brackets, do they land the same way?The short answer is “no”. Frankston sits around 35% above its Q1 2018 base, having peaked in the mid-forties, a climb steady enough to read as a trend on roughly 30 sales a quarter. Hawthorn’s quarterly median swings between -22% and +120% against the same base, at times on as few as two sales.

A six-to-twelve-month smoothing would calm Hawthorn’s line considerably, at the cost of hiding how thin the underlying data is. Frankston’s line, built on far more sales, is the sturdier of the two, and the only one of the pair from which a rate of change can honestly be quoted. Its gain means the suburb that once offered genuine entry-level access to Melbourne has become significantly more expensive in absolute terms. The city has not found a more affordable floor; it has simply shifted where the pressure lands.

Beyond price: the quieter indicators Price is only the headline, and in a suburb as thinly traded as Hawthorn it is the least stable thing we can measure. Underneath it sits a handful of quieter indicators that often tell a steadier story than the median can: days on market, the auction-versus-private-treaty mix, and the bedroom composition of sales. Each adds a dimension the price charts alone cannot capture, from how buyers behave in each market to whether a trend is broad-based or driven by a shift in what is being sold.

In principle days on market moves before price does, because price indices are backward-looking, capturing only deals that have already closed, while it reflects what is happening right now. The gap in level is the reliable part. A Frankston house takes about three weeks to sell, a median of 21 days across the period; a Hawthorn house takes about five, a median of 35. Hawthorn is the slower of the two in 27 of 31 quarters, and it drifts upward, which may reflect the small pool of buyers who can bid at that level. Read that drift as a direction, not a rate: it rests on 30 to 60 sales a quarter, and one quarter in early 2026 reads 141 days before snapping straight back to 32. Days on market (DOM): how long a property sits listed before a buyer commits. In a hot market properties sell quickly; when demand softens they linger.

How you sell and which market you are in In a strong market vendors run auctions: buyers compete, bids exceed reserve, the sale is unconditional. When confidence softens they shift to private treaty, testing the market quietly and negotiating terms.The chart splits every sale three ways: cleared under the hammer, sold before auction day, and negotiated privately. In Hawthorn it drifts down roughly ten points, from the high forties before 2022 to the high thirties after. The growing private-treaty share suggests buyer appetite at the premium end has thinned. Frankston tells the opposite story: private treaty is the default for first-home buyers, who need conditional contracts to arrange finance, and its mix has held steady, moving less than a point.

What if we could zoom in even further? Breaking the data down by bedroom count tells us whether what sells has changed, not just what it sells for. If the median moves because the mix shifted rather than because homes changed price, that is a composition effect, not a price signal. Reading the charts together tells you which.In Hawthorn the mix itself moved, and it moved upward. Four- and five-bedroom homes were 35% of house and townhouse sales in 2018 and around half from 2022 on, so the suburb has been selling larger properties. Their median held between $3.5M and $3.9M every year from 2022 to 2025, rather than falling away. Both cut against reading a decline into Hawthorn’s median: when the mix drifts to bigger houses, the median rises even if no home got dearer. Whatever pressure the premium end is under, this chart does not show it.

Frankston shows the opposite kind of stability. Three-bedroom houses are the largest segment throughout, at close to 60% of sales, and the mix barely shifts. Its price growth runs at a similar rate across every bedroom tier, so Frankston’s gains are real and broad-based rather than an artefact of what happened to sell. Every segment moving together is, in its own way, the less forgiving outcome for buyers who were hoping the affordable end might stay within reach. The finest cuts need the most care: splitting one suburb by bedroom count leaves only a handful of sales per quarter - enough to suggest a direction, not to prove one.

In closing: what the property data showed

The flat median was two unlike stories averaged together. Hawthorn’s premium market went sideways, up 13% in eight years, while Frankston’s affordable outer ring rose 43%. Melbourne’s city line sits between them at +18%, the weakest of the eight cities in the companion tale, and it describes neither suburb. Only suburb-level data pulls them apart.

Price was the least readable measure of the three. Hawthorn’s median sold price swings so hard from quarter to quarter that no rate of change can be taken from it, while how long a sale takes and how it is conducted both separate the two suburbs cleanly and consistently. When a price series is this thin, the behavioural measures carry more information than the prices do.

The finest cuts need the most care. Splitting a single suburb by bedroom count leaves only a handful of sales per quarter - enough to suggest a direction, not to prove one. Where the sample is thin, the tale flags it rather than over-claiming.

One endpoint, one kind of question

The residential listings endpoint is built for depth: each record carries property data - type, bedrooms and coordinates - which is what made the Hawthorn vs Frankston comparison possible. The cost is higher - more API calls, more data handling, a real credit budget - but some questions cannot be answered any other way. Paired with the statistics endpoint from the companion tale, it supports a natural research sequence: start broad, spot an anomaly in the averages, then pull the property data to look inside it.

Want to use this data in your own research?

The Domain data used throughout this tale is available to approved Australian researchers through AURIN. Access requires an expression of interest, an institutional affiliation (a university or research organisation login) and signing a data agreement, managed at data.aurin.org.au. Once signed, your account receives 5,000 API credits per month, resetting automatically. That allowance is enough for the kind of targeted analysis shown here without exhausting your budget, provided you design your queries before running them rather than exploring open-endedly.

Materials, reference and access

Training materials: step-by-step guides covering setup, credential management, credit budgeting, known API quirks, and hands-on Jupyter notebooks for the most common query patterns are available at github.com/AURIN-OFFICE/domain_public.

API endpoint reference and metadata: a complete listing of available endpoints with parameters, response fields, coverage notes, and known data limitations is at the domain_public repository.

Request access: Submit your expression of interest for the Domain API at aurin.org.au/domain and retrieve your credentials at data.aurin.org.au.

The code patterns in the training notebooks are written to be adapted directly to your own study area, time window, and research question. If your work involves Australian property markets, this is the fastest route from a research question to clean, queryable, and well-documented data.

This story as source: the complete source behind this analysis, code and narrative together, is on GitHub at datatale/. Every chart on this page is reproducible from it, and the “The Code” tab above lists the code for each figure.

💻 The Code

Every chart in the Story tab is built from the code below, shown for reading only (these blocks are not re-executed). Order: the one-time data acquisition first, which the Story itself never runs, then setup, then the code behind each visual in Story order.

Everything needed to reproduce this tale is here, and the blocks run top to bottom on their own, from the datatale/ directory. The Quarto document behind this page, code and narrative together, lives in the datatale/ folder of the public repository. One thing to know before running it:

• You need your own AURIN credentials. The Domain data is licensed, not open: the download blocks read AURIN_USERNAME and AURIN_PASSWORD from a .env at the repo root and call the AURIN proxy. Without them the code loads whatever is already cached and nothing else. The training materials walk through requesting access and setting those credentials up, step by step.

Setup and data acquisition

Manifest and download helpers

import sys, json
from pathlib import Path
from urllib.parse import quote
from datetime import datetime, timezone

# Import AURIN proxy helpers from the project root (domain_api/utils.py)
_ROOT = Path(".").resolve().parent
sys.path.insert(0, str(_ROOT))
from utils import PROXY_BASE, APICallTracker

_DATATALE = Path(".")          # renders from datatale/
_MANIFEST    = _DATATALE / "manifest.json"
_STATS_BASE  = f"{PROXY_BASE}/v2/suburbPerformanceStatistics"


# ── Manifest helpers (resume-safe: already-downloaded files are skipped) ────

def _load_manifest():
    return json.loads(_MANIFEST.read_text()) if _MANIFEST.exists() else {"captures": []}

def _save_manifest(m):
    _MANIFEST.write_text(json.dumps(m, indent=2))

def _already_captured(m, rel):
    return str(rel) in {e["file"] for e in m["captures"]}

def _record_capture(m, rel, endpoint, params, status, section, empty=False):
    m["captures"].append({
        "file":             str(rel),
        "endpoint":         endpoint,
        "params":           params,
        "captured_at":      datetime.now(timezone.utc).isoformat(),
        "http_status":      status,
        "empty_response":   empty,
        "datatale_section": section,
    })
    _save_manifest(m)

def _save_json(rel, data):
    dest = _DATATALE / rel
    dest.parent.mkdir(parents=True, exist_ok=True)
    dest.write_text(json.dumps(data, indent=2))

def _save_probe(rel, response):
    _save_json(rel, {
        "x_total_count": int(response.headers.get("X-Total-Count", 0)),
        "body":          response.json() if response.text.strip() else {},
    })

Download suburb performance statistics

Show code

# Download suburb performance statistics for every city used in the tales.
# Each suburb+category pair becomes one JSON file in statistics/{city}/.
# Files already recorded in the manifest are skipped automatically.

STATS_SUBURBS = [
    # ── Melbourne (VIC) ──────────────────────────────────────────────────────
    # Premium (10)
    {"city": "melbourne", "suburb": "Toorak",       "postcode": "3142", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Hawthorn",     "postcode": "3122", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Brighton",     "postcode": "3186", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Kew",          "postcode": "3101", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Camberwell",   "postcode": "3124", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Canterbury",   "postcode": "3126", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Malvern",      "postcode": "3144", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "South Yarra",  "postcode": "3141", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Balwyn",       "postcode": "3103", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Armadale",     "postcode": "3143", "state": "VIC", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "melbourne", "suburb": "Carlton",      "postcode": "3053", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Fitzroy",      "postcode": "3065", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Preston",      "postcode": "3072", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Brunswick",    "postcode": "3056", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Richmond",     "postcode": "3121", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Box Hill",     "postcode": "3128", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Frankston",    "postcode": "3199", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Footscray",    "postcode": "3011", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Coburg",       "postcode": "3058", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Ringwood",     "postcode": "3134", "state": "VIC", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "melbourne", "suburb": "Reservoir",    "postcode": "3073", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Tarneit",      "postcode": "3029", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Point Cook",   "postcode": "3030", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Dandenong",    "postcode": "3175", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Broadmeadows", "postcode": "3047", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Melton",       "postcode": "3337", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Springvale",   "postcode": "3171", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Cranbourne",   "postcode": "3977", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Doveton",      "postcode": "3177", "state": "VIC", "categories": ["House", "Unit"]},
    {"city": "melbourne", "suburb": "Pakenham",     "postcode": "3810", "state": "VIC", "categories": ["House", "Unit"]},
    # ── Sydney (NSW) ─────────────────────────────────────────────────────────
    # Premium (10)
    {"city": "sydney", "suburb": "Vaucluse",     "postcode": "2030", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Mosman",       "postcode": "2088", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Neutral Bay",  "postcode": "2089", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Paddington",   "postcode": "2021", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Double Bay",   "postcode": "2028", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Rose Bay",     "postcode": "2029", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Balmain",      "postcode": "2041", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Pymble",       "postcode": "2073", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Killara",      "postcode": "2071", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Gordon",       "postcode": "2072", "state": "NSW", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "sydney", "suburb": "Surry Hills",  "postcode": "2010", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Newtown",      "postcode": "2042", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Chatswood",    "postcode": "2067", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Parramatta",   "postcode": "2150", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Leichhardt",   "postcode": "2040", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Glebe",        "postcode": "2037", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Randwick",     "postcode": "2031", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Burwood",      "postcode": "2134", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Ryde",         "postcode": "2112", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Hornsby",      "postcode": "2077", "state": "NSW", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "sydney", "suburb": "Penrith",      "postcode": "2750", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Tregear",      "postcode": "2770", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Campbelltown", "postcode": "2560", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Fairfield",    "postcode": "2165", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Liverpool",    "postcode": "2170", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Blacktown",    "postcode": "2148", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Bankstown",    "postcode": "2200", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Auburn",       "postcode": "2144", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Cabramatta",   "postcode": "2166", "state": "NSW", "categories": ["House", "Unit"]},
    {"city": "sydney", "suburb": "Kingswood",    "postcode": "2747", "state": "NSW", "categories": ["House", "Unit"]},
    # ── Brisbane (QLD) ───────────────────────────────────────────────────────
    # Premium (10)
    {"city": "brisbane", "suburb": "New Farm",      "postcode": "4005", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Teneriffe",     "postcode": "4005", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Ascot",         "postcode": "4007", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Hamilton",      "postcode": "4007", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Paddington",    "postcode": "4064", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Bulimba",       "postcode": "4171", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Hawthorne",     "postcode": "4171", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "St Lucia",      "postcode": "4067", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Indooroopilly", "postcode": "4068", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Toowong",       "postcode": "4066", "state": "QLD", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "brisbane", "suburb": "West End",      "postcode": "4101", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Chermside",     "postcode": "4032", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Wynnum",        "postcode": "4178", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Nundah",        "postcode": "4012", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Stafford",      "postcode": "4053", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Sunnybank",     "postcode": "4109", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Coorparoo",     "postcode": "4151", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Moorooka",      "postcode": "4105", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Zillmere",      "postcode": "4034", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Carina",        "postcode": "4152", "state": "QLD", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "brisbane", "suburb": "Logan Central", "postcode": "4114", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Woodridge",     "postcode": "4114", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Inala",         "postcode": "4077", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Deception Bay", "postcode": "4508", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Redcliffe",     "postcode": "4020", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Ipswich",       "postcode": "4305", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Springfield",   "postcode": "4300", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "North Lakes",   "postcode": "4509", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Caboolture",    "postcode": "4510", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "brisbane", "suburb": "Narangba",      "postcode": "4504", "state": "QLD", "categories": ["House", "Unit"]},
    # ── Perth (WA) ───────────────────────────────────────────────────────────
    # Premium (10)
    {"city": "perth", "suburb": "Dalkeith",         "postcode": "6009", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Peppermint Grove", "postcode": "6011", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Cottesloe",        "postcode": "6011", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Nedlands",         "postcode": "6009", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Claremont",        "postcode": "6010", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Subiaco",          "postcode": "6008", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Mount Lawley",     "postcode": "6050", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Bicton",           "postcode": "6157", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Floreat",          "postcode": "6014", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Mosman Park",      "postcode": "6012", "state": "WA", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "perth", "suburb": "Fremantle",        "postcode": "6160", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Victoria Park",    "postcode": "6100", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Bayswater",        "postcode": "6053", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Bassendean",       "postcode": "6054", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Midland",          "postcode": "6056", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Belmont",          "postcode": "6104", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Maylands",         "postcode": "6051", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Morley",           "postcode": "6062", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Canning Vale",     "postcode": "6155", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "East Fremantle",   "postcode": "6158", "state": "WA", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "perth", "suburb": "Armadale",         "postcode": "6112", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Medina",           "postcode": "6167", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Gosnells",         "postcode": "6110", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Kelmscott",        "postcode": "6111", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Rockingham",       "postcode": "6168", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Mandurah",         "postcode": "6210", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Girrawheen",       "postcode": "6064", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Thornlie",         "postcode": "6108", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Maddington",       "postcode": "6109", "state": "WA", "categories": ["House", "Unit"]},
    {"city": "perth", "suburb": "Kenwick",          "postcode": "6107", "state": "WA", "categories": ["House", "Unit"]},
    # ── Adelaide (SA) ────────────────────────────────────────────────────────
    # Premium (10)
    {"city": "adelaide", "suburb": "Medindie",        "postcode": "5081", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Norwood",         "postcode": "5067", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Burnside",        "postcode": "5066", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Unley",           "postcode": "5061", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Glenelg",         "postcode": "5045", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Toorak Gardens",  "postcode": "5065", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Walkerville",     "postcode": "5081", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Springfield",     "postcode": "5062", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Glen Osmond",     "postcode": "5064", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Malvern",         "postcode": "5061", "state": "SA", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "adelaide", "suburb": "Prospect",        "postcode": "5082", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Magill",          "postcode": "5072", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Tranmere",        "postcode": "5073", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Goodwood",        "postcode": "5034", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Henley Beach",    "postcode": "5022", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Glenunga",        "postcode": "5064", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Hawthorn",        "postcode": "5062", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Parkside",        "postcode": "5063", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Plympton",        "postcode": "5038", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Torrensville",    "postcode": "5031", "state": "SA", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "adelaide", "suburb": "Elizabeth",       "postcode": "5112", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Morphett Vale",   "postcode": "5162", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Davoren Park",    "postcode": "5113", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Salisbury",       "postcode": "5108", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Parafield Gardens","postcode": "5107", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Smithfield",      "postcode": "5114", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Christies Beach", "postcode": "5165", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Old Noarlunga",   "postcode": "5168", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Seaford",         "postcode": "5169", "state": "SA", "categories": ["House", "Unit"]},
    {"city": "adelaide", "suburb": "Andrews Farm",    "postcode": "5114", "state": "SA", "categories": ["House", "Unit"]},
    # ── Sunshine Coast (QLD) ─────────────────────────────────────────────────
    # Premium (10)
    {"city": "sunshine_coast", "suburb": "Noosa Heads",      "postcode": "4567", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Noosaville",       "postcode": "4566", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Peregian Beach",   "postcode": "4573", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Buderim",          "postcode": "4556", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Mooloolaba",       "postcode": "4557", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Bokarina",         "postcode": "4575", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Tewantin",         "postcode": "4565", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Coolum Beach",     "postcode": "4573", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Sunshine Beach",   "postcode": "4567", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Peregian Springs", "postcode": "4573", "state": "QLD", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "sunshine_coast", "suburb": "Maroochydore",     "postcode": "4558", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Caloundra",        "postcode": "4551", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Alexandra Headland","postcode": "4572", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Nambour",          "postcode": "4560", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Sippy Downs",      "postcode": "4556", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Mountain Creek",   "postcode": "4557", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Bli Bli",          "postcode": "4560", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Palmview",         "postcode": "4553", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Wurtulla",         "postcode": "4575", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Birtinya",         "postcode": "4575", "state": "QLD", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "sunshine_coast", "suburb": "Landsborough",     "postcode": "4550", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Woombye",          "postcode": "4559", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Maleny",           "postcode": "4552", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Yandina",          "postcode": "4561", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Eumundi",          "postcode": "4562", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Pomona",           "postcode": "4568", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Cooroy",           "postcode": "4563", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Currimundi",       "postcode": "4551", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Beerwah",          "postcode": "4519", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "sunshine_coast", "suburb": "Glass House Mountains", "postcode": "4518", "state": "QLD", "categories": ["House", "Unit"]},
    # ── Gold Coast (QLD) ─────────────────────────────────────────────────────
    # Premium (10)
    {"city": "gold_coast", "suburb": "Mermaid Beach",       "postcode": "4218", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Broadbeach Waters",   "postcode": "4218", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Hope Island",         "postcode": "4212", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Mermaid Waters",      "postcode": "4218", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Burleigh Heads",      "postcode": "4220", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Palm Beach",          "postcode": "4221", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Currumbin",           "postcode": "4223", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Clear Island Waters", "postcode": "4226", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Broadbeach",          "postcode": "4218", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Tallebudgera",        "postcode": "4228", "state": "QLD", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "gold_coast", "suburb": "Surfers Paradise",    "postcode": "4217", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Southport",           "postcode": "4215", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Varsity Lakes",       "postcode": "4227", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Robina",              "postcode": "4226", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Nerang",              "postcode": "4211", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Bundall",             "postcode": "4217", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Ashmore",             "postcode": "4214", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Runaway Bay",         "postcode": "4216", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Biggera Waters",      "postcode": "4216", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Benowa",              "postcode": "4217", "state": "QLD", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "gold_coast", "suburb": "Coomera",             "postcode": "4209", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Pimpama",             "postcode": "4209", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Ormeau",              "postcode": "4208", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Oxenford",            "postcode": "4210", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Helensvale",          "postcode": "4212", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Arundel",             "postcode": "4214", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Labrador",            "postcode": "4215", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Mudgeeraba",          "postcode": "4213", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Molendinar",          "postcode": "4214", "state": "QLD", "categories": ["House", "Unit"]},
    {"city": "gold_coast", "suburb": "Worongary",           "postcode": "4213", "state": "QLD", "categories": ["House", "Unit"]},
    # ── Canberra (ACT) ───────────────────────────────────────────────────────
    # Premium (10)
    {"city": "canberra", "suburb": "Forrest",     "postcode": "2603", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Griffith",    "postcode": "2603", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Red Hill",    "postcode": "2603", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Deakin",      "postcode": "2600", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Yarralumla",  "postcode": "2600", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Barton",      "postcode": "2600", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "O'Malley",   "postcode": "2606", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Kingston",    "postcode": "2604", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Narrabundah", "postcode": "2604", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Phillip",     "postcode": "2606", "state": "ACT", "categories": ["House", "Unit"]},
    # Mid-range (10)
    {"city": "canberra", "suburb": "Kambah",      "postcode": "2902", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Wanniassa",   "postcode": "2903", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Garran",      "postcode": "2605", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Curtin",      "postcode": "2605", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Hughes",      "postcode": "2605", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Weston",      "postcode": "2611", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Cook",        "postcode": "2614", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Chifley",     "postcode": "2606", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Torrens",     "postcode": "2607", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Pearce",      "postcode": "2607", "state": "ACT", "categories": ["House", "Unit"]},
    # Affordable (10)
    {"city": "canberra", "suburb": "Charnwood",   "postcode": "2615", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Banks",       "postcode": "2906", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Holt",        "postcode": "2615", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Macgregor",   "postcode": "2615", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Latham",      "postcode": "2615", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Fraser",      "postcode": "2615", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Florey",      "postcode": "2615", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Theodore",    "postcode": "2905", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Calwell",     "postcode": "2905", "state": "ACT", "categories": ["House", "Unit"]},
    {"city": "canberra", "suburb": "Gilmore",     "postcode": "2905", "state": "ACT", "categories": ["House", "Unit"]},
]

# One-time retrieval, off by default so this block can be read or run without
# spending credits. Files already recorded in the manifest are skipped, so a
# re-run on a populated cache is free. Left unguarded it fires live calls: a
# verification run on 2026-09-04 spent 2 credits and hit 401s mid-resume.
RUN_DOWNLOAD = False

if RUN_DOWNLOAD:
    _tracker_stats = APICallTracker()
    _manifest      = _load_manifest()
    _skipped       = 0

    print("Downloading suburb performance statistics...")
    print("(Files already on disk are skipped automatically)\n")

    for _entry in STATS_SUBURBS:
        _suburb   = _entry["suburb"]
        _postcode = _entry["postcode"]
        _state    = _entry["state"]
        _city     = _entry["city"]

        for _category in _entry["categories"]:

            _slug     = _suburb.lower().replace(" ", "_")
            _rel_path = f"statistics/{_city}/{_slug}_{_category.lower()}.json"

            if _already_captured(_manifest, _rel_path):
                _skipped += 1
                continue

            _url    = f"{_STATS_BASE}/{_state}/{quote(_suburb, safe='')}/{_postcode}"
            _params = {
                "propertyCategory":               _category,
                "periodSize":                     "quarters",
                "totalPeriods":                   45,
                "startingPeriodRelativeToCurrent": 0,
            }
            _r     = _tracker_stats.get(_url, params=_params)
            _empty = _r.status_code == 200 and not _r.text.strip()

            if _r.status_code == 200 and not _empty:
                _save_json(_rel_path, _r.json())
            elif _empty:
                _save_json(_rel_path, {})
                print(f"  WARNING: empty response for {_suburb} {_category}")
            else:
                print(f"  ERROR {_r.status_code}: {_suburb} {_category} -- {_r.text[:100]}")

            _record_capture(_manifest, _rel_path,
                            endpoint="GET /v2/suburbPerformanceStatistics",
                            params={"suburb": _suburb, "postcode": _postcode,
                                    "state": _state, "propertyCategory": _category,
                                    "totalPeriods": 45},
                            status=_r.status_code,
                            section="part1_statistics",
                            empty=_empty)
            print(f"  {'OK' if not _empty else 'EMPTY':6}  statistics/{_city}/{_slug}_{_category.lower()}.json"
                  f"  [{_tracker_stats.total} credits used]")

    _tracker_stats.checkpoint("Statistics download")
    print(f"\nStatistics done. {_skipped} files already on disk, {_tracker_stats.total} credits spent.")
    _tracker_stats.summary()

Download sold listings for the two suburbs

Show code

# Download sold listing records for the two Melbourne suburbs in this tale:
# Hawthorn (inner premium) and Frankston (outer affordable).
#
# The Domain API caps results at pageNumber * pageSize <= 1000. This paginator
# handles that by advancing the listedSince cursor whenever a 400 is received,
# restarting from page 1 with the new cursor. Progress is checkpointed to
# fetch_state.json inside each suburb directory so an interrupted run resumes.

_LISTINGS_TARGETS = [
    {
        "role":       "highvalue_melbourne",
        "suburb":     "Hawthorn",
        "postcode":   "3122",
        "state":      "VIC",
        "start_date": "2018-01-01",
        "dir":        "listings/part2_highvalue/melbourne_hawthorn",
    },
    {
        "role":       "midring_melbourne",
        "suburb":     "Frankston",
        "postcode":   "3199",
        "state":      "VIC",
        "start_date": "2018-01-01",
        "dir":        "listings/part2_midring/melbourne_frankston",
    },
]


def _extract_listings(data):
    if not isinstance(data, list):
        return []
    out = []
    for item in data:
        raw = item.get("listing") or item.get("listings")
        if isinstance(raw, dict):
            out.append(raw)
        elif isinstance(raw, list):
            out.extend(raw)
    return out


def _fetch_listings_pages(tracker, manifest, suburb, postcode, state,
                          start_date, out_dir, datatale_section):
    out_abs    = _DATATALE / out_dir
    state_file = out_abs / "fetch_state.json"
    page_size  = 200

    if state_file.exists():
        st       = json.loads(state_file.read_text())
        cursor   = st["cursor"]
        run      = st["run"]
        seen_ids = set(st["seen_ids"])
        print(f"  Resuming: cursor={cursor}  completed runs={run}  seen={len(seen_ids)}")
    else:
        cursor   = start_date
        run      = 0
        seen_ids = set()

    probe_path = f"{out_dir}/probe.json"
    if not _already_captured(manifest, probe_path):
        probe_r = tracker.post(
            f"{PROXY_BASE}/v1/listings/residential/_search",
            json_body={
                "listingType": "Sold",
                "listedSince": start_date,
                "sort": {"sortKey": "DateListed", "direction": "Ascending"},
                "locations": [{"state": state, "suburb": suburb,
                               "postCode": postcode, "includeSurroundingSuburbs": False}],
                "pageSize": 1, "pageNumber": 1,
            },
        )
        _save_probe(probe_path, probe_r)
        total_count = int(probe_r.headers.get("X-Total-Count", 0))
        _record_capture(manifest, probe_path,
                        endpoint="POST /v1/listings/residential/_search",
                        params={"suburb": suburb, "postcode": postcode,
                                "listedSince": start_date},
                        status=probe_r.status_code,
                        section=datatale_section)
        print(f"  Probe: {total_count:,} total records for {suburb}")
    else:
        probe_data  = json.loads((_DATATALE / probe_path).read_text())
        total_count = probe_data.get("x_total_count", 0)
        print(f"  Probe already captured: {total_count:,} records")

    if total_count == 0:
        print(f"  WARNING: probe returned 0 -- check suburb name and postcode")
        return

    payload = {
        "listingType": "Sold",
        "sort":        {"sortKey": "DateListed", "direction": "Ascending"},
        "locations":   [{"state": state, "suburb": suburb,
                          "postCode": postcode, "includeSurroundingSuburbs": False}],
    }

    while True:
        run      += 1
        page      = 1
        run_batch = []
        hit_limit = False
        fetch_err = False

        while True:
            page_path = f"{out_dir}/run_{run:03d}_page_{page:03d}.json"

            if _already_captured(manifest, page_path):
                existing = json.loads((_DATATALE / page_path).read_text())
                run_batch.extend(_extract_listings(existing))
                print(f"    run {run:03d} page {page:03d} -- already captured, skip")
                page += 1
                continue

            r = tracker.post(
                f"{PROXY_BASE}/v1/listings/residential/_search",
                json_body={**payload, "listedSince": cursor,
                           "pageSize": page_size, "pageNumber": page},
            )

            if r.status_code == 400:
                hit_limit = True
                break
            if r.status_code != 200:
                print(f"    ERROR {r.status_code} run {run:03d} page {page:03d}: {r.text[:100]}")
                print(f"  Stopping -- re-run cell to resume from cursor {cursor}")
                fetch_err = True
                break

            data     = r.json()
            listings = _extract_listings(data)
            new      = [l for l in listings if l.get("id") and l["id"] not in seen_ids]

            _save_json(page_path, data)
            _record_capture(manifest, page_path,
                            endpoint="POST /v1/listings/residential/_search",
                            params={**payload, "listedSince": cursor,
                                    "pageSize": page_size, "pageNumber": page},
                            status=r.status_code,
                            section=datatale_section)
            run_batch.extend(listings)
            seen_ids.update(l["id"] for l in listings if l.get("id"))
            print(f"    run {run:03d} page {page:03d}: {len(listings)} records  "
                  f"({len(new)} new  {len(seen_ids)} unique total)  [{tracker.total} credits]")

            if len(listings) < page_size:
                break
            page += 1

        if fetch_err:
            break

        if not hit_limit:
            print(f"  Done. {len(seen_ids):,} unique listings for {suburb}.")
            break

        if not run_batch:
            print("  Empty batch after 400 -- stopping")
            break

        dates = [l.get("dateListed", "") for l in run_batch if l.get("dateListed")]
        if not dates:
            print("  No dateListed in batch -- cannot advance cursor, stopping")
            break
        new_cursor = max(dates)[:10]
        if new_cursor == cursor:
            print(f"  Cursor stuck at {cursor} (all same date) -- stopping")
            break

        print(f"  Limit hit. Advancing cursor {cursor} -> {new_cursor}")
        cursor = new_cursor
        state_file.write_text(json.dumps(
            {"cursor": cursor, "run": run, "seen_ids": list(seen_ids)}, indent=2,
        ))


# One-time retrieval, off by default so this block can be read or run without
# spending credits. Files already recorded in the manifest are skipped, so a
# re-run on a populated cache is free. Left unguarded it fires live calls: a
# verification run on 2026-09-04 spent 2 credits and hit 401s mid-resume.
RUN_DOWNLOAD = False

if RUN_DOWNLOAD:
    _tracker_listings = APICallTracker()
    _manifest         = _load_manifest()   # reload in case stats run updated it

    print("Downloading sold listing records for the two suburbs...")
    print("(Files already on disk are skipped automatically)\n")

    for _cfg in _LISTINGS_TARGETS:
        print(f"--- {_cfg['role']}: {_cfg['suburb']} {_cfg['postcode']} ({_cfg['state']}) ---")
        (_DATATALE / _cfg["dir"]).mkdir(parents=True, exist_ok=True)
        _fetch_listings_pages(
            _tracker_listings, _manifest,
            suburb=_cfg["suburb"], postcode=_cfg["postcode"], state=_cfg["state"],
            start_date=_cfg["start_date"],
            out_dir=_cfg["dir"],
            datatale_section=f"part2_listings_{_cfg['role']}",
        )
        print()

    _tracker_listings.checkpoint("Listings download")
    print(f"Listings done. {_tracker_listings.total} credits spent.")
    _tracker_listings.summary()

Imports and configuration

# Standard library modules for file I/O and suppressing noisy warnings
import json, pathlib, warnings
# Numerical computing and tabular data
import numpy as np
import pandas as pd
# Interactive charting (graph_objects for fine control, express for quick plots, subplots for multi-panel)
import plotly.graph_objects as go
import plotly.express as px
from plotly.subplots import make_subplots
# Interactive geographic map with clickable markers
import folium
from folium.plugins import MarkerCluster
import requests, time
import plotly.io as pio

# One shared chart template so every figure gets the same breathing room: a tall
# top margin gives the title + subtitle their own band (clear of the plot area and
# of band labels like "COVID" / "RBA hikes"), and trimmed side margins let the plot
# use the full panel width.
pio.templates["datatale"] = go.layout.Template(layout=dict(
    margin=dict(l=70, r=50, t=40, b=70),
))
pio.templates.default = "plotly+datatale"
# Plotly + Quarto/closeread render fix. Every stock notebook renderer injects a global
# init (requirejs + an ES-module import of plotly.js) into the page <head>; both stop the
# plain Plotly.newPlot(...) calls from finding window.Plotly, so every chart renders blank.
# This renderer emits one self-contained block per figure -- a plain <script> that loads
# plotly.js from the CDN and draws directly -- with no global init and no requirejs.
from plotly.io.base_renderers import MimetypeRenderer
class _QuartoCDNRenderer(MimetypeRenderer):
    def __init__(self, config=None):
        self.config = config or {}
    def to_mimebundle(self, fig_dict):
        from plotly.io import to_html
        # include_plotlyjs=False: plotly.js is loaded once at body start by _amd-fix.html.
        # Emitting it per figure would make embed-resources inline the whole ~5MB bundle
        # once per chart.
        return {"text/html": to_html(fig_dict, include_plotlyjs=False, full_html=False,
                                     config={"responsive": True}, auto_play=False, validate=False)}
pio.renderers["quarto_cdn"] = _QuartoCDNRenderer()
pio.renderers.default = "quarto_cdn"


# Suppress pandas warnings about implicit sort order when concatenating DatetimeIndexes
warnings.filterwarnings("ignore", category=FutureWarning)
# Format floats with comma thousands-separators in DataFrame displays (e.g. 1,234,567)
pd.options.display.float_format = "{:,.0f}".format

# Root of the data tree; set to "." so paths resolve relative to this tale
DATATALE = pathlib.Path(".")
STATS_DIR   = DATATALE / "statistics"   # quarterly suburb performance aggregates
LIST_DIR    = DATATALE / "listings"     # individual property records

# Analysis window: 2018 to 2026; Q1 2018 is the index baseline for every indexed chart
STUDY_START = pd.Timestamp("2018-01-01")
STUDY_END   = pd.Timestamp("2026-12-31")

# Discover which cities have statistics data present on disk at runtime
cities_found = sorted(d.name for d in STATS_DIR.iterdir()
                      if d.is_dir() and d.name != "edge_cases")
print("Statistics available for:", cities_found)

Load statistics into tidy DataFrames (load_stats, all_stats)

def load_stats(path):
    """Read one suburb/category statistics JSON file and return a tidy DataFrame.

    The Domain API returns a nested 'series' > 'seriesInfo' structure; this
    function flattens it into one row per quarter.

    Returns None if the file is empty, missing the 'series' key, or contains
    no quarters with a medianSoldPrice (i.e. the suburb had too few sales to
    report in any quarter).

    Columns in the returned DataFrame (date-indexed, sorted ascending):
        medianSoldPrice, daysOnMarket, numberSold, p25, p75, meanRentYield
    """
    data = json.loads(path.read_text())
    if not data or "series" not in data:
        return None
    rows = data["series"]["seriesInfo"]
    records = [
        {
            "date":            pd.Timestamp(year=r["year"], month=r["month"], day=1),
            "medianSoldPrice": r["values"].get("medianSoldPrice"),
            "daysOnMarket":    r["values"].get("daysOnMarket"),
            "numberSold":      r["values"].get("numberSold"),
            "p25":             r["values"].get("25thPercentileSoldPrice"),
            "p75":             r["values"].get("75thPercentileSoldPrice"),
            "meanRentYield":   r["values"].get("meanRentYield"),
        }
        for r in rows
        if r["values"].get("medianSoldPrice")   # skip quarters with no price data
    ]
    if not records:
        return None
    return pd.DataFrame(records).set_index("date").sort_index()


def pct_change_from_q1_2018(df, col="medianSoldPrice"):
    """Add an 'indexed' column expressing price as % change from the Q1 2018 level.

    Q1 2018 is defined as any record where year==2018 and month in {1, 2, 3}.
    The baseline is the first available price in that window.

    Examples:
        indexed = +80  ->  price is 80% above its Q1 2018 level
        indexed =  -5  ->  price is 5% below its Q1 2018 level
        indexed =   0  ->  price equals its Q1 2018 level (the baseline quarter)

    Returns df unchanged (without an 'indexed' column) if no Q1 2018 data exists.
    """
    mask     = (df.index.year == 2018) & (df.index.month.isin([1, 2, 3]))
    baseline = df.loc[mask, col].dropna()
    if baseline.empty:
        return df
    df = df.copy()
    df["indexed"] = (df[col] / baseline.iloc[0] - 1) * 100   # 0 = Q1 2018 level
    return df


# Nested dict: all_stats[city][suburb_slug][category] = DataFrame
#   city         e.g. "melbourne", "brisbane"
#   suburb_slug  e.g. "hawthorn", "frankston"  (derived from the JSON filename stem)
#   category     "house" or "unit"
all_stats = {}
for city_dir in sorted(STATS_DIR.iterdir()):
    if not city_dir.is_dir() or city_dir.name == "edge_cases":
        continue
    city = city_dir.name
    all_stats[city] = {}
    for f in sorted(city_dir.glob("*.json")):
        # Filename convention: {suburb_slug}_{category}.json
        parts = f.stem.rsplit("_", 1)
        if len(parts) != 2:
            continue
        slug, cat = parts
        df = load_stats(f)
        if df is not None:
            df = pct_change_from_q1_2018(df)
            df = df[df.index >= STUDY_START]
        all_stats[city].setdefault(slug, {})[cat] = df

total = sum(len(s) for s in all_stats.values())
print(f"Loaded {total} suburb/category series across {len(all_stats)} cities")

City colours, labels, and centroids

# One consistent hex colour per city used across every chart in this tale
CITY_COLOURS = {
    "melbourne":      "#1f77b4",
    "sydney":         "#ff7f0e",
    "brisbane":       "#2ca02c",
    "perth":          "#d62728",
    "adelaide":       "#9467bd",
    "sunshine_coast": "#8c564b",
    "gold_coast":     "#e377c2",
    "canberra":       "#bcbd22",
}
# Display names used in chart titles, legend entries, and tooltips
CITY_LABELS = {
    "melbourne":      "Melbourne",
    "sydney":         "Sydney",
    "brisbane":       "Brisbane",
    "perth":          "Perth",
    "adelaide":       "Adelaide",
    "sunshine_coast": "Sunshine Coast",
    "gold_coast":     "Gold Coast",
    "canberra":       "Canberra",
}
# National divergence story: the six mainland capitals
MAJOR_CITIES   = ["melbourne", "sydney", "brisbane", "perth", "adelaide", "canberra"]
# Sea-changer section: Queensland lifestyle/coastal markets
COASTAL_CITIES = ["sunshine_coast", "gold_coast"]

# Approximate city centroids (lat, lon) for the geographic bubble map
CENTROIDS = {
    "melbourne":      (-37.8136, 144.9631),
    "sydney":         (-33.8688, 151.2093),
    "brisbane":       (-27.4698, 153.0251),
    "perth":          (-31.9505, 115.8605),
    "adelaide":       (-34.9285, 138.6007),
    "canberra":       (-35.2809, 149.1300),
    "sunshine_coast": (-26.6500, 153.0667),
    "gold_coast":     (-28.0167, 153.4000),
}

Chart code: a magnifier on Melbourne

Load Hawthorn and Frankston sold listings

# Melbourne property map: Hawthorn (inner, premium) vs Frankston (outer, affordable)
# Each dot is one property, coloured by relative price band; hover for type and bedrooms.

# Load listings first (needed for lat/lon)
# Listings Domain returns under the wrong suburb. One record is labelled Hawthorn but
# belongs to a neighbouring suburb, and it is the only Hawthorn record that sits well
# outside the suburb's footprint. Dropped so the Hawthorn vs Frankston comparison contains
# only those two suburbs. A suburb-scoped listings query can return neighbours, because
# Domain's own suburb field is sometimes wrong.
MISLABELLED_IDS = {2016558838}


def load_listings_dir(dir_path, study_start=STUDY_START, study_end=STUDY_END):
    """Load all run_*_page_*.json listing files from a directory into a single DataFrame.

    The Domain residential listings API uses 'soldData' (not 'soldDetails') for
    sold price and date, and 'propertyDetails' for type, bedrooms, and coordinates.

    Each JSON file contains a list of listing objects. Multiple files per suburb
    are expected (one per API page); all are concatenated and deduplicated by listing id.

    Columns returned:
        id, sold_date, sold_price, sale_method, property_type,
        bedrooms, lat, lon, quarter (Period), sold_year

    Filters applied:
        - sold_date within [study_start, study_end]
        - sold_price > $100,000 (removes data entry errors and non-arm's-length sales)
        - rows missing sold_date or sold_price are dropped
        - listing ids in MISLABELLED_IDS (Domain suburb-label errors)
    """
    dir_path = pathlib.Path(dir_path)
    if not dir_path.exists():
        print(f"  WARNING: {dir_path} not found -- run Phase D first")
        return pd.DataFrame()
    records = []
    for f in sorted(dir_path.glob("run_*_page_*.json")):
        data = json.loads(f.read_text())
        if not isinstance(data, list):
            continue
        for item in data:
            lst = item.get("listing") or item.get("listings")
            if not isinstance(lst, dict):
                continue
            sold = lst.get("soldData") or {}
            prop = lst.get("propertyDetails") or {}
            records.append({
                "id":            lst.get("id"),
                "sold_date":     sold.get("soldDate"),
                "sold_price":    sold.get("soldPrice"),
                "sale_method":   sold.get("saleMethod", ""),
                "property_type": prop.get("propertyType", "unknown").lower(),
                "bedrooms":      prop.get("bedrooms"),
                "lat":           prop.get("latitude"),
                "lon":           prop.get("longitude"),
            })
    if not records:
        return pd.DataFrame()
    df = pd.DataFrame(records).drop_duplicates(subset="id")
    df = df[~df["id"].isin(MISLABELLED_IDS)]
    df["sold_date"]  = pd.to_datetime(df["sold_date"], errors="coerce").dt.tz_localize(None)
    df["sold_price"] = pd.to_numeric(df["sold_price"], errors="coerce")
    df["lat"]        = pd.to_numeric(df["lat"], errors="coerce")
    df["lon"]        = pd.to_numeric(df["lon"], errors="coerce")
    df = df.dropna(subset=["sold_date", "sold_price"])
    df = df[(df["sold_date"] >= study_start) & (df["sold_date"] <= study_end)]
    df = df[df["sold_price"] > 100_000]
    df["quarter"]    = df["sold_date"].dt.to_period("Q")
    df["sold_year"]  = df["sold_date"].dt.year
    return df.sort_values("sold_date").reset_index(drop=True)


print("Loading Hawthorn...")
hawthorn = load_listings_dir(LIST_DIR / "part2_highvalue" / "melbourne_hawthorn")

print("Loading Frankston...")
frankston = load_listings_dir(LIST_DIR / "part2_midring" / "melbourne_frankston")

print(f"\nHawthorn: {len(hawthorn):,} property records, "
      f"${hawthorn['sold_price'].median()/1e6:.2f}M median")
print(f"Frankston: {len(frankston):,} property records, "
      f"${frankston['sold_price'].median()/1e6:.2f}M median")

Build the Folium property map

# Build a folium map with two togglable suburb layers.
# Each dot represents one sold property, coloured by price within that suburb's own range
# (using a 5-step palette from light to dark). This makes price variation visible
# within each suburb without being distorted by the large price gap between the two suburbs.

# Maps a sold price to one of 5 hex colours: ratio 0 -> palette[0] (cheapest),
# ratio 1 -> palette[4] (most expensive). Clipped to [0, 1] so outliers beyond
# the 5th-95th percentile range don't overflow the index.
def price_to_colour(price, p_min, p_max, palette):
    ratio = min(1.0, max(0.0, (price - p_min) / max(1, p_max - p_min)))
    idx = min(4, int(ratio * 5))
    return palette[idx]


HAWTHORN_PALETTE  = ["#fcc5c0", "#f768a1", "#dd3497", "#ae017e", "#49006a"]  # pink to purple
FRANKSTON_PALETTE = ["#c6dbef", "#6baed6", "#2171b5", "#08519c", "#08306b"]  # light to dark blue

# One colour per suburb, used by EVERY chart in this tale so each suburb keeps the same
# identity throughout. Taken from the mid-dark end of the two map palettes above, so the
# line charts agree with the map a reader has just scrolled past.
SUBURB_COLOURS = {"hawthorn": "#ae017e", "frankston": "#2171b5"}

# Basemap: Esri World Light Gray Base, served without an API key.
# NOT CartoDB Positron, which this map used until CARTO began requiring a key for
# their basemaps. Unauthenticated tiles still return HTTP 200, but with
# "API KEY REQUIRED" stamped diagonally across every one, so the map looked broken
# rather than failing outright. Esri's light grey canvas is the closest keyless
# match to Positron and keeps the basemap quiet enough for the suburb dots to read.
# Note the {z}/{y}/{x} order, which is Esri's and is the reverse of the usual.
# Built with tiles=None and the basemap added separately with control=False. Passing the
# tile URL to folium.Map names the base layer after that URL, so the layer control listed
# the whole https://server.arcgisonline.com/... string as a clickable legend entry.
m = folium.Map(
    location=[-37.87, 144.895],
    zoom_start=10,
    tiles=None,
)
folium.TileLayer(
    tiles=(
        "https://server.arcgisonline.com/ArcGIS/rest/services/"
        "Canvas/World_Light_Gray_Base/MapServer/tile/{z}/{y}/{x}"
    ),
    attr="Tiles © Esri — Esri, DeLorme, NAVTEQ",
    name="Basemap",
    control=False,
).add_to(m)

# Fly-to extents for the scroll bridge, taken from each suburb's own sold-listing
# coordinates rather than a boundary polygon: no outline is drawn, so the suburb
# footprint is implied by where its dots sit. The 2nd-98th percentile is used so a
# single stray sale on the edge cannot stretch the frame.
SUBURB_BOUNDS = {}   # {"hawthorn": [[S,W],[N,E]], ...} -> fed to the scroll bridge
for _bdf, _key in [(hawthorn, "hawthorn"), (frankston, "frankston")]:
    _pts = _bdf.dropna(subset=["lat", "lon"])
    if not _pts.empty:
        SUBURB_BOUNDS[_key] = [
            [_pts["lat"].quantile(0.02), _pts["lon"].quantile(0.02)],
            [_pts["lat"].quantile(0.98), _pts["lon"].quantile(0.98)],
        ]

# Open tight on the two suburbs rather than on all of greater Melbourne. zoom_start=10
# framed the whole bay, leaving the two clusters as specks in a lot of empty grey. This
# fits the union of the same 2nd-98th percentile extents the scroll bridge flies to, so
# the opening view and the per-suburb views are derived from one set of numbers. The
# bridge captures its "home" view from the map's live state, so it returns here.
_fit_pts = [pt for _b in SUBURB_BOUNDS.values() for pt in _b]
if _fit_pts:
    m.fit_bounds(
        [[min(p[0] for p in _fit_pts), min(p[1] for p in _fit_pts)],
         [max(p[0] for p in _fit_pts), max(p[1] for p in _fit_pts)]],
        # Weighted to the top rather than even. The extents are 2nd-98th percentile, so
        # ~2% of dots fall outside them and Hawthorn's northern edge was landing flush
        # against the top of the frame. Leaflet picks the zoom from the TOTAL padding,
        # so an even 40px cost a whole zoom level and undid the point of fitting; this
        # totals less than an even 18px inset while still pushing the content down.
        padding_top_left=(12, 30),
        padding_bottom_right=(12, 4),
    )

PRICE_RANGE = {}   # {"Hawthorn": (p5, p95), ...} -> the value scale under each legend ramp

# Readable tooltip names for Domain's propertyType codes (lowercased by load_listings_dir).
# Anything not listed falls back to the code with its first letter capitalised.
TYPE_LABELS = {
    "house":             "House",
    "apartmentunitflat": "Apartment",
    "townhouse":         "Townhouse",
    "vacantland":        "Vacant land",
    "villa":             "Villa",
    "studio":            "Studio",
    "blockofunits":      "Block of units",
    "newhouseland":      "House and land",
    "newapartments":     "New apartment",
    "developmentsite":   "Development site",
    "duplex":            "Duplex",
    "penthouse":         "Penthouse",
    "newland":           "Land",
    "semidetached":      "Semi-detached",
}

# Plain suburb names on the feature groups. They used to carry the colour key as well
# ("Hawthorn (pink = lower, purple = higher price)"), which duplicated the legend below
# and disagreed with it - one called Frankston "light to dark", the other "blue scale".
for suburb_df, name, palette in [
    (hawthorn,  "Hawthorn",  HAWTHORN_PALETTE),
    (frankston, "Frankston", FRANKSTON_PALETTE),
]:
    fg = folium.FeatureGroup(name=name, show=True)

    plot_df = suburb_df[suburb_df["lat"].notna() & suburb_df["lon"].notna()].copy()
    # Use 5th-95th percentile as the colour scale range to reduce distortion from outliers
    p5, p95  = plot_df["sold_price"].quantile([0.05, 0.95])
    PRICE_RANGE[name] = (p5, p95)

    # Cap at 500 markers per suburb for browser rendering performance
    if len(plot_df) > 500:
        plot_df = plot_df.sample(500, random_state=42)

    for _, row in plot_df.iterrows():
        colour = price_to_colour(row["sold_price"], p5, p95, palette)
        # Tooltip carries property attributes only. No sale price or sale date is shown for
        # any individual property: that is often Valuer General data, which must not be
        # republished on a free asset. Price survives only as the relative colour band.
        label = TYPE_LABELS.get(row["property_type"], row["property_type"].capitalize())
        if pd.notna(row["bedrooms"]):
            label += f" · {int(row['bedrooms'])} bed"
        # Coordinates rounded to 3 decimals (~100m) so a dot shows the suburb's pattern
        # without marking a specific address.
        folium.CircleMarker(
            location=[round(row["lat"], 3), round(row["lon"], 3)],
            radius=5,
            color=colour,
            fill=True,
            fill_color=colour,
            fill_opacity=0.7,
            weight=0.5,
            tooltip=label,
        ).add_to(fg)
    fg.add_to(m)

# The map's single legend. The swatches are generated from HAWTHORN_PALETTE and
# FRANKSTON_PALETTE, the same lists price_to_colour draws the markers from, so the key
# cannot fall out of step with the map. Written with placeholders rather than an f-string
# because the block is mostly CSS braces.
SWATCH_W = 60   # px per colour step; wide enough for a price label under each boundary

def _money(v):
    """Round a price for the legend: $X.XM at or above $1M, else to the nearest $10K."""
    return f"${v/1e6:.1f}M" if v >= 1e6 else f"${round(v, -4)/1e3:.0f}K"

def _scale(name, palette):
    """One suburb's colour ramp with a value axis beneath it, like a histogram's x-axis.

    The tick values are the bin edges price_to_colour uses: five equal steps from the
    suburb's 5th to 95th percentile price. They are suburb-wide aggregates, rounded, so
    they give the scale without exposing any one sale."""
    lo, hi = PRICE_RANGE[name]
    edges = np.linspace(lo, hi, len(palette) + 1)
    swatches = "".join(
        '<span style="display:inline-block;width:%dpx;height:13px;background:%s;"></span>'
        % (SWATCH_W, c) for c in palette
    )
    ticks = ""
    for i, v in enumerate(edges):
        # End labels are pinned inside the ramp so they cannot spill past the legend box
        shift = "0" if i == 0 else ("-100%" if i == len(edges) - 1 else "-50%")
        ticks += (
            '<span style="position:absolute; left:%dpx; top:0; width:1px; height:4px; background:#6b7a72;"></span>'
            '<span style="position:absolute; left:%dpx; top:4px; transform:translateX(%s); white-space:nowrap;">%s</span>'
            % (min(i * SWATCH_W, len(palette) * SWATCH_W - 1), i * SWATCH_W, shift, _money(v))
        )
    return (
        '<div style="margin-bottom:10px;">'
        '<div style="font-size:14.5px; font-weight:600; margin-bottom:3px;">%s</div>'
        '<div style="line-height:0;">%s</div>'
        '<div style="position:relative; width:%dpx; height:22px; font-size:12.5px; color:#6b7a72;">%s</div>'
        '</div>'
    ) % (name, swatches, len(palette) * SWATCH_W, ticks)

legend_html = """
<style>
  /* On a phone the legend spanned most of the map's width and hid half the dots. Shrunk
     and tucked into the corner when the map frame is narrow; !important beats the inline
     position below. */
  @media (max-width: 500px) {
    .map-legend { transform: scale(0.62); transform-origin: left bottom;
                  bottom: 8px !important; left: 8px !important; }
  }
</style>
<div class="map-legend" style="position:fixed; bottom:26px; left:26px; z-index:9999;
            background:rgba(255,255,255,0.94); padding:14px 17px; border-radius:8px;
            border:1px solid #d8ddda; box-shadow:0 2px 6px rgba(0,0,0,0.14);
            font:15px/1.5 -apple-system, Segoe UI, Roboto, sans-serif; color:#2b2b2b;">
  <div style="font-weight:700; font-size:16px; margin-bottom:9px;">Property data, 2018-2026</div>
  __HAWTHORN__
  __FRANKSTON__
</div>
""".replace("__HAWTHORN__", _scale("Hawthorn", HAWTHORN_PALETTE)) \
   .replace("__FRANKSTON__", _scale("Frankston", FRANKSTON_PALETTE))
m.get_root().html.add_child(folium.Element(legend_html))

# Scroll bridge: listen for messages from the closeread narrative (parent page)
# and fly the Leaflet map to a suburb's extent, or back to the full view.
# `m.get_name()` is the JS global Folium assigns to this map inside the iframe.
zoom_bridge = """
<script>
(function () {
  var BOUNDS = __BOUNDS__;
  var MAP_NAME = "__MAP_NAME__";
  var home = null;
  function getMap() { return window[MAP_NAME]; }
  // On a phone the map fills the middle of the screen, so a one-finger drag on it must
  // scroll the story rather than pan the map. The +/- buttons still zoom, and the story
  // still flies the map to each suburb.
  window.addEventListener('load', function () {
    var map = getMap();
    if (map && window.innerWidth < 576) {
      map.dragging.disable();
      if (map.tap) map.tap.disable();
    }
  });
  window.addEventListener('message', function (ev) {
    var d = ev.data || {};
    if (d.type !== 'cr-zoom') return;
    var map = getMap();
    if (!map) return;
    if (!home) home = { center: map.getCenter(), zoom: map.getZoom() };
    if (d.suburb && BOUNDS[d.suburb]) {
      map.flyToBounds(BOUNDS[d.suburb], { padding: [25, 25], duration: 1.1 });
    } else {
      map.flyTo(home.center, home.zoom, { duration: 1.1 });
    }
  });
}());
</script>
""".replace("__BOUNDS__", json.dumps(SUBURB_BOUNDS)).replace("__MAP_NAME__", m.get_name())
m.get_root().html.add_child(folium.Element(zoom_bridge))

Render the map iframe

# Embed the Folium map directly in the page via srcdoc rather than writing it to
# images/ and pointing an iframe at it. embed-resources inlines CSS/JS but does NOT
# follow an iframe src, so a file-based map would break the moment the .html is moved.
import html as _html
print(f'<iframe class="cr-map" srcdoc="{_html.escape(m.get_root().render())}"></iframe>')

Hawthorn vs Frankston: 4-bedroom comparison

# Like-for-like comparison: 4-bedroom houses and townhouses, indexed to each suburb's own Q1 2018 price.
# The shaded band shows the 25th-75th percentile range, giving a sense of price spread
# within the 4-bedroom segment (not just the median line).
fig = go.Figure()

for df, name, colour in [
    (hawthorn,  "Hawthorn",  SUBURB_COLOURS["hawthorn"]),
    (frankston, "Frankston", SUBURB_COLOURS["frankston"]),
]:
    if df.empty:
        continue
    # Filter to 4-bedroom houses and townhouses only
    four_bed = df[
        df["property_type"].isin(["house", "townhouse"]) &
        (df["bedrooms"] == 4)
    ].copy()
    if len(four_bed) < 8:
        print(f"  WARNING: {name} has only {len(four_bed)} 4-bed records")
        continue

    # Resample to quarterly medians
    qmed = (
        four_bed.set_index("sold_date")["sold_price"]
        .resample("QS").median()
        .rename("medianSoldPrice")
        .to_frame()
    )

    # Index to Q1 2018 = 0% for this suburb (same logic as pct_change_from_q1_2018)
    mask = (qmed.index.year == 2018) & (qmed.index.month.isin([1, 2, 3]))
    baseline_vals = qmed.loc[mask, "medianSoldPrice"].dropna()
    if baseline_vals.empty:
        continue
    baseline = baseline_vals.iloc[0]
    qmed["pct"] = (qmed["medianSoldPrice"] / baseline - 1) * 100

    # 25th and 75th percentile bands for context
    q25 = four_bed.set_index("sold_date")["sold_price"].resample("QS").quantile(0.25)
    q75 = four_bed.set_index("sold_date")["sold_price"].resample("QS").quantile(0.75)
    pct25 = (q25 / baseline - 1) * 100
    pct75 = (q75 / baseline - 1) * 100

    # Convert hex colour to rgb for the semi-transparent fill
    r, g, b = int(colour[1:3], 16), int(colour[3:5], 16), int(colour[5:7], 16)

    # Shaded interquartile band (rendered as a filled polygon)
    fig.add_trace(go.Scatter(
        x=list(pct25.index) + list(pct25.index[::-1]),
        y=list(pct75.values) + list(pct25.values[::-1]),
        fill="toself",
        fillcolor=f"rgba({r},{g},{b},0.12)",
        line=dict(color="rgba(0,0,0,0)"),
        name=f"{name} 25-75th pct",
        legendgroup=name,
        showlegend=True,
        hoverinfo="skip",
    ))

    fig.add_trace(go.Scatter(
        x=qmed.index, y=qmed["pct"],
        mode="lines+markers",
        name=f"{name} median",
        legendgroup=name,
        line=dict(color=colour, width=3),
        marker=dict(size=5),
        hovertemplate=f"<b>{name} (4 bed)</b><br>%{{x|%b %Y}}: %{{y:+.1f}}%<extra></extra>",
    ))

fig.add_hline(y=0, line_dash="dot", line_color="#888", opacity=0.7,
              annotation_text="Q1 2018 (0%)", annotation_position="bottom right")

fig.add_vrect(x0="2020-03-01", x1="2020-12-01",
              fillcolor="grey", opacity=0.14, line_width=0,
              annotation_text="COVID", annotation_position="top left",
              annotation_font_size=12)

fig.add_vline(x="2022-05-01", line_dash="dot", line_color="#555", opacity=0.8,
              annotation_text="RBA hikes begin", annotation_position="top right",
              annotation_font_size=12)

fig.update_layout(
    yaxis_title="% change since Q1 2018",
    xaxis_title="",
    height=660,
    hovermode="x unified",
    plot_bgcolor="white",
    legend=dict(
        orientation="v", x=1.01, y=1, xanchor="left",
        bgcolor="rgba(255,255,255,0.8)", bordercolor="#ddd", borderwidth=1,
    ),
    yaxis=dict(gridcolor="#eee", zeroline=True, zerolinecolor="#bbb", zerolinewidth=1),
    xaxis=dict(gridcolor="#eee"),
)
fig

Chart code: beyond price

Median days on market

# Days on market (DOM) from the statistics endpoint (not available in listing records).
# Bar chart with a line overlay for each suburb to make the trend easier to read.
# DOM is often described as a leading indicator. In this data the reliable signal is the
# gap in level between the two suburbs, not the trend within either one.
melb = all_stats.get("melbourne", {})
fig = go.Figure()

for slug, name, colour in [
    ("hawthorn",  "Hawthorn",  SUBURB_COLOURS["hawthorn"]),
    ("frankston", "Frankston", SUBURB_COLOURS["frankston"]),
]:
    df_s = melb.get(slug, {}).get("house")
    if df_s is None or "daysOnMarket" not in df_s.columns:
        continue
    dom = df_s["daysOnMarket"].dropna()
    dom = dom[(dom.index >= STUDY_START) & (dom.index <= STUDY_END)]

    # Bar for per-quarter values; line overlay for the trend
    fig.add_trace(go.Bar(
        x=dom.index, y=dom.values,
        name=name,
        marker_color=colour,
        opacity=0.55,
        hovertemplate=f"{name}: %{{y:.0f}} days<extra></extra>",
    ))
    fig.add_trace(go.Scatter(
        x=dom.index, y=dom.values,
        name=f"{name} trend",
        legendgroup=name,
        mode="lines",
        line=dict(color=colour, width=2),
        showlegend=False,
    ))

fig.update_layout(
    yaxis_title="Days on market",
    height=620,
    barmode="group",
    plot_bgcolor="white",
    hovermode="x unified",
    yaxis=dict(gridcolor="#eee"),
    xaxis=dict(gridcolor="#eee"),
)
fig

Sale method mix by quarter

# Sale method constants: the three values returned by the Domain API for sold listings
METHOD_ORDER  = ["SoldByAuction", "SoldPriorToAuction", "SoldByPrivateTreaty"]
METHOD_LABELS = {
    "SoldByAuction":       "Auction",
    "SoldPriorToAuction":  "Prior to auction",
    "SoldByPrivateTreaty": "Private treaty",
}
# Green = auction (confident market); red = private treaty (softer market)
METHOD_COLOURS = {
    "SoldByAuction":       "#2ca02c",
    "SoldPriorToAuction":  "#98df8a",
    "SoldByPrivateTreaty": "#d62728",
}

fig = make_subplots(rows=1, cols=2,
                    subplot_titles=["Hawthorn", "Frankston"])

for col_idx, (df, name) in enumerate([(hawthorn, "Hawthorn"), (frankston, "Frankston")], start=1):
    if df.empty:
        continue

    df2 = df.copy()
    # Any unrecognised sale method (rare API values) is folded into private treaty
    df2["method"] = df2["sale_method"].where(
        df2["sale_method"].isin(METHOD_ORDER), "SoldByPrivateTreaty"
    )
    # Pivot: rows = quarter, columns = sale method, values = count of sales
    pivot = (df2.groupby(["quarter", "method"])
               .size()
               .unstack(fill_value=0)
               .reindex(columns=METHOD_ORDER, fill_value=0))
    # Convert to percentage share so both suburbs are on a 0-100% scale
    pivot_pct = pivot.div(pivot.sum(axis=1), axis=0) * 100

    bottom = np.zeros(len(pivot_pct))
    for method in METHOD_ORDER:
        vals = pivot_pct[method].values if method in pivot_pct else np.zeros(len(pivot_pct))
        fig.add_trace(go.Bar(
            x=[str(q) for q in pivot_pct.index],
            y=vals,
            name=METHOD_LABELS[method],
            legendgroup=method,
            showlegend=(col_idx == 1),   # only show legend once (shared across both panels)
            marker_color=METHOD_COLOURS[method],
            hovertemplate=f"{METHOD_LABELS[method]}: %{{y:.1f}}%<extra></extra>",
        ), row=1, col=col_idx)

fig.update_layout(
    height=640,
    barmode="stack",
    plot_bgcolor="white",
    yaxis=dict(title="% of sales", gridcolor="#eee", range=[0, 100]),
    yaxis2=dict(gridcolor="#eee", range=[0, 100]),
    hovermode="x unified",
)
fig

Bedroom composition and pricing grid

# Two-row, two-column layout:
#   Row 1 (volume):  quarterly count of sales by bedroom tier (stacked bar)
#   Row 2 (price):   quarterly median sold price by bedroom tier (line)
# Reading the columns together shows whether composition shifts or genuine price
# moves are driving the suburb-level median.
BED_PAL = {
    "1": "#aec7e8", "2": "#ffbb78", "3": "#98df8a",
    "4": "#ff9896", "5+": "#c5b0d5", "unknown": "#c7c7c7",
}

def classify_beds(x):
    # Returns "1" through "4", "5+", or "unknown" for NaN values
    if pd.isna(x): return "unknown"
    return "5+" if x >= 5 else str(int(x))


fig = make_subplots(
    rows=2, cols=2,
    subplot_titles=[
        "Hawthorn: sales volume by bedrooms",
        "Frankston: sales volume by bedrooms",
        "Hawthorn: median sold price by bedrooms",
        "Frankston: median sold price by bedrooms",
    ],
    vertical_spacing=0.14,
    horizontal_spacing=0.1,
)

for col_idx, (df, name) in enumerate([(hawthorn, "Hawthorn"), (frankston, "Frankston")], start=1):
    if df.empty:
        continue
    df2 = df.copy()
    df2["bed_group"] = df2["bedrooms"].apply(classify_beds)

    # Count sales per bedroom group per quarter
    pivot = (df2.groupby(["quarter", "bed_group"])
               .size()
               .unstack(fill_value=0))
    qlabels = [str(q) for q in pivot.index]

    # Row 1: stacked bar showing what bedroom mix is selling each quarter
    bottom_vals = np.zeros(len(pivot))
    for bed in ["1", "2", "3", "4", "5+", "unknown"]:
        if bed not in pivot.columns:
            continue
        vals = pivot[bed].values
        fig.add_trace(go.Bar(
            x=qlabels, y=vals,
            name=f"{bed} bed",
            legendgroup=bed,
            showlegend=(col_idx == 1),
            marker_color=BED_PAL[bed],
            hovertemplate=f"{bed} bed: %{{y}}<extra></extra>",
        ), row=1, col=col_idx)

    # Row 2: median price per bedroom group, in $M, resampled to quarterly
    for bed in ["2", "3", "4", "5+"]:
        subset = df2[df2["bed_group"] == bed]
        if len(subset) < 4:
            continue
        qmed = (subset.set_index("sold_date")["sold_price"]
                .resample("QS").median() / 1e6)
        qmed = qmed[(qmed.index >= STUDY_START) & (qmed.index <= STUDY_END)]
        fig.add_trace(go.Scatter(
            x=qmed.index, y=qmed.values,
            name=f"{bed} bed",
            legendgroup=bed,
            showlegend=False,
            mode="lines",
            line=dict(color=BED_PAL[bed], width=2),
            hovertemplate=f"{bed} bed: $%{{y:.2f}}M<extra></extra>",
        ), row=2, col=col_idx)

fig.update_layout(
    height=820,
    barmode="stack",
    plot_bgcolor="white",
    hovermode="x unified",
)
for axis in ["yaxis", "yaxis2", "yaxis3", "yaxis4"]:
    fig.update_layout(**{axis: dict(gridcolor="#eee")})
for i in [3, 4]:
    fig.update_yaxes(tickprefix="$", ticksuffix="M", row=2, col=(i - 2))
fig