Two answers to the same Dubai property question share about half their firms

Merging spelling variants moves individual pairs, not the middle

This section comes first because it's the part I'd argue with hardest if someone else had written it.

ChatGPT calls the same firm several things. Betterhomes appears as "Betterhomes", "betterhomes - Dubai (Head Office)" and "betterhomes - Marina Office". fäm Properties holds 14 office registration numbers in the DLD register. Count those as different firms and the answers look less consistent than they are.

So I resolved every name against the register, merged branches and spelling variants onto one licence identity, and recomputed.

Nearly half the pairs moved and the middle barely did. That's worth separating, because the two facts point different ways. Resolution matters a great deal for any individual firm's number and very little for the headline.

The strict counts move more. On "most reputable", the number of firms named in all 5 runs goes from 1 to 4 once variants are merged. The 4 are Allsopp & Allsopp, Betterhomes, haus & haus and Metropolitan Premium Properties. Betterhomes was in every run the whole time, under three different strings. Anyone measuring this without a register would have reported 1.

I had drafted a version of this piece claiming that a large share of the volatility other people report is their own name-matching. The table above says that's wrong at the aggregate level, so the claim is gone. It survives only for per-firm figures, where it's severe.

Dubai Marina

Naming a district instead of the city produced the steadiest answers in the edition: median 0.66, and 10 firms named in every one of the 5 runs out of 21 total.

Against the matched question ("I want to buy an apartment in Dubai", 0.24) that's 2.75x. Against the edition median, 2.1x. Pick whichever you prefer, and note this rests on one question in one edition. n = 1. I'm not going to defend it further until Singapore.

My first guess was that Marina has fewer agencies to choose from. My second was that a district name triggers the map block, which is fed by a places provider and ought to be steadier than prose. The data doesn't support the second one cleanly. Marina is the most stable question on map-block names alone (0.68) and also on prose names alone (0.54), and it's the highest of all 10 questions on both. The Marina result isn't explained by the map-block names alone.

Every Dubai brokerage named in at least 5 of the 50 runs

42 registered firms cleared that bar. "Registered" here means the name resolved to an office registration number in the 16 September snapshot. It does not mean the firm is trading: a licence can sit on the register with no active brokers attached, and at least one entry below is in that position. Each one is matched to the DLD broker register; the register's own spelling is in the second column, and branch listings and spelling variants are merged onto one licence identity.

Read it as reach, not as a ranking. Rows are ordered by how many runs named the firm, because that is the readable order, and the ordering is not a ranking of brokerages. These figures pool 10 different questions, and renting, luxury and listing-to-sell should produce different firms even from a perfectly steady engine.

Read the interval, not the percentage. Every rate here comes from 50 runs, which is a small sample. The interval column gives descriptive 95% binomial intervals, computed as though the 50 runs were independent trials. They are not: 5 runs belong to each question, the questions are deliberately different, and a firm's chance of being named is correlated within a question. So read these as a rough width on a small sample rather than as confidence intervals for how ChatGPT will behave in general. Almost every pair of firms in this table has overlapping intervals, and treating the order of these rows as a ranking is the specific mistake this column exists to stop.

The register writes almost every firm in capitals. "Impressive Real Estate" is the exception in this table and is stored in the DLD export exactly as shown, mixed case. I've reproduced each spelling as the export has it rather than tidying it.

These counts are floors. The audit found that firms named in a sentence rather than a list are missed, so any firm recommended in prose is undercounted here by an unknown amount. Chestertons at 3 runs, Provident at 10 and Dacha at 11 each have at least one prose mention the extractor did not catch. The audit covered 12 of the 50 runs, so I can't say how far this moves the rest of the table.

37 of the 42 appeared in fewer than 4 answers in 10. Only 5 reached 20 runs or more, and only Exclusive Links passed 30. Reach is out of all 50 runs; on the 49 that named anyone, Exclusive Links is 71% rather than 70%.

The "Questions" column is the one I'd watch if I ran a brokerage, because it captures something raw run count doesn't: whether a firm shows up across different kinds of buyer, or repeatedly under one. 10 questions is itself a small sample of the questions people ask. Exclusive Links turned up under 9 of the 10 questions. Espace, Agency Eight and Aaronz & Co each appeared under 1, which means their presence rests entirely on one kind of buyer. Two firms can share a reach percentage and a confidence interval and still be in completely different positions on this column.

23 more licensed firms, named in 2 to 4 runs

No rate published for these. At 2 to 4 appearances out of 50 the interval runs from about 1% to 19% and the number says more about the sample than the firm. My threshold for a published rate is 5. They're listed because they were named and they're licensed, which is checkable either way.

A further 24 licensed firms were named exactly once. I've left those out.

21 names I couldn't match to the register

Four are my own matching falling short, not missing firms.

The other 17 returned nothing against the DLD broker export under any spelling I tried. Trade names diverge from brands constantly: White & Co is registered as White and White Real Estate, D&B Properties as D A N D B Properties. Others may be licensed in another emirate, since my location parameter covers the whole UAE. Some may be international firms with no Dubai office. I haven't checked the DLD developer register or the free-zone register against these, so I'm not calling any of them unlicensed or non-existent.

What this method can't see

Firms named in a sentence are missed. My extractor reads the map block, bulleted lists and tables, and nothing else. A recommendation written as prose is invisible to it.

The audit put a number on that. 50 of the 54 misses sit in inline prose, across four different questions. It's one failure mode rather than a scatter of unrelated gaps, and it's the same one that had me record the off-plan answers as empty.

What the extraction audit found

An earlier version of this audit reported 93.8% recall. That figure was wrong in a way worth describing, because the mistake is easy to make. 98 of its 227 names had been pulled by regex out of the map block rather than read by anyone, so 43% of the audit was comparing the extractor against itself. It also happened to sample none of the runs where the extractor had failed.

Redone on 12 runs, with ground truth written by a separate model instance that never saw the extractor's output. Blind to the extractor, not independent of me: I wrote both the extractor and the ruleset the reader worked from, and the detail is below the table.

The two recall figures answer different questions, and the gap between them is almost entirely regulators, portals and named systems: DLD, RERA, Trakheesi, Oqood, Dubai REST, Google, Bayut, Property Finder, TruBroker. Those get named constantly and none of them belongs in a brokerage scoreboard. 90.9% is the figure that bears on the tables above. 59.4% is what the extractor does with everything else.

Both interval columns on this page give descriptive 95% binomial intervals, calculated as though each run or each name were an independent trial. Runs are clustered by question and names are clustered within runs, so these are a rough width on a small sample, not population-level confidence intervals for how ChatGPT behaves in general.

The two precision figures are worth separating. Excluding the checklist labels the extractor is right 79 times out of 81, and the 13 errors it does make are almost all one known class: words like "Budget" and "Payment plan" that open a numbered list and get read as names. That class is visible, countable and in the published data.

All of these are scored against the answer text, which is what the ground-truth reader saw. Scoring the map block as well changes recall by nothing at all, because every map entry the reader saw was repeated in prose, and it drops apparent precision to 34.2% purely by adding names the audit never covered. That lower number measures the audit's scope rather than the extractor, so it is not quoted as precision here. Both variants are in the published scoring output.

Two scorers were written for this, independently, and they were compared after both had run. They agree on every one of the 133 hand-read names and all 92 extracted names, and disagree on 3 rows, all about which names count as brokerages rather than about whether the extractor found them. One scorer counted Sobha Realty, which holds a licence but is a developer everywhere else in this study. The other counted two copies of a map-listing title as a firm. That disagreement is the whole gap between 90.9% and 91.0%. The figure quoted here is the one the committed rule produced, including where the rule is arguably worse than the alternative, and edition 2 fixes both defects in the rule rather than in the number.

Parse fidelity is a code test rather than a hand read, and it's reported separately for that reason. Every map-block entry in every run became a stored mention, with no drift in position, title or URL.

All 7 missed brokerages were named in prose. Chestertons, Dacha and Provident in one sentence of the Dubai Marina question ("A Dubai Marina directory also currently identifies Chestertons, Dacha, Espace and Provident"). Allsopp & Allsopp and Betterhomes in one sentence of the investor question. Sotheby's and fäm the same way.

Three runs were audited separately because the extractor had scored them as empty: both off-plan runs and the clarifying answer. Recall there was 0 of 14. Those numbers are reported on their own and never folded into the 12-run figures, since deliberately sampling known failures would bias the result.

Ground truth came from a separate model instance with no access to the extractor's output, one session per run, working from a fixed ruleset. The ruleset changed 6 times over the first 6 runs and held for the last 9; 3 runs were re-read under the final version, 1 run was replaced after I supplied a name list before it ran, and 4 results were amended by me afterwards, each time to add a name the ruleset covered and the session had left out. I wrote the extractor and the ruleset, so this is a blind read rather than an independent one.

Extractor false positives are still in the counts. They're checklist labels like "Budget" and "Payment plan" that open numbered lists, and the resolver classes them as not-a-firm rather than dropping them. This edition has 170 distinct names, 83 of which appear in exactly one run. 28 of the 170 fall in the not-a-firm class, and 27 of those 28 are singletons. Excluding them gives 142 distinct names and 56 singletons. None reached position 1 in any run.

No control question. All 10 prompts ask for an agency, so this edition has no null condition, and precision (85.9% with checklist labels counted) is the only bound on the extractor over-firing. Edition 2 adds a question that asks about the buying process and names no firm; unsolicited naming there will be worth more than anything in the tables above.

The model is not identified. The endpoint did not report a model identifier on any of the 50 runs. This measures what came back through one vendor's ChatGPT search-mode interface on 16 September 2026. It cannot be attributed to a named model, and if the model behind that interface changed the day after, nothing here would show it.

One moment, not many. All 50 runs returned inside 3 minutes 24 seconds of each other. That limits how much ordinary market change could plausibly explain the differences between the first answer and the last. It does not identify what produced them: the endpoint returned no search_results and no fan_out_queries, so the retrieval behind each answer is not visible here and the variation cannot be attributed to any particular mechanism. The flip side is that this measures instability within one moment and says nothing about whether the same question answered next week returns the same firms. Those are different quantities and this edition only has the first. Edition 2 repeats the pack on separate days to get the second.

Country-level location. Dubai reaches the engine through the question text, not the geography setting, and one run stopped to ask me where I was. This isn't the same as asking from a Dubai IP, and I haven't tested whether the two agree. That calibration run is the first thing scheduled for edition 2.

English only. Arabic and Russian are real buyer languages in this market and transliterated firm names are a matching problem this method doesn't handle.

One engine, one day. ChatGPT search mode on 16 September 2026, 5 runs per question. Gemini, Perplexity and Google's AI Overviews may behave completely differently.

What I'd take from this if I ran a brokerage

Being named once tells you close to nothing. A firm can appear in one answer and be gone from the next, and both are normal.

The number worth tracking is how often you appear across many runs of many questions, and how many different questions reach you. A firm at 70% across 9 questions and a firm at 10% under 1 question are in different positions even if both can produce a screenshot.

Which is also the problem with a screenshot. "Best agencies" produced one pair of runs overlapping at 0.79 and another at 0.12, minutes apart, on the same question. Either one makes a convincing image.

And check what question you're competing for before optimising for it. Off-plan returns developers in 5 of 5 runs.

Edition 2 runs 18 questions on a frozen prompt set at deeper repetition, with a control question added, recommendation strength scored rather than presence counted, and a same-day calibration run from a Dubai IP. Singapore follows on the same pack.

Appendix: the 10 questions, verbatim

Every run sent one of these strings exactly as written, with no system prompt and no conversation history.