How we test search and booking
We give the same San Francisco requests to WhatsDo and to other search services. A judge reads two answers without knowing whose they are and picks the one that fits the request better.
Results on 91 requests
Wins for WhatsDo against wins for the other service. Ties are not counted. Against Google Places the result is close.
- WhatsDo vs Exa46–16
- WhatsDo vs Keenable56–13
- WhatsDo vs Google Places28–19
- Exact place asked by name10 of 12found in the first five results
How the test works
- 1. Real requests
91 requests people make in San Francisco: food, mood, nearby, nightlife, kids, hotels, typos and follow-ups.
- 2. Same request everywhere
Each service gets the same words and returns its first five places.
- 3. Blind judge
An AI judge sees two lists without service names. We remove our ratings and booking labels so the judge cannot tell which list is ours.
- 4. Judged twice
Every pair is judged a second time with the lists swapped. A verdict counts only when both rounds agree.
What a good answer is
The judge looks only at fit: the right kind of place, the mood or occasion asked for, the price and the area. Length and detail do not count.
An empty answer beats a list with nothing relevant. A guide page counts only for the places it names.
Where we still lose
Other services still give better answers for hotels, event venues, activities and places for pets. These gaps come from our catalog and tags, and we work on them first.
- Hotels
- Event venues
- Activities
- Pets
Booking
Every booking request ends with one of three answers, and the agent shows it to the person as is.
- Confirmed
- Waiting for the venue
- Not possible
Limits
- San Francisco only, where our catalog is built out: 22,379 places, 1,206 of them take booking requests.
- One judge model, relevance only. Pairs where the two rounds disagree are left out.
- Other services got the request with the city added when it was missing.