ISSUE 31 · ME+MACHINE
I was hungry. My stomach had started filing complaints. I asked Claude for a list of the best places to eat in Hilo. It pulled one together, but my favorite restaurant in town, Cafe Pesto, wasn’t even on it.
Rude. It’s the best spot in Hilo!
So I sent Claude the Yelp page and asked what people had been saying lately… and Claude refused. “Yelp blocks tools like mine from reading it, and I shouldn’t work around that block.”
Claude’s refusal.
Interesting. Yelp is a public website. Anyone with a phone can scroll through those reviews. Why would one of the most capable AI tools around be shut out of the site?
So I looked into it. Behind Claude’s answer was a one-line error message: “Site disallows automated access.”
Okay, well now I was curious, so I went on a spree. I sent the exact same request to four AI setups that I use, and got four different responses.
ChatGPT’s chat mode couldn’t read the current page either. It gave me reviews from last January. It was working from an older snapshot and couldn’t verify the newest reviews. Nine months old. Useful background, maybe, but not quite what I had asked for.
ChatGPT’s chat answer.
Meanwhile, Codex, OpenAI’s work assistant, thought for 53 seconds and successfully handed me a table of the newest reviews.
I asked how it got the answer, and it said its direct web tool initially had been turned away, so instead it opened a Chrome browser and read the reviews from there.
It never mentioned the detour until I asked.
Codex explaining the browser detour, and what it couldn’t verify.
And lastly, my Grok bot returned a summary of the reviews: great food, great servers, book ahead.
But when I asked how it got in, it said nothing had stopped its browser. It had opened the page in a normal browser on its own computer, the same way I would on mine.
My Grok bot, Gardevoir, explaining its browser access.
So I had one refusal, one old snapshot, and two bots reporting fresh reviews through their browser access.
Websites can put up a sign
Think of the “No Soliciting” sign on someone’s front door. It tells salespeople what the owner wants, but it can’t actually stop anyone. There’s no lock behind it.
Websites have their own version, a short note written for robots. Yelp’s note asks the tools Claude and ChatGPT use to read web pages to stay out of its business and review pages.
Some sites also have a bouncer. My Grok bot said Yelp showed a “Verifying the device” check, although it let its browser through without intervention.
On top of that, the AI companies have different approaches to following rules, too.
Anthropic says its agent follows the requests.
OpenAI says the sign “may not apply” to certain visits when they’re made at a user’s request.
So it matters which AI you use, and what tools it has.
Some websites build a side door
Now picture the back of a restaurant. There’s a side door for deliveries, but you only get access to the rooms you need. The produce guy goes to the walk-in fridge, not the office.
In the same way, a connector can let an AI request information directly from a service, instead of reading it off a webpage.
A license is different. It’s a deal to use a site’s information, like Yelp’s reviews.
In fact, Yelp has a licensing agreement with OpenAI. Deals like that can give an AI service access to a site’s information without relying on a bot reading the public webpage. The site can set limits on what gets used and how. But that still doesn’t give every OpenAI tool a pass to open any Yelp page you send it.
What this means for you
When your AI does a job on a website, you usually see the answer, not the route it took. It might have used a side door, opened a browser, or worked from an old copy.
If a task fails, your AI might not be broken. The site may have said no, or the tool may not have access. A job that works today can stop working later, when a site changes its rules or an integration changes. And a confident answer can still be nine months out of date.
Next time, I’m asking: “How did you get this, how current is it, and what couldn’t you check?”
I shouldn’t have to play detective to find out what my AI actually read. Tell me up front.
THE 30-SECOND VERSION
Websites can post a “please don’t come in” sign for AI. The sign itself isn’t a lock, though sites sometimes add a bouncer.
Different AIs have different access rules and tools.
A connector can be a side door for a particular job. It doesn’t open every room.
Ask what your AI actually read, how current it was, and what it couldn’t check.
me+machine.
P.S. No matter what the reviews say, Cafe Pesto is still the best spot in Hilo.