How to Tell Whether an AI SEO Tool Is Actually Doing the Work

Buying an AI SEO tool is unusually hard to get right, and the reason is specific. These tools rarely fail in a way you can see. They fail by returning something.

A rank check runs, reports no errors, and writes nothing to the database. A keyword tool returns two hundred phrases and forty of them describe a business that is not yours. A schema validator shows a green tick that was never connected to a validator. An audit exports a PDF button that produces an HTML file. In every case the interface says the work happened. Nothing in the interface says it did not.

Traditional tools failed loudly. A crawler that could not reach your site threw an error. AI tooling sits on top of models that always produce output, so the failure mode moved: instead of an error you get a confident, plausible, empty result. The evaluation problem is no longer "does it work." It is "is there anything behind this number."

Here are four tests. Each one takes minutes, each one can be run during a trial, and each one catches a different class of silent failure.

Test 1: Ask where a keyword came from

Pick one keyword out of the list the tool generated for you. Ask the tool which page of your site it came from, or which query in Search Console, or which competitor ranked for it.

A tool that researched your site can answer. It scraped your pages, it read your Search Console, it pulled a SERP, and the phrase is attached to the source it came out of. A tool that asked a language model to brainstorm keywords for your industry cannot answer, because there is no source. The phrase was generated, not found.

This matters more than it sounds. A generated keyword list looks correct. The phrases are real English, they are related to your category, and several of them have plausible volume estimates attached. What they are not is evidence about your site. You will build a content plan on them, publish against them, and measure nothing, because the demand was never there in a form connected to what you sell.

The version of this test we use internally: take the keyword list, and for each phrase check whether any page on the site contains it. If a large share of the list appears nowhere in your own content and nowhere in your Search Console, the tool generated those phrases rather than finding them. There is a legitimate case for a phrase you have never used, that is the point of keyword research, but it should be the minority of the list and the tool should be able to say what suggested it.

Test 2: Find out what "not ranking" is stored as

Every rank tracker has to store something when a keyword does not appear in the results. The column is usually not nullable, so the value is a placeholder. One hundred is the common choice, sometimes 101, sometimes zero.

Now look at any chart that averages position over time. If the placeholder is inside that average, the chart is not measuring rank. A keyword that was at 4 and dropped out of the top hundred appears as a 96 place fall. A keyword that was never measured at all reads as a catastrophic loss on its first check. Averaged across a site, the whole line drifts toward the placeholder value as coverage gets worse, which means the chart goes down exactly when the tool stops finding things, and that looks identical to a ranking decline.

How to test it in a trial: track a keyword you know your site does not rank for. Let it run one cycle. Look at what the position column says, then look at whether that keyword contributes to the site-wide average position shown at the top of the page. If the average moved toward 100, the tool is doing arithmetic on a value that is not a rank.

Leaving the results is real and worth telling you about. It is not a number of places moved, and a tool that reports it as one is manufacturing a figure.

Test 3: Check whether the validator can say no

Schema markup, meta tags, content scores, readability grades. Anything the tool presents as validated should be able to fail.

Feed it something broken. Give the schema generator an article with no author and no date. Give the content scorer an empty draft. Give the meta checker a title of four hundred characters. If the green tick stays green, it was never a validator. It was a piece of interface that renders next to the output regardless of the output.

This one is common because the honest version is expensive. Real JSON-LD validation means implementing the required and recommended property sets for each schema type and checking them. Rendering a checkmark costs nothing. From the outside the two are indistinguishable until you hand it something that should fail.

The same test applies to content scores. If a page scores 82 out of 100, the tool should be able to tell you what the 18 was deducted for and what the 82 was awarded for. A score with no visible arithmetic behind it is a number, not a measurement.

Test 4: Follow one item end to end

The largest gap in most AI SEO platforms is not inside any single tool. It is between them.

Pick a real piece of work and follow it. Take a keyword from the research surface into a brief, into a draft, into a published page, into the rank tracker. Count the times you copy something out of one screen and paste it into another. Count the times a piece of context you already provided, a brand voice, a target audience, a site, has to be provided again.

Most platforms are separate products behind one login. Each surface works. The seams are where the work leaks out, and the leak is invisible in a feature comparison table because every feature is present. What is missing is the connection between them, and you only see that by moving one item through the whole path.

A related check: whatever you generate, can you get it out. If the only way to move a draft into your CMS is to select it and copy, that is not an integration. If an export button produces a format other than the one it names, that is worth knowing before you build a workflow on it.

What automation is honestly good at

None of this is an argument against AI in SEO work. It is an argument for knowing which part of the job you handed over.

Automation is genuinely good at the parts that are wide and shallow. Crawling a site and checking every page against a fixed list of technical conditions. Clustering a few thousand keywords by semantic similarity. Drafting forty meta descriptions that respect a character limit. Reading a SERP and extracting what the ranking pages have in common. These are tasks where the work is repetitive, the correct answer is checkable, and a person doing it by hand is slower without being more accurate.

Automation is bad at deciding what matters. Which of the 71 issues the audit found are worth an engineer's afternoon. Whether a keyword with 90 monthly searches is worth a page because of who searches it. Whether a draft is true. Those decisions need someone who knows the business, and a tool that pretends to make them is producing confident output about a question it cannot see.

The useful split is that the tool should do the work and show the evidence, and the person should make the call. A tool that hides its evidence has quietly moved the call to itself.

How we build against this

Quietly is built by people who found every failure described above, most of them in our own code, which is why the tests are specific rather than general.

Keyword research is grounded in your pages and your Search Console, and each phrase carries where it came from. Rank tracking distinguishes a measured position from a keyword that was absent, and refuses to report a change between the two as a number of places. Content optimization shows five scores with the weight behind each one printed next to the number. The site audit files each of its 34 checks against the page the finding was found on, across five severities, so a finding is a location rather than a count on a summary card.

The claims on every feature page here are checked against the code that runs, and the pages that describe something we do not have say so.

Run the four tests on us. Start with the first one.