If you have sat through several AI-tutor demos, you may share the view of one r/edtech commenter: "They're all generic wrappers." Another said of one tool that "it feels very very generic." The category is crowded, and the demos all look impressive for the first five minutes. What separates a tool worth buying from one that will sit unused by February is rarely visible in a demo, because a demo shows the best case on a topic the vendor chose.
This scorecard helps you look past the demo. It covers six areas, each scored from 0 to 3, with questions to put to vendors and a template at the end. Adapt the weights to your priorities. We make Tutor AI and hold ourselves to the same questions. The scoring is simple: 0 means no evidence or a failure, 1 is weak, 2 is adequate, and 3 is strong with proof.
1. Pedagogy: does the tool teach, or just answer?
The first question is what the tool does when a student asks for the answer. A tool that immediately supplies it may help students finish work while doing little for learning. Ask whether it asks questions first and gives hints in stages, how it checks that a student understood, and whether you can see example sessions on your own content.
Why it matters: in a randomised trial with nearly 1,000 high school maths students (PNAS), an unrestricted AI helper improved practice scores by 48% but produced 17% lower results on an unaided exam, while a hint-based version avoided the harm. Test this yourself in the demo by typing "just give me the answer" and seeing what happens.
2. Evidence: what shows it works?
Ask what independent evidence exists that the tool improves learning, what outcomes were measured, on whom and over what period, and what the results were on usage as well as learning. Ask, too, whether there are any negative or null results, because a vendor that has run honest studies will have some.
Be cautious of vendor-authored case studies without comparison groups. Varsity Tutors for Schools shut down in 2026 after districts demanded more proof of learning gains, according to The 74. It is fair to ask for proof.
3. Student engagement: will students use it?
Ask what share of students with access use it weekly, how a session starts, and what happens after week three. Access is not participation, so ask for usage data from comparable customers, not from a flagship school with an enthusiastic champion.
4. Teacher control and visibility
Ask whether teachers can set rules per class, for example no direct answers, whether they can see what students did and where they struggled, whether content can be aligned to your curriculum and materials, and whether features can be turned off. Teachers in forums repeatedly name control and curriculum alignment as what separates useful tools from generic ones: for example, spaces created by the teacher that students enter, and a tutor "aligned to what the schools are using in class."
5. Safety, accuracy and privacy
Ask what data is collected, stored and shared, and who the subprocessors are. Ask whether student data is used to train models, what compliance the vendor meets, for example FERPA, COPPA or your local equivalents, how inaccurate or inappropriate outputs are handled and reported, and whether there is a data-deletion process. Our student data privacy checklist goes through these questions in more detail, and a written answer to each is a reasonable thing to require.
6. Cost and sustainability
Ask for the total cost, including training and integration, and what happens to pricing at renewal. Ask about the vendor's funding position and continuity plan, since several edtech tools have closed or been dropped as funding shifted, so ask about exit and data export. And ask whether there are free tiers or pilots, which lower the risk of finding out too late.
How do you run a demo that tells you something?
A vendor demo shows the best case, so take control of it. Bring five tasks of your own and ask the vendor to run them live: a question from your curriculum in your class's vocabulary, a student who types "just give me the answer," a student who gives a wrong answer with confidence, a student who writes only "I don't understand," and a request the tool should refuse or redirect. Then ask to see what a teacher sees afterwards. If the vendor cannot show a realistic student session and the teacher's view of it, you have learned something important.
How should you read the evidence a vendor gives you?
When a vendor shares a study, check five things. Was there a comparison group of similar students who did not use the tool? How large was the sample, and did the setting resemble yours? Was the outcome measured by an independent test or by the vendor's own quiz? Was the time frame a week or a year? And who ran the study and who paid for it?
Be equally interested in usage. Khan Academy itself has said that only about 15 percent of students with access to Khanmigo regularly use it (EdTech Innovation Hub), and in a two-year randomised trial in Tennessee, students mostly stopped using the tutor when it asked questions instead of giving answers (Hechinger Report). A tool that is available is not the same as a tool that is used, so ask every vendor for weekly-active figures from schools like yours.
Who should be in the room?
Do not leave the decision to one person. A useful review group includes teachers who will use the tool, to judge whether it fits their lessons, and someone responsible for data and privacy, to work through the questions in area five. Add a budget holder to check total cost and renewal terms, and, if you can, students, because a short trial with real students shows things a demo hides.
What does the scorecard look like?
| Area | Weight | Score (0–3) | Notes |
|---|---|---|---|
| Pedagogy | 25% | ||
| Evidence | 15% | ||
| Engagement | 15% | ||
| Teacher control | 20% | ||
| Safety and privacy | 15% | ||
| Cost and sustainability | 10% |
Any score of 0 in safety or privacy is a stop, whatever the total.
What red flags should stop you?
Watch for a demo that never shows a student struggling, since real students get stuck and a demo that avoids it is hiding something. Claims of "personalisation" with no specifics, no independent evidence beyond testimonials, an inability to answer the data questions clearly, and locked-in pricing or no export are all reasons to slow down.
What should a pilot measure?
A pilot should answer a few concrete questions. What share of invited students use the tool each week, and does that hold after week three? How do students who use it compare with similar students who do not, on an unaided task? What do teachers say about time saved or lost? Were there any safety or accuracy incidents, and how did the vendor respond? Agree the measures and the decision rule before the pilot starts, because if you decide afterwards what counts as success, it is easy to find it.
Should you pilot before you sign?
Yes. Combine the scorecard with a short pilot on your own students and content. See our six-week pilot plan for metrics and a week-by-week schedule. A tool that scores well on the scorecard and then fails in a pilot has still saved you a contract.
The short version
Look for a tool that teaches, that students use, that teachers can steer, and that you can trust with data. A polished demo covers only the first of those.
Frequently asked questions
How many vendors should we compare? Three is usually enough. More than that and the comparison becomes a chore, and the differences that matter show up quickly.
Should price be the deciding factor? It should be one factor. A cheap tool that students do not use, or that cannot protect their data, is expensive in the ways that matter.
What if a vendor will not share usage data? Treat it as a warning. Ask again for anonymised figures from comparable schools, and weigh a refusal against the score for engagement.
Do free tools need the same scrutiny? Yes. Free tools still handle student data, and a free tool that disappears has the same effect on your plans as a paid one.