Most of the AI training I see at work is about the tool. You learn how to write a better prompt, or how to get a cleaner draft out of a chatbot. I’ve designed that kind of programme myself, including a 24-week course for busy professionals built around short weekly practice sessions. People leave knowing more about AI. Many still can’t do the work skill their job has just handed them.
The use I care about more is AI as a tutor for that skill. I’ll use financial management as the example: budgeting and reading a profit and loss statement. Plenty of capable managers can run a team and still freeze when they have to build next year’s budget or explain a variance in a meeting. A chatbot can walk you through a P&L in clear language. Give it the whole job and it will produce the budget for you, and leave you unable to defend a single number once the laptop is closed.
A tutoring session and a chatbot doing the work can look the same on the screen.
Why fluent explanations mislead
A well-written explanation leaves you feeling that you understand the material. That is a poor guide to whether you could do the work.
In 2013, John Dunlosky and colleagues reviewed common study techniques and found that rereading and highlighting were of low usefulness, while practice testing and spacing study out over time came out best. Most of those studies involved students. The same sense of familiarity misleads adults at work too. You reread a note on gross margin, it feels obvious, and then you can’t explain it the next morning.
Leonid Rozenblit and Frank Keil described a related effect they called the illusion of explanatory depth. People rated their understanding of everyday mechanisms, such as a zip, quite highly. When they had to explain the steps, those ratings fell.
AI makes this easier to fall into, because the explanation is fluent and the output can look finished. A 2025 field experiment by Hamsa Bastani and colleagues, with high school students in mathematics, is the clearest warning I know. Students who practised with an unguarded chatbot did better while they had it. On a later test without the tool, they did worse than students who had never used it. A second version of the same model, designed to give hints and push students to think rather than hand over answers, largely avoided that drop. Those were school students doing maths, so I can’t quote that as a result about professionals. If you have used a chatbot at work and then been asked to do the same task in the room, with no screen, you have seen the same split between performance with the tool and skill without it.
Yizhou Fan and colleagues found something similar with university students on a writing task. The group using ChatGPT improved their essays more than the other groups during the session. On later tests of knowledge and transfer, they did no better. The authors warn about what they call metacognitive laziness: letting the tool do the planning, monitoring and checking you would otherwise do yourself.
What I do when I learn with a chatbot
When I am trying to learn a skill with a chatbot, I work in this way.
Try the task cold first
Before you ask for help, make a genuine attempt. If you’re learning to read a P&L, look at a sample statement and write down what you think the gross margin is telling you, or sketch the first cost lines of a budget from the figures you already have. Keep the attempt short. If you can’t start at all, study a worked example, then try again on a fresh one.
The way you ask still matters, and I have a short write-up of a session on the fundamentals of prompting. Ask for a hint on the attempt you have already made. Don’t ask the tool to hand you a finished spreadsheet you couldn’t rebuild.
Get the AI to quiz you
Explanations help after you have already attempted the task. If a session is only reading what the chatbot wrote, you haven’t practised.
Ask the tutor to test you: five questions on yesterday’s cost lines, or a short set of variance questions on a P&L it has not already explained. Close the explanation and answer from memory, then mark yourself.
Henry Roediger and Jeffrey Karpicke showed what that kind of practice does for memory. In their 2006 study, students who reread a passage did well on a test straight away. A week later, the students who had practised recalling the passage remembered much more. The rereading group was also more confident they would remember it. Their confidence did not match what they could recall a week later.
Explain it back, with the chat closed
Once or twice a week, close the chatbot and explain the idea as if you had to brief a colleague. Try putting into your own words the difference between a budget and a cash view, or why staff costs might fail to match the headcount list. The parts you can’t explain are what to practise next. This is Rozenblit and Keil’s test, applied to your actual work.
Check it against a real deliverable
Start by agreeing one real piece of work, early, and someone who can check it. For budgeting, that might be next year’s team budget, used in the real budget round, checked by a finance colleague who can see whether every figure traces to an assumption and whether the totals reconcile. For a P&L, it might be a short review memo for a monthly meeting.
Write down what “done” looks like in performance language, such as being able to explain the three largest variances on a monthly P&L. “Get better at finance” is too vague to check. If you need help naming the performance, I have a practical guide to Bloom’s Taxonomy for writing that kind of objective.
Because AI can already produce a polished budget or memo, you need a check that the tool cannot complete for you. I’ve written more about that in a piece on assessment when AI can produce finished work. Get a human to check the work, and do a short task without AI at the start and again at the end.
A 30-day version of this
I’m putting this method into a 30-day sprint and a book, The 30-Day Skill: How to Learn Any Work Skill With AI as Your Tutor, Without Fooling Yourself. You pick one work skill. On day 1 you agree one real deliverable with a checker. For 30 days, AI is your tutor and practice partner, under rules that stop it doing the learning for you. On day 30 the work is finished and signed off. You also do a short task without AI at the beginning and the end, so you can compare what you could do at the start with what you can do at the end.
The flagship example is the one above: budgeting and reading a P&L. The same structure will take other work skills, as long as you can name a deliverable small enough to finish in a month and a person who can judge the work.
This week, pick one skill your job needs, such as reading a P&L, and try a short task on it before you open the chatbot. When you have a draft, ask a colleague who knows the work to check it.
Discover more from Niall McNulty
Subscribe to get the latest posts sent to your email.