Interviewthepeopledoingthemanualworkbeforeyouautomateit

At a law firm, the lawyers were already using AI to read case files, each in their own chat, with a set of instructions written by the firm itself. When I set out to build the legal AI agent for them, the question was which part of that work to automate first.

I didn’t ask how they wanted the product. I documented how they already worked, from the screens of a real work session. From that picture came the requirements that most often get lost in a migration: the binary conclusion (“would I file suit today?”), the documentation gap paired with the defense it opens up for the other side, and the obligation to declare what couldn’t be read. None of them would have come out of “what would you like the system to do?”

The second round came with the product live, when I brought a list of open items to the lawyer who uses it. His feedback changed rules I thought were settled.

One of them was small and embarrassing. The case pipeline had a yellow alert, and the on-screen caption said “7 business days.” The code counted 10 calendar days. The person who caught the difference was the one who works with deadlines every day. Another was more fundamental: the legal opinion he used as his quality benchmark was deliberately conservative in its prognosis. If the machine came out more assertive than he was, that wasn’t an improvement. It was an error to fix. That sentence became an evaluation criterion for the system.

None of this would have come up if I’d asked “what do you think of the product?” It came up because the conversation was about his work, with his material on the table. This article is about how to spark that kind of conversation with the people doing the manual work you want to automate: who to talk to, what to ask, what not to ask, and what to do with your notes.

What an interview is for

Interviews are good for understanding past behavior, context, and motivation: how the person solves the problem today and what they’ve already tried.

Interviews are bad for:

• Predicting future behavior. “Would you pay for this?” gets you a cheap yes.
• Validating usability. To find out whether someone can use a screen, put the screen in front of them and watch.
• Measuring. A handful of conversations shows patterns and hypotheses, not percentages.

For anyone looking to automate, that has a practical consequence. What’s worth automating first is what the person does today, the way they do it, at the cost it has. Not what they imagine a system could do. That’s why the interview looks backward, not at the product.

Recruiting: who you listen to decides what you learn

If recruiting brings in the wrong people, the best interview guide in the world won’t save you. The criteria I consider most important:

• The people who do it, not the people who ask for it. The manager knows the work exists. The person doing it knows where it gets stuck, and that’s where the automation will or won’t work.
• Behavior, not demographics. “Filed a debt enforcement action in the last month” filters better than “lawyer aged 30 to 45.”
• Recency. Someone who lived the problem last week remembers the details.
• Avoid your favorite user, and include people who gave up. The first one says little about the average; the second explains what no active user can.

A product built for a single law firm has an obvious limit here: the user base is small, and one person holds much of the knowledge. That makes it even more important to separate what they do from what they say they do, because there’s no second sample to break the tie.

The interview guide

The guide is a list of topics, not a questionnaire read out loud. For 30- to 45-minute conversations:

1. Opening (2 to 3 min). Who you are, what the conversation is for, asking permission to record.
2. Context (5 min). The person’s role, their routine, where the topic fits into their day.
3. The last time (15 to 20 min). Reconstruct, step by step, the last real situation where the problem came up.
4. Attempts and costs (5 to 10 min). What they’ve tried, what they use today, what made them switch.
5. Closing (3 min). “Is there anything I should have asked and didn’t?”

With a product already in use, “the last time” takes a better form: the last real case. The open items I brought to the lawyer weren’t abstract questions about the product. They were concrete situations from the pipeline, and he answered each one with what he does in practice.

Ask about the past, not the future

The Mom Test, by Rob Fitzpatrick, sums up the idea: your mom will tell you your idea is great, because she loves you. A good question is one that even your mom can’t answer with a kind lie. Three rules:

• Talk about their life, not your idea.
• Ask about specific situations in the past, not future intentions.
• Talk less and listen more.

In practice, the difference is in the wording. The examples below are illustrative:

Bad question	Why it’s bad	Good question
Would you use a system that drafts the petition?	It asks for a prediction, and the polite answer is yes	Show me the last petition you filed. Where did each part of it come from?
Do you think this would be useful?	It asks for an opinion on your idea	How do you handle this today?
How much would you pay for this?	A hypothetical price isn’t a price	Do you already pay for a tool for this? Which one, and why?
Do you have trouble keeping track of deadlines?	It suggests the answer and fits a yes or no	What was the last deadline that got tight? How did you find out?
What would you like to exist?	It hands the design over to the interviewee	What did you do the last time this went wrong?
How often do you do this?	It produces a made-up average	When was the last time? And the time before that?

The most useful question is almost always some version of “tell me about the last time…”, or, even better, “show me.”

The request hides the need

A variation of the same problem: the ready-made request. Early on, the firm asked for “the agents to learn.” Taken literally, the request would lead to training a model. I researched that path and it didn’t add up: not offered for current models, expensive to maintain, and risky with client data.

Behind the request was a different question: how does the system get better at the way we work? That one had a different answer. The model stays fixed and the context evolves: short lessons learned from the firm’s corrections, proposed by the machine and promoted by a partner. Treating the request as a hypothesis, not a requirement, is what opened up that alternative.

The biases that most often ruin a conversation

Confirmation bias

You show up with a hypothesis and pay more attention to what confirms it. I lived through the automated version of this bias: an automated audit of the project flagged as a critical defect the working page the draft carries at the end (“delete before filing”). By the firm’s standard, that page is exactly what the house requires. The audit’s criterion was my assumption, not his practice. The fix was to keep the page and only clean the jargon out of the body.

The antidote is to write down your hypothesis before the round and actively look for what would disprove it.

Leading questions and courtesy bias

“Wouldn’t it be easier if this were automatic?” already contains the answer. And compliments, enthusiasm, and “I’d definitely use it” are the weakest signal in an interview. Short, neutral questions (“how did that go?”, “what happened next?”) and steering back to behavior solve both.

During the conversation

• Two people, separate roles. One leads, the other takes notes.
• Silence. After an answer, wait three seconds before the next question.
• Ask them to show you. A filing, a spreadsheet, a real document are worth more than any description.
• Write down their word, not yours. Vocabulary is data.

That last point had a direct effect on the product. After the lawyer’s manual, the interface changed its vocabulary: “Aprovar” (approve) became Homologar (the formal sign-off term he uses), “briefing” became Orientação para a peça (guidance for the filing), and the analysis types got the names he uses. The filings started following language rules only a practitioner makes explicit: the parties’ names change depending on the type of action and are never mixed, statutes are always cited in full, and it’s “contrarrazões,” never “contraminuta” (two Brazilian terms for a response brief; only the first is right in his practice).

The same principle applies outside the law. At a lubricant distributor, orders came in as a photo of a handwritten note on WhatsApp, and someone retyped everything on the other end. The starting point wasn’t asking buyers how they wanted to order, but describing what the operation already did. The system kept the gesture of taking a photo, removed the retyping, and learned the counter’s vocabulary: “balde” (bucket) means 20 liters, “tb” means a drum.

Synthesize right after each conversation

Memory of a conversation degrades fast. Set aside 15 minutes right afterward for a short synthesis, separating facts (what the person did and told you), rules (what they stated as mandatory), vocabulary, and open questions.

Facts 3

• The caption promised 7 business days; the system counted 10 calendar days
• A field I expected in the batch spreadsheet is entered in another system
• The “delete before filing” page at the end is the firm’s standard

Rules 3

• A machine more assertive than the lawyer is an error, not a refinement
• When in doubt, a summary collection action, with the reason
• An appeal waived on stated grounds never becomes a filing

Vocabulary 3

• Aprovar → Homologar (sign off)
• Briefing → Orientação para a peça
• Contrarrazões, never contraminuta

Open questions 3

• The parties’ names in a repossession action
• The amount in controversy for this type of action
• Does the standard closing change by type of filing?

The open questions didn’t stay in a notebook. They were written into the very instructions that draft that type of filing, so the next person to touch them knows the rule hasn’t been validated yet.

The full cycle

An interview isn’t an isolated event. It’s a short cycle that repeats as long as each round still teaches you something new.

1. 01Bring the real thing

   The user’s own cases, screens, and documents, not a presentation.

2. 02Ask about practice

   What do you do in this case? Why does this page exist?

3. 03Synthesize

   Facts, rules, vocabulary, and open questions, the same day.

4. 04Return it in the product

   Each rule becomes behavior; each question is noted where it will be used.

What I’d do differently

• Bring the product to the user earlier, with their own material. The caption error and the audit error only showed up when the lawyer held the system up against his own practice. A round like that earlier would have cost less.
• Ask “why does this exist?” before calling something a defect. The working page at the end of the draft was a decision by the firm. I classified it without asking.
• Check the user’s documents the way I check my own. His manual was the best source in the project, and it still had two inaccuracies. Good sources need verifying too.

Checklist for the next round

• Does screening filter by recent behavior, not by profile?
• Does the conversation start from a real case, document, or screen from the user?
• Were the team’s hypotheses written down before the conversations?
• Does the guide ask about the last time, not about what the person would do?
• Were ready-made requests (“I want it to learn”) treated as hypotheses?
• Is there someone just for taking notes, and were the user’s words written down verbatim?
• Was what the person says they do checked against what they actually produce?
• Was each open question logged where the answer will be used?

If the team walks out of the round with a wish list and no concrete stories of work actually done, the interviews didn’t fail for lack of people. They failed at the question.