Hire AI engineers against the work they will own: a product, an evaluation system, a model adaptation task or the infrastructure that keeps it running. A list of models and frameworks tells you little about whether someone can make that work dependable.
The interview should reveal how the candidate defines success, investigates failure and balances quality with cost and response time. It should also establish what they personally did. A useful demonstration is one they can explain, challenge and improve when the conditions change.
Start with our distinction between AI engineers, research engineers and research scientists. Then choose a small number of assessment areas that match your actual role. The following framework is Corellia Talent’s recommended approach.
Separate the different jobs inside the title
An AI engineering assessment should begin with a concrete description of the output. Otherwise, you can accidentally test research knowledge for a product role or application development for a model-training role.
| Work the person will own | Evidence to explore |
|---|---|
| Product features built on model APIs | User requirements, application code, retrieval, permissions and fallbacks |
| Evaluation and quality | Test cases, grading decisions, failure analysis and release criteria |
| Fine-tuning or model adaptation | Training-data quality, baselines, held-out evaluation and deployment decisions |
| Model infrastructure | Serving, capacity, observability, resource constraints and recovery |
Prompting belongs in the conversation, but it is one component of a working system. Context, retrieval and model adaptation address different problems. The official model-accuracy guidance describes a practical distinction between improving the information available to a model and improving how it behaves. Use that distinction to ask why a candidate chose an intervention, rather than awarding points for its complexity. [Source 1]
For a mixed role, state the balance explicitly. If the first priority is shipping a customer feature, do not let an impressive training project obscure weak application engineering. If you need original research, do not treat an API integration as equivalent evidence.
Start with one system they understand deeply
Ask the candidate to choose a project they can discuss without disclosing confidential material. Follow one user request through the system: what arrives, what happens, which data is accessed, what the model returns and what the user can do next.
Then ask what they owned. Did they write the retrieval service, design the evaluation, tune the model, integrate the interface or operate the service? Listen for decisions and boundaries, including work done by colleagues.
Useful follow-ups include:
- What was the simplest version, and why was it insufficient?
- Which failure changed your view of the design?
- What did you measure before and after the change?
- What would you keep if your budget or response-time allowance fell?
- What did you decide not to automate?
A strong answer can include an unsuccessful experiment. What matters is whether the candidate can connect evidence to a decision. Someone who can describe a polished demonstration but cannot explain a failure needs further assessment.
Give evaluation its own exercise
An evaluation exercise should test whether the candidate can turn a vague quality goal into observable checks. Give them a small, fictional product brief and a handful of representative inputs. Ask them to identify the important behaviours before proposing metrics.
For example, an internal policy assistant must answer from permitted documents, recognise when evidence is missing and avoid taking action on instructions hidden in retrieved material. Ask the candidate to propose cases for ordinary questions, conflicting documents, changed policies and requests outside the user’s access.
Have them explain how each case would be graded. Exact matching may suit a structured field; a richer answer may need a rubric and human review. An agent that changes state also needs checks on what actually happened, not only the final sentence it produced. Primary guidance on agent evaluation makes this distinction between a run’s transcript and its outcome. [Source 2]
The candidate should explain how they would keep the evaluation useful as the product changes. Look for a process that captures failures and checks regressions, not a fixed score chosen because it looks reassuring.
Debug a failure together
A short debugging session reveals how someone works when a model-backed feature produces a plausible but wrong result. Supply a prepared trace with the input, retrieved context, tool calls, output and relevant timings.
Choose a failure that reflects your role. It could be retrieval returning the wrong document version, a tool being called twice, a permission check missing from a data path or a timeout that leaves the user uncertain about whether an action completed.
Ask the candidate to locate the failure before changing the prompt. Then ask what evidence would confirm their diagnosis, what the smallest safe fix is and what regression test they would add.
Keep a stable version of the exercise for comparison between candidates. Agree in advance whether AI assistance is allowed and use the same rule for everyone. If it is allowed, assess how they inspect and validate the output. If you need an unaided discussion, explain which part and why.
Test cost and latency as product decisions
Cost and latency questions should begin with a user need. A background document-processing job and an interactive assistant have different requirements. Ask what must happen immediately, what can be deferred and what should happen when a dependency is slow.
Give a fictional constraint rather than demanding a universal target. For instance: the user needs an acknowledgement quickly, some requests need a detailed answer and others can be routed to a simpler path. Ask the candidate to identify measurements, alternatives and failure modes.
They might discuss model selection, output length, caching, parallel work, batching or reducing unnecessary calls. The useful signal is the reasoning. What changes quality? What risks stale data? What creates a new operational dependency? Which metric tells them whether the change helped?
Also ask about the cost of failure. A cheaper answer that causes repeated user retries or manual correction may be a poor product decision. A candidate should be able to discuss that trade-off without reaching for a larger model as the default answer.
Make autonomy a deliberate choice
Agent design should test judgement about control, not enthusiasm for autonomy. Give the candidate a workflow involving an external action and ask where deterministic code, a model decision and human approval belong.
Primary engineering guidance distinguishes predetermined workflows from agents that decide their own sequence of actions. It also recommends beginning with simpler designs when they can do the job. [Source 3]
In the interview, ask what the system may read, what it may change and when it must stop. How would the team inspect an unsuccessful run? How would a user recover? What happens if instructions inside a document conflict with the task?
The candidate’s answer should match the consequences of the action. A suggested draft and an irreversible external change do not need the same controls.
Use a scorecard that can support a decision
A scorecard should record evidence against the role’s priorities. Choose a small set of dimensions, such as problem framing, implementation, evaluation, operational judgement and communication. Describe what a sufficient answer looks like before the interview.
Record observations rather than impressions. “Identified the missing permission boundary and proposed a regression test” is useful evidence. “Seems strong on agents” is difficult to compare or defend.
Have interviewers score independently before discussing the candidate. Where evidence is missing, choose one focused follow-up rather than adding a general extra round. Explain the remaining question to the candidate so the conversation has a purpose.
For the wider brief, use founding engineer role definition. For scope and pay, use the salary guide alongside the actual responsibilities. The hiring decision should connect the work, the evidence and the conditions the person will join.
Finish with a realistic working conversation
The final conversation should let the candidate assess your team’s readiness. Explain the current product, available data, evaluation habits, access to users and the support around the role. Be clear about what already works and what they would need to establish.
Ask what they would investigate first and which assumptions they would challenge. You are looking for someone who can improve the system and the team’s understanding of it. A well-designed process makes that visible without turning the interview into a catalogue of fashionable tools.
If you are defining a technical hire, Start a conversation.
Sources
- Model accuracy: context, evaluation and optimisation, official technical guidance, checked 31 October 2026.
- Agent evaluation: transcripts, outcomes and grading, primary engineering guidance, 9 January 2026.
- Workflows, agents and simple designs, primary engineering guidance, 19 December 2024. Tooling has evolved; the draft uses the architectural distinction, not a current product feature claim.
The assessment sequence, fictional exercises and scorecard recommendations are Corellia Talent’s editorial framework.