The AI engineering interview is a standard software engineering loop with additional rounds on LLM fundamentals, RAG architecture and evaluation design. This guide covers the questions that actually come up - organised by area - and what strong answers look like.
5-7
Typical rounds in an AI engineering on-site loop (coding, design, AI concepts, behavioral)
2-3
AI-specific rounds added on top of a standard SWE loop, depending on role seniority
Evals
The topic most candidates skip and most interviewers probe hard on in 2026
How the loop is structured
The AI engineering interview at most companies follows the same skeleton as a standard software engineering loop, with AI-specific rounds replacing or supplementing the domain-design round.
- 1
Recruiter screen
Filter20-30 minBackground, motivation and logistics. AI roles: expect a high-level question on your experience with LLMs, RAG or agents. Prepare a 90-second narrative of your most recent AI project. - 2
Technical phone screen
1-2 rounds45-60 minOne or two coding problems plus a conceptual question on LLM fundamentals or RAG. This is the biggest filter - most candidates pass behavioral but fail here on coding speed or LLM gaps. - 3
LLM / AI concepts round
Core45-60 minDeep questions on how LLMs work, RAG architecture, prompt design and evals. Interviewers test whether you have shipped production AI vs read about it. Working-system examples are your strongest asset. - 4
ML system design round
Senior+45-60 minDesign a production AI system - a RAG-powered search, an LLM copilot, a content moderation pipeline. Scored on handling ambiguity, making principled trade-offs, and knowing where models fail in production. - 5
Coding round(s)
2 rounds45-60 min eachStandard DSA problems - the bar is the same as a general SWE role at the same company. AI roles do not replace coding with AI questions; they add AI questions on top. - 6
Behavioral round
All levels45-60 minPast-behavior questions focused on ambiguity, speed, shipping under uncertainty and cross-functional influence. AI teams prize bias-for-action and comfort with non-determinism.
The added AI rounds, not a replacement
Most candidates underestimate the coding bar because they assume AI roles test only AI concepts. They do not - you need both. Budget your prep time accordingly: roughly 50% DSA + system design, 50% AI-specific knowledge.
LLM fundamentals questions
These questions test whether you understand how large language models actually work at the level needed to build reliable systems - not academic depth, but engineering intuition.
| Question | What a strong answer covers |
|---|---|
| How does a transformer generate text? | Autoregressive token generation, attention over the full context, softmax over vocab, temperature and sampling strategies (greedy, top-k, top-p). Mention that the model has no memory between calls. |
| What is a context window and what happens when you exceed it? | The fixed input length in tokens; what gets truncated (typically the beginning or middle); sliding window and chunking strategies; the cost and latency tradeoff of longer contexts. |
| What causes hallucinations and how do you reduce them in production? | Hallucination is confident generation not grounded in training data or context; causes include knowledge gaps, poor prompting, and ambiguous instructions. Mitigations: RAG, citations, constrained output formats, evals, retrieval confidence thresholds. |
| What is the difference between fine-tuning and prompting? | Prompting adjusts behavior at inference time without changing weights - cheap, fast, reversible. Fine-tuning updates weights for a specific domain or style - expensive, requires data, stronger for behavior change at scale. RAG is often the better choice before fine-tuning. |
| How do you handle a model that keeps ignoring your instructions? | Prompt structure (system message position, delimiters, explicit constraints), few-shot examples, format enforcement (JSON mode, function calling), and - if prompting is exhausted - fine-tuning. Always check with evals whether changes actually work. |
| What is tokenization and why does it matter for engineers? | The mapping of text to integer IDs; different models use different tokenizers (BPE, SentencePiece); token count determines cost and context usage; non-English text and code often tokenize less efficiently than English prose. |
RAG questions
RAG (retrieval-augmented generation) is now a core AI engineering primitive. Expect detailed questions on the full pipeline - not just "what is RAG?" but how specific components fail and how you fix them. See RAG Explained for a deep dive.
| Question | What a strong answer covers |
|---|---|
| Walk me through how you would build a RAG pipeline from scratch. | Chunking strategy (fixed vs semantic, chunk size, overlap), embedding model choice, vector store (HNSW indexing, similarity metric), retrieval (top-k, hybrid BM25 + dense), reranking, context assembly, prompt construction, and eval loop. Mention failure modes at each step. |
| How do you choose chunk size? | Chunk size is a tradeoff: too small loses context, too large dilutes relevance. The right answer depends on the retrieval unit needed, the embedding model's sweet spot, and empirical eval results. Semantic chunking (split at natural boundaries) often beats fixed-size for complex documents. |
| What is hybrid retrieval and when do you use it? | Combining dense (vector) retrieval with sparse (BM25/TF-IDF) retrieval, merged via reciprocal rank fusion or a learned router. Dense retrieval handles semantic similarity; sparse handles exact keyword matches and rare proper nouns. Use hybrid when recall is the priority. |
| How do you evaluate a RAG system? | Component-level: retrieval precision/recall, answer faithfulness, answer relevance. End-to-end: RAGAS framework or custom eval harness with human-labeled Q&A pairs. LLM-as-judge for scalable evaluation; human review for calibration. Always track retrieval metrics separately from generation metrics. |
| A user asks a question but the retrieved chunks are not relevant. What do you do? | Debug the retrieval step first: check the embedding model, query expansion, chunk quality, and whether the information exists in the corpus. Consider query rewriting (HyDE), metadata filtering, or a fallback to a broader retrieval pool. Add this failure case to your eval suite. |
Prompting and evals questions
Evals are where most candidates reveal whether they have shipped real AI systems or only prototypes. Interviewers at AI-native companies probe hard here.
| Question | What a strong answer covers |
|---|---|
| How do you test a prompt change before shipping it? | Build an eval suite: a labeled dataset of input/output pairs (golden set), metrics (exact match, BLEU/ROUGE for extraction, LLM-as-judge for open-ended), automated regression on each push. Treat prompt changes like code changes - they need a test before they merge. |
| What is LLM-as-judge and what are its failure modes? | Using a larger or equal-quality LLM to score another model's outputs on criteria like relevance, faithfulness, or helpfulness. Failure modes: positional bias (favoring first answer), verbosity bias (favoring longer answers), sycophancy. Mitigate with calibration, pairwise comparison, and human spot-checking. |
| How do you handle a prompt that works for 90% of inputs but fails on the remaining 10%? | First, characterise the 10%: are they a coherent cluster? Then: add few-shot examples covering that cluster, use a router to send hard cases to a stronger model, add guardrails and fallbacks, or fine-tune if the failure pattern is stable and large enough to justify the cost. |
| What is a system prompt and how do you keep it from being leaked? | The persistent instruction context prepended to every conversation. Leakage mitigation: explicit instructions not to reveal it, prompt injection guardrails, output filtering, not putting secrets in system prompts (they will leak eventually). Accept that determined users can often extract it and design accordingly. |
ML basics questions
AI engineering interviews at most companies do not require deep ML theory, but you need working knowledge of the concepts that come up when integrating models into systems.
| Question | What a strong answer covers |
|---|---|
| What is overfitting and how does it manifest when fine-tuning an LLM? | Overfitting: the model memorises training examples rather than learning generalisable behavior. In fine-tuning: the model outputs training verbatim, performance on out-of-distribution inputs drops, and eval metrics diverge from training metrics. Mitigate with smaller LR, early stopping, data augmentation, and more diverse training data. |
| What is an embedding and why does it matter for search? | A dense vector representation capturing semantic meaning - similar concepts cluster close in the embedding space. For search, embeddings enable semantic retrieval beyond keyword matching. Choice of embedding model (OpenAI text-embedding-3, Cohere, open models) affects quality, cost and latency significantly. |
| When would you use fine-tuning vs RAG vs prompting? | Prompting first (cheapest, fastest, reversible). RAG when the knowledge is too large or too dynamic for the context window. Fine-tuning when the behavior change is stable, prompt engineering has hit its ceiling, and you have labeled data to train on. Fine-tuning and RAG can combine. |
| What metrics do you use to evaluate a classification model in production? | Precision, recall, F1 - and which matters depends on the cost of false positives vs false negatives. AUC-ROC for ranking. Calibration (does P=0.8 mean 80% of cases are positive?). Business metric alignment: does the model metric actually correlate with the business outcome? |
ML system design questions
The ML system design round asks you to design a complete AI-powered system, not just the model component. Use a repeatable structure: scope and requirements - data and retrieval - model and serving - evals - monitoring and iteration. See System Design Fundamentals for the general framework.
Common prompts
- Design a RAG-powered customer support chatbot handling 10,000 queries per day, with a 500ms latency budget.
- Design a code review assistant that comments on GitHub PRs with real-time suggestions.
- Design a content moderation pipeline for a social platform processing 1M posts/day, with human-in-the-loop escalation.
- Design a semantic document search system over a 100GB corpus with sub-200ms p99 latency.
| Dimension | What interviewers want to see |
|---|---|
| Scoping | Clarify what success looks like before designing anything. Ask about latency, throughput, quality bar, human oversight requirements and budget. |
| Data and retrieval | How is data ingested, chunked, embedded and indexed? What changes over time and how do you keep it fresh? |
| Model selection | Which model and why - based on latency, cost, capability and data-privacy constraints. API vs self-hosted tradeoffs. |
| Evals | How do you know it is working? What is in the eval set, what are the metrics, and how do you prevent regressions? |
| Failure modes | What breaks first under load? What happens when the LLM hallucinations? How do you degrade gracefully? |
| Iteration | How do you improve the system after launch? What signals tell you quality is degrading? |
Coding round
The coding round for AI engineering roles uses the same DSA problems as a standard SWE loop. AI knowledge does not substitute for coding performance - it adds to it. The patterns that appear most in tech interviews are covered in LeetCode Patterns.
- Arrays, hash maps, two pointers - the most common category across all levels.
- Trees and graphs - BFS, DFS, and shortest path are standard at mid-to-senior level.
- Dynamic programming - appears more at senior+ level; recognising the subproblem structure is the key.
- String manipulation - common at companies with text and NLP products; parsing and sliding window variants.
AI-adjacent coding questions
Some AI-native companies include a Python coding question with a light ML theme - implementing a simple tokeniser, computing cosine similarity over vectors, or writing a basic BM25 scorer. These are rare but growing. The underlying skill tested is the same: clean, readable Python under time pressure.
Behavioral questions
AI teams prize engineers who move fast under uncertainty, can ship despite incomplete information, and know when to use judgment vs wait for data. The behavioral round tests for these specifically, on top of the standard impact/ownership/collaboration axes.
| Question | What a strong answer covers |
|---|---|
| Tell me about a time you shipped something that turned out to be wrong. | The decision, what you learned quickly, how you corrected course. AI teams value the ability to ship fast and iterate over paralysed perfectionism. Show a bias for action with a correction loop, not avoidance. |
| How have you handled a situation where the AI model behaved unexpectedly in production? | Specific failure mode, how you diagnosed it (evals, logging, user reports), what you changed (prompt, retrieval, model, guardrail), and what you put in place to catch it earlier next time. |
| Tell me about an ambiguous project where you had to define the problem as well as solve it. | Structured approach to ambiguity: what questions you asked, how you validated the framing, how you got stakeholder alignment before building. AI work is deeply ambiguous - show you can operate in that environment. |
| How do you decide when a prototype is ready to ship? | Your quality bar, your eval coverage, your monitoring plan. Not "when it's perfect" - but when the risk is bounded and measurable. Show you understand that shipping starts the learning, not ends it. |
How to prepare
The preparation split for an AI engineering loop is roughly:
- 50% coding and system design. The coding bar is identical to a standard SWE role. Do not underinvest here because the AI-specific content feels more interesting to prep. Use LeetCode Patterns to drill the 15 recurring shapes; use System Design Fundamentals for the architecture framework.
- 30% AI-specific concepts. LLM fundamentals, RAG architecture, prompt engineering and evals. Read Chip Huyen's blog and, if time allows, her book. Build one real RAG system with evals - the experience is worth more than 10 hours of reading.
- 20% behavioral preparation. Prepare 5-6 STAR stories that cover ownership, ambiguity, shipping under uncertainty, and a meaningful failure. These stories answer 80% of behavioral questions with minor framing adjustments.
tip
The fastest way to close the gap is realistic mock interviews with someone who actually runs these loops. Book a free call with a TopCoding mentor to map your prep and practice the AI-specific rounds before the real thing.
Sources & further reading
- 1Machine Learning Interviews Book — Chip Huyen, huyenchip.com
- 2RAG survey: the state of the art — arXiv
- 3LeetCode patterns and problem lists — LeetCode
- 4RAGAS: automated evaluation of RAG pipelines — GitHub / Exploding Gradients