---
slug: how-to-design-technical-screening-question-set
title: "How to Design a Technical Screening Question Set (Without Being an Engineer)"
description: "A criteria-first method for building technical screening questions, expected-answer rubrics, constraint follow-ups, role-specific scenarios, and a scorecard without needing to be an engineer."
publishedAt: "Jul 18, 2026"
updatedAt: "Jul 18, 2026"
author: "Denys Muzyka"
readingTime: 13
tags:
  - Technical Screening
  - Interview Questions
  - Non-Technical Recruiters
  - Scorecards
  - Hiring Process
canonical: https://www.hireduce.cloud/blog/how-to-design-technical-screening-question-set
---
A useful technical screening question set is not a list of facts copied from the internet. It is a measurement tool: each question should collect evidence about a role criterion, each follow-up should test depth, and each score should mean the same thing across candidates.

You do not need to be an engineer to design that structure. You do need a technical partner to validate the role-specific bar, especially where correctness depends on architecture, security, data, or language-level expertise.

## Step 1: Define Criteria Before Questions

Start with five to eight signals that matter in the first 30–40 minutes. Write observable criteria, not technology nouns. "Knows AWS" is a topic. "Can explain how they diagnosed a production reliability issue using logs and metrics" is a criterion.

1. Ask the hiring manager what a person must do in the first six months.
2. Identify the costly failure modes: poor debugging, weak ownership, unsafe changes, unclear communication, or missing domain fundamentals.
3. Separate must-pass criteria from evidence that can wait for the specialist round.
4. Write one observable behavior for each criterion.
5. Ask a technical reviewer to correct the bar and flag anything a non-engineer should not score alone.

| Vague requirement | Observable screening criterion | Leave for specialists |
| --- | --- | --- |
| Strong backend engineer | Can sequence a production API investigation and explain rollback tradeoffs | Detailed distributed-systems architecture |
| Good at React | Can isolate whether a UI defect comes from state, browser behavior, network, or API data | Code quality and framework-specific implementation |
| Excellent with SQL | Can explain how joins, filters, grain, nulls, and freshness produce a wrong metric | Query optimization on a real schema |
| Senior mindset | Names constraints, risks, stakeholders, and evidence used to make a decision | Depth of architecture judgment |

## Step 2: Use a Four-Part Question Architecture

1. Context question: establish what the candidate personally worked on.
2. Scenario question: present a realistic problem tied to a must-pass criterion.
3. Constraint follow-up: change one fact to test whether the reasoning adapts.
4. Reflection question: ask what evidence, tradeoff, or lesson changed the candidate's decision.

Example: "An API started returning errors after a deploy. Walk me through your first 15 minutes." Follow with: "What changes if you cannot access application logs?" Then: "What evidence would make you roll back instead of continue investigating?" This sequence reveals prioritization and adaptability without asking the recruiter to judge code.

## Step 3: Write the Expected-Answer Rubric

Never give interviewers a question without an expected-answer rubric. Strong, Partial, and Weak should describe evidence—not confidence, charisma, accent, or whether the answer matches one preferred phrase.

| Criterion | Strong | Partial | Weak |
| --- | --- | --- | --- |
| Debugging sequence | Starts with impact and recent changes; forms hypotheses; checks evidence; includes mitigation or rollback | Names useful checks but order or decision points are unclear | Lists tools or buzzwords without a sequence |
| Ownership | Separates personal actions from team actions and explains the outcome | Describes the team result but needs prompts to clarify personal contribution | Cannot identify what they personally decided or delivered |
| Tradeoff reasoning | Names at least two options, relevant constraints, risk, and why one option fit | Names a tradeoff but does not connect it to evidence | Presents one approach as universally correct |
| Communication | Explains the situation accurately in plain language and checks understanding | Mostly clear after prompting | Uses jargon in place of an explanation |

Include an "insufficient evidence" option. It is different from Weak. A dropped call, ambiguous prompt, or skipped follow-up should not become a negative judgment about the candidate.

## Step 4: Build Constraint Follow-Ups

A polished first answer may be rehearsed. Change one constraint and listen for adaptation. Do not pile on random difficulty; every follow-up should test a criterion.

- Missing evidence: "What if the logs are unavailable?"
- Scale: "What changes if traffic increases tenfold?"
- Time pressure: "You have 15 minutes before peak usage. What do you do now?"
- Risk: "What could make the seemingly fastest fix unsafe?"
- Stakeholder conflict: "Product wants to ship today; engineering wants another week. How do you frame the decision?"
- Communication: "Explain the issue to a customer-support lead in two sentences."
- Ownership: "Which part did you personally decide, and what did someone else own?"

## Role Examples You Can Adapt

| Role | Scenario question | Constraint follow-up | Evidence to capture |
| --- | --- | --- | --- |
| Backend engineer | A previously healthy endpoint becomes slow after a release. How do you investigate? | Metrics look normal, but customer reports continue. What next? | Scope, recent changes, hypotheses, observability, mitigation |
| Frontend engineer | Checkout works in one browser but intermittently fails in another. How do you isolate the cause? | You cannot reproduce it locally. What evidence do you request? | Reproduction, browser and network checks, logging, fallback |
| Data analyst | A dashboard total disagrees with the finance export. How do you reconcile it? | Both queries look correct. What assumptions do you compare? | Grain, joins, filters, time zones, freshness, definitions |
| DevOps / platform | Deployments succeed, but error rates rise several minutes later. How do you respond? | Rollback reduces errors but creates a data-compatibility risk. What now? | Impact control, telemetry, rollback judgment, coordination |
| Engineering manager | Two teams disagree over ownership of a recurring production failure. How do you move forward? | The fastest short-term fix increases long-term operational load. How do you decide? | Accountability, facilitation, tradeoffs, measurable follow-through |

These prompts are starting points, not answer keys. A hiring manager should adapt them to the actual stack, seniority, risks, and operating environment.

## Avoid Trivia Unless Recall Is the Job

Trivia is attractive because it is easy to score. It is often weak evidence of job performance because candidates can memorize definitions while experienced practitioners may look them up in normal work.

- Replace "Define database normalization" with "A table is producing duplicate customer records after a join. How would you investigate?"
- Replace "Name React lifecycle methods" with "A page refetches data repeatedly and becomes slow. How would you isolate why?"
- Replace "What is Big O?" with "This process slows sharply as the dataset grows. What would you measure before changing it?"
- Keep factual recall questions only when immediate recall is genuinely safety-critical or central to daily performance.
- Do not treat one unfamiliar term as a rejection when the candidate demonstrates the underlying concept.

## Step 5: Create a Simple Scorecard

| Field | What to record | Rule |
| --- | --- | --- |
| Criterion | One of the pre-agreed role signals | Do not add criteria after hearing an answer |
| Rating | Strong / Partial / Weak / Insufficient evidence | Use the written anchors |
| Evidence | A short factual note or candidate example | Avoid personality labels |
| Follow-up result | How reasoning changed under the constraint | Score adaptability, not speed |
| Confidence | High / Medium / Low | Low confidence triggers review, not forced certainty |
| Next step | Advance / Review / Do not advance | Apply must-pass rules agreed in advance |

Score each criterion immediately after its question while the evidence is fresh. Make the overall recommendation only after reviewing every must-pass criterion. A numeric average can hide a critical weakness, so do not let four easy strengths cancel one essential failure.

## Question-Set Quality Checklist

- Every question maps to a written role criterion.
- Every criterion has Strong, Partial, Weak, and Insufficient Evidence anchors.
- At least one follow-up changes a meaningful constraint.
- The set fits the scheduled time with room for candidate questions.
- Prompts test work-like reasoning rather than obscure recall.
- A technical reviewer has validated expected evidence and specialist-only boundaries.
- The same core prompts and scoring anchors are used for comparable candidates.
- Accommodation and alternative-format paths are documented.
- Interviewers know what data may be recorded and how long it is retained.
- The team reviews pass-through, downstream outcomes, and false-negative samples.

## Where a Live Copilot Fits

A document or spreadsheet is enough to run this process at low volume. When recruiters manage several roles, a live copilot such as Hireduce can surface the prepared criteria, suggest relevant constraint follow-ups, and produce a structured handoff during a human-led call. The tool should operationalize a validated question set, not invent the hiring bar.

For more prompts, see "50 Questions Recruiters Should Ask Software Engineers." For a broader evaluation framework, see "How to Evaluate Technical Candidates Without Being an Engineer." Both are linked in the sources below.

## A 60-Minute Setup Workshop

1. Minutes 0–10: hiring manager names first-six-month outcomes and common failure modes.
2. Minutes 10–20: group selects five to eight observable screening criteria.
3. Minutes 20–35: write three scenario questions and one constraint follow-up for each.
4. Minutes 35–50: define Strong, Partial, Weak, and Insufficient Evidence anchors.
5. Minutes 50–55: mark which judgments require a specialist.
6. Minutes 55–60: assign an owner to pilot, review candidate evidence, and revise the set.

Pilot the question set with a small cohort and ask specialist interviewers which first-call evidence predicted useful depth. Revise the rubric—not the standard for one favored candidate—and version the question set so future comparisons remain meaningful.