The Question Is the Contract
Competency Questions for System Architectures
Every information system ever built shares a very important job—to satisfy questions. Questions are the reason the system exists. Build without questions in mind, you are left with a system unable to justify its existence.
Questions date back centuries, as the bedrock of most all Western and Eastern intellectual traditions — and just so happen to be largely ignored in modern system design.
Questions All The Way Down
Aristotle organized knowledge by asking what kinds of questions each domain was equipped to answer. Plato’s dialogues are structured around the pursuit of answerable questions — Socrates famously claimed to know nothing, only to ask. The philosophy of inquiry treats questions as prior to propositions—you cannot begin to know something until you have formulated what it is you are asking.1
R.G. Collingwood took this so far as to argue that propositional logic should be replaced by the logic of question and answer, in which neither the question nor its answer is the more fundamental unit, but both together.2 His provocation taps into the intuitive nature of questions as vehicles for inquiry—a necessary process from which knowledge can be organized, communicated, or retrieved.
The philosophical literature on questions is dense because philosophy as a discipline is about questions. Philosophy distinguishes question types, presuppositions, direct and indirect answers, and the conditions under which a question can be said to be resolved.3 Relevant for system designers, there is a simpler version of the same structure—a question defines a space of possible answers, and a system that cannot identify that space cannot reliably produce anything meaningful, to begin with.
Contemporary epistemologists describe the activity of inquiry as intentional and directed — not a passive acquisition of information but a deployment of capacities aimed at a specific epistemic target.4 Inquiry involves wondering, examining, deliberating, and acting on information once found. The question is not the end of that process, but the beginning and the framework.
Traditional System Design Methodologies
General system design methodology — whether the scalability-oriented framework that decomposes requirements into components and allocates them to hardware and software layers, or the human-centered design approach that begins with user empathy and iterates toward a solution through prototyping — treats the system as the primary object of design.56 The verification question present in both traditions is essentially behavioral—does the system perform its function, handle its load, solve its users’ problem?
Requirements are stated as capabilities — “the system shall” — and validated by testing whether the capability exists.7 Buchanan’s analysis of the relationship between systems thinking and design thinking identifies a key limitation of both. Systems analysis provides no clear identification of the specific problems designers must address; it describes the whole without specifying what the system must produce in response to any particular demand.8 The problem space remains open-ended. What counts as “done” is diffuse.
Information retrieval is different in kind, not degree. When a system exists to answer queries — whether from a human using a search interface, an AI agent traversing a knowledge graph or a retrieval layer deciding what context to surface — the output is an answer to a specific question, and that answer is either correct or it is not.9 The verification question is closed, not open.
And for these reasons, competency questions fit information retrieval system design, in a way that general requirements frameworks do not. A competency question is stated in order to guide the architecture of a system that is expected to answer the questions it was designed to answer. The competency question is a natural-language question with known correct answers, which means it can be evaluated before the system is built and re-evaluated every time the system changes. The design methodology and the acceptance test are the same artifact, which means the requirements of the system persist throughout all parts and pieces.
General system design frameworks do not introduce this type of consistency and rigor in design, approach, architecture and testing as most system frameworks, because they were built for different problems.
General system design frameworks — whether waterfall requirements engineering, agile user stories, or human-centered design — evolved to solve the challenge of building systems that do things: process transactions, render interfaces, route messages, allocate resources. The primary design question in those cases is behavioral. What should the system be capable of? The output is a function, a feature, a service. Whether that output is correct is largely a matter of whether it executes as specified.
Information retrieval has been treated, for most of its history, as a subspecialty of system design. In one framing it’s a library problem. In another framing it’s a database problem, then maybe it becomes a search problem — rather than being treated as a foundational, system design concern. The broader system design tradition absorbed it as one feature among many—“the system shall support search.” That framing immediately loses the thing that makes retrieval distinct, which is that the quality of the output depends entirely on the relationship between the question asked and the knowledge represented. A transaction either processes or it doesn’t. A query returns results that are relevant or irrelevant, complete or partial, correctly attributed or hallucinated — and those distinctions are invisible to a capability-based requirement.
There’s also a historical asymmetry in who was doing the theorizing. The people building systems frameworks — software engineers, systems architects — were not the same people studying query negotiation, relevance, and information need. Library science and information retrieval research developed a sophisticated theory of the question as a design object, but that literature stayed largely inside its own domain. It didn’t get absorbed into software engineering methodology the way, say, database normalization did.
The results are that every system built to answer questions — search indexes, knowledge graphs, RAG pipelines, agentic workflows — tends to be designed with tools that weren’t built for the job. They can specify that retrieval exists but they can’t specify what retrieval must return.
Ultimately, failure to persist questions throughout the entire system design and architect process, results in a system that is not designed for information retrieval and fails to answer questions accurately, if at all. General system design frameworks treat information retrieval as a game of slots—any part of the system can be blamed for its failure but it’s a gamble, at best.
