In 2014, Amazon built an experimental hiring tool. Feed it a stack of resumes, and it would score each candidate from one to five stars. The model was trained on ten years of the company’s own recruitment data. It had more information, more processing power, and more historical depth than any human recruiter could bring to bear.
By 2015, the engineers noticed a problem. The system was systematically downgrading resumes containing the word “women’s” — as in “women’s chess club” or “women’s” in the name of an all-women’s college. It preferred candidates whose resumes used language patterns common among the company’s existing engineering workforce, which was overwhelmingly male. The model had not been told to discriminate. It had been told to find good candidates, and it optimised the only proxy it was given: people who looked like the people Amazon had already hired.
Amazon shut the tool down in 2018. The failure cost years of engineering effort and produced a public relations problem. But it illustrates something more interesting than bias. The organisation asked the wrong question, gave the model the wrong target, and received a confident, precise, and wrong answer. More AI did not make the decision smarter. It made the wrongness crisper.
The Bounds You Can See
The economist Herbert Simon worked this out in the 1940s. Human beings and the organisations they form, Simon argued, are not optimisers. They are satisficers. We do not search for the best possible decision. We search for a decision that is good enough, then we stop.
Simon called this bounded rationality. The reasoning is structural, not psychological. Three constraints make true optimisation impossible: we never have complete information, our minds cannot process every alternative, and we never have enough time. Faced with these limits, we set an aspiration level, a threshold of acceptability, and take the first option that clears it.
This is not laziness. It is the only rational response to a world where the cost of searching for the optimum often exceeds the value of finding it. A manager who spends three months identifying the absolute best supplier has lost more in delay than the marginal supplier quality is worth. Simon’s point was that satisficing is not a deviation from rationality. It is what rationality actually looks like when you account for its costs.
The humbling part of Simon’s framework is its honesty. When a human satisfices, they usually know it. The hiring manager who picks the third decent candidate rather than interviewing fifty more is aware they are cutting a corner. They may not call it satisficing, but they feel the trade-off. The bound is visible to the person operating inside it.
The Relocation of Bounds
AI is sold as lifting these constraints. More data solves the information problem. Faster computation solves the cognitive capacity problem. Automation solves the time problem. And to a degree, this is true. A model can process millions of resumes in seconds, weigh hundreds of variables simultaneously, and never tire.
But AI does not remove bounded rationality. It relocates it.
The old bounds were human: limited information, limited cognition, limited time. They were visible, and because they were visible, they bred humility. A manager who knows they are working with partial information stays alert to the risk of being wrong.
The new bounds live inside the model. They are three. First, the training data: the model can only learn from what it was fed, and what it was fed is always a sample, never the whole. Second, the objective function: the model optimises the target it was given, not the target you wish you had given it. Third, the frame: the model answers the question you asked, not the question you should have asked.
None of these bounds appear in the output. A confidence score of 94% looks the same whether the underlying question was well-formed or malformed. The precision of the number implies a precision of reasoning that may not exist.
Confidence Inflation
Daniel Kahneman spent decades studying the gap between confidence and accuracy. His finding, replicated across domains from medicine to forensic science, is that the two are almost uncorrelated. People who are certain are not more likely to be right. They are just more likely to be certain.
AI introduces the same gap, but at organisational scale. When a model produces a recommendation, it arrives with an aura of objectivity that no human judgment carries. The numbers have decimals. The ranking has percentages. The dashboard has colours. The output looks like measurement, so the organisation treats it as measurement, and the visible satisficing that used to happen inside a human head disappears.
The UK learned this in 2020. When COVID cancelled A-level exams, the government deployed an algorithm to assign grades. The system used each school’s historical grade distribution as a baseline and adjusted individual students’ teacher-predicted grades toward it. A brilliant student at a school that historically performed poorly could be pulled down a grade or two, regardless of their actual ability. The algorithm had optimised the question “what grades are statistically consistent with this school’s track record?” The question everyone actually wanted answered was “what does this student deserve?”
Nearly 40% of grades were downgraded from teacher predictions. Students from disadvantaged backgrounds were disproportionately affected. The backlash was immediate and total. Within days, the government abandoned the algorithm and reverted to teacher-assessed grades. The system was precise and wrong, and nobody knew it until real students opened real envelopes.
The Objective-Function Trap
The deepest bound is the one hiding in plain sight: the question you chose to ask.
Every AI system optimises an objective function. The objective function encodes what you decided to measure and, more importantly, what you decided not to measure. Amazon’s hiring tool optimised for similarity to existing top performers. The Ofqual algorithm optimised for statistical consistency with historical school performance. Predictive policing systems optimise for arrest rates, which means they send more officers to neighbourhoods that already have more arrests, which generates more arrests, which confirms the prediction. The loop closes.
Gerd Gigerenzer, who extended Simon’s work on heuristic decision-making, makes the point sharply: the power of a heuristic lies in its fit to the structure of the environment, not in its computational sophistication. A simple heuristic that targets the right variable will outperform a sophisticated model that targets the wrong one. The model is not the bound. The objective is.
This is the trap organisations fall into when they deploy AI for strategic decisions. They spend enormous effort selecting the right model, tuning the right parameters, and collecting the right data. They spend almost no effort interrogating whether the thing they are optimising is the thing they actually care about. “Best candidate given our model” is not “best candidate.” It is best candidate given the data you collected, the proxy you chose, and the question you framed. Those qualifiers carry the entire weight of the decision, and they are invisible in the output.
Keeping the Bounds Visible
You cannot eliminate bounded rationality. Simon was clear about that. The goal is not optimisation. The goal is honesty about where you are satisficing.
The first practice is to red-team the objective function before deployment. Have someone whose job is to ask what you chose not to measure, and where the proxy diverges from the thing you actually want. Amazon’s hiring tool optimised “looks like current engineers” as a proxy for “will be a good engineer.” The gap between those two definitions was the entire failure. Naming the gap before deployment is cheaper than discovering it after.
The second is to keep a human satisficing step in the loop. The most dangerous AI deployments are the ones that remove human judgment entirely. A human in the loop is not a formality. It is a different kind of satisficing: messier, slower, biased in its own ways, but visible. A human who overrides a model recommendation is making a visible, accountable choice. A model that runs unobserved is making an invisible one. Simon’s framework says the human satisficing is rational given its costs. Removing it does not make the decision more rational. It makes the satisficing harder to see. This echoes a broader pattern: as we have argued before, AI cannot be managed in the conventional sense — only cultivated.
The third is to treat AI output as a proposal, not an answer. The framing matters more than people think. “The model recommends X” invites challenge. “X is the optimal choice” forecloses it. The same output, described differently, produces different downstream behaviour. Organisations that frame AI recommendations as provisional and contestable keep the space for the kind of second-order thinking that catches proxy errors before they compound.
The Permanent Condition
Simon received the Nobel Prize in Economics in 1978. In his lecture, he offered a distinction worth keeping: decision makers can satisfice either by finding optimum solutions for a simplified world, or by finding satisfactory solutions for a more realistic world.
AI does not remove that choice. It obscures it. The organisation that deploys AI and believes it has eliminated bounded rationality has not become smarter. It has become confidently wrong in a place where it used to be humbly approximate. The bounds have not lifted. They have moved from the manager’s desk into the model’s objective function, and the organisation has stopped looking for them.
Bounded rationality is permanent. The question is not whether you are satisficing. You are. The question is whether you know where.