“The ideal racing car would be one that beat its nearest rival by a couple of metres and then immediately fell to pieces.” This apocryphal statement is most convincingly attributed to Colin Chapman of Lotus but it might also have been Ferdinand Porsche who said it. Regardless of its true origin, the quip captures a certain kind of design philosophy. When there is a specific task to be achieved, the most elegant solution is the one that precisely achieves that objective with minimum resources. Anything extra – a slightly bigger winning margin, a little more robustness – is a waste.

But it’s a bit of a conjuring trick. This approach only makes any sense at all because winning a race is a very specific and unusual objective for an automobile: driving a known distance, over known terrain, with a defined victory condition. If you wanted to apply this philosophy to any other kind of car, how would you even begin to do so? Is the ideal SUV one that finishes the school run at exactly 08.59 and then explodes? Of course not.

A strange thing has happened in our political and economic system, however. Without really realising, many organisations and companies have decided to adopt this proverb as a philosophy of management. In the name of efficiency, governments and businesses have spent the past few decades trying to remove management, cut bureaucracy and eliminate waste.

The results are disappointing and occasionally disastrous. Trying to slim down the government has resulted in higher ratios of tax to GDP and lower economic growth. Companies that have managed themselves to maximise financial returns have ended up stripping out innovation and stumbling from one crisis to the next. Nearly all of the actual value that has been created this century has been in companies that concentrated on technology and customer service, not on shareholder returns.

A large proportion of the modern world’s problems comes down to something as simple as this: the pursuit of a mistaken set of design principles and an excessive concern with efficiency in a world of uncertainty. As computer scientist Donald Knuth once put it, “Premature optimisation is the root of all evil.” This means that the measures making a system more efficient in one environment can be disastrous in another. Because of this, you absolutely must not try to optimise something until you’re sure of the problem for which you are optimising.

So what might be a better design philosophy? How can we design systems that won’t fall to pieces as soon as they cross a hypothetical finish line? Weirdly, the answer might be that we need to take another look at one of the most maligned classes of knowledge workers: the middle manager.

What do middle managers actually do? A lot of the time, they seem to get in everyone’s way. They hold interminable meetings, compile handbooks and policies, and spend every spare moment engaged in internal politics. If we define a “middle manager” as somebody who is not directly involved in producing the product or talking to the customer, it’s not hard to see why they have always attracted suspicion. If there is any waste or dead wood in an organisation, middle management is often where it’s found.

But let’s look again at those activities and ask a question that used to be taken much more seriously: what is an organisation for? Why do companies employ a permanent staff at all, rather than just buying services and hiring staff by the day on an open market? If we go beyond the assumptions of conventional economics, in which information is assumed to be frictionless and perfect, and think about the actual mechanisms by which it is preserved and transmitted, we get some real answers.

The application of the theory of information to organisations used to be called “management cybernetics”, a school of management theory so totally eclipsed by “shareholder value” that it scarcely exists today beyond second-hand bookshops and the personal idiosyncrasies of some ageing professors.

A company produces products but its management produces decisions. A company, or any other organisation, is an information-processing system – in effect, an artificial intelligence consisting of people, memos, telephone calls and emails. The reason that a large permanent organisation exists, rather than a “virtual” company made up of outsourced services purchased by a single founder, is that the information-processing system needs more bandwidth and more storage than any single human can provide. Humans form organisations to extend their capacity and take on bigger projects than they can achieve on their own. That’s as true of decision-making as it is of bridge building and iron smelting. The Industrial Revolution was a revolution in the industrialisation of decision-making, not just of manufacturing.

That’s what good middle managers do. They hold knowledge and make communication happen so that the organisation can work. The real job of middle management is to ensure that information arrives where it needs to be, in a form in which it can be the basis for a decision and in time to be useful. To do this, middle managers use a lot of standard techniques. They write down policy manuals so that commonly recurring decisions can be sped up or delegated to front-line staff. They send memos and emails to get the facts to the right places. And, yes, a lot of the time, they hold meetings, partly to keep looking for information and to share it but also to build the formal and informal communication channels that will come in handy at some future date.

The failure to employ enough of these people (or, worse, to employ them but not listen) leads to a kind of corporate cognitive disorder that’s sometimes called “founder’s syndrome” when it happens in start-up companies. (When it happens in larger or established corporations, it’s just called “being badly managed”.) Founder’s syndrome is what happens when a company’s first management team is unable to understand, or to accept psychologically, that the organisation is now bigger than its capacity to process information. The symptoms of founder’s syndrome can be obvious (except to the person who’s causing it). Details get missed, deadlines are stretched, processes can’t be scaled up and everyone starts talking about “micromanagement”. But, even more destructively, the organisation develops tunnel vision. Only information that can get through the founder’s screens can affect anything, just at the moment in time when the founder is least likely to have the spare bandwidth to think about the outside world. Venture capitalists are often highly attuned to the risk of founder’s syndrome and start to demand that visionaries stick to having visions, then “hire good people and let them get on with it”.

So who’s right? To answer anything other than “it depends” would show a profound lack of respect for the question. To solve a problem in any specific case, we need to go back to the design principles and take seriously the idea that an organisation is an information- processing system. Taking this approach is like putting on a pair of magic glasses that suddenly reveal the true structure of management reality. Is this role helping the flow of information or hindering it? Does having this policy help to free up time for creative thinking or does it consume time in dealing with exceptions and details of the policy? Is the organisation focused on immediate here-and-now problems or is it able to look outside and to the future – and can it effectively deal with outside information once it has been gathered? And, perhaps most crucially, how would things look if the world changed?

What is interesting here is that none of these questions seems to be closely related to cost efficiency or shareholder value. There is a pretty good reason for that: you can’t understand a business by looking at its accounts. An accounting system is one way of organising some kinds of information about certain aspects of a company. And it’s always backwards-looking. This makes it into exactly the kind of information-reducing filter that can dangerously impair your ability to adapt.

You might put it like this: almost always, the amount of money that a project will cost is vastly less important than the simple question of whether it’s likely to work or not. But all too often, people spend all of their effort on the former rather than the latter. We don’t even have a word equivalent to “costing” to describe a process of thoroughly and rigorously estimating the benefits of doing something. And so we get the kind of companies and projects that we deserve because we treat financial returns as the only goal of business, rather than one constraint among many that has to be met on the way to doing something truly interesting.

In our current economy, founders are often the only people who have the social and institutional permission to look beyond the accounts, take risks and think about the big picture. But it doesn’t have to be that way. Consider Nokia, for example. A century ago, the company made rubber boots; today, it makes telecom switching equipment. Content and entertainment group Vivendi used to run the sewerage system in the town of Lyon. Less dramatically, IBM and Apple have reinvented themselves several times.

The good thing about “founder mode” might be that a company with a strong sense of identity and purpose has the ability to adapt to its environment. It isn’t a supercar that delivers outstanding performance in a straight line, then wrecks itself on the first speed bump.

Perhaps the last lesson of cybernetics is exactly that a sense of identity and purpose is at the heart of what makes an organisation work. You can only start thinking about a company from an information-centred perspective once you have first decided what counts as information, what the system “cares about”. This is the true responsibility of founders in a start-up. In an established corporation, it might rest with the CEO but it’s more likely to be distributed across the whole of the management.

The distinction between founders and managers, modes and syndromes, is a false one. Excess bureaucracy is not always bad, nor is efficiency always good. What matters for a company is viability – the ability to respond to unanticipated changes. And the cornerstone of viability is self-knowledge, the understanding of what the company is for.

That is the start of being able to describe what kind of environment it needs to be able to cope with – and only then does it make any sense to start optimising. Middle managers are the brain of the firm but the brain needs to follow the heart.

Read next: How we work must change – let’s not have a meeting about it