Do we provision double the bandwidth on a newly opened cross-border line? If we do, the 95th-percentile peak at month end may never get close, and the extra port fees and transit (过点费) are real money out the door. If we don’t, one traffic spike or one flaky peer and the customer’s experience falls over, with complaints and churn right behind it. A lot of us made calls like this while self-taught, learning on the job, going back and forth between cost and experience for years, as the product itself went from a bought theme on a borrowed framework to launch, to rewrite, to re-architecture. At some point you notice that everything each past version left behind is both the foundation you stand on and, quietly, the thing that has locked the road forward.
One thing we are clear on: stand still and you die. Competition, the march of the technology, the product cycle — all of it is rowing upstream, and there is no option to just coast. The colleagues who have grown up with a product tend, when the existing architecture gets harder and harder to maintain, to wish for a “wallbreaker.” But if the next iteration points the wrong way, it is itself a trap — easy to sink into some dilemma. Only by climbing out do you get a shot at the next step up.
Finger-pointing is really a game
Here is a set of complaints none of us is a stranger to — between QA and dev:
1. Process: the test request writes no test scope, or a line or two that may as well be nothing. Devs don't act on the bugs filed in the system and don't read the test emails!
2. Standards: why do devs throw code over the wall without self-testing? The bugs QA files can't be reproduced, and they're set to major priority!
3. Resources: who's debugging in the test environment? Who touched our test data? Our dev environment has no data — whose are we supposed to use if not yours?
4. Attitude: QA understands nothing... there's no explaining it, a user would never do that. You never account for testing when you plan — three days, how is anyone supposed to test it in three days!
Swap the roles and the same conversation replays between sales and backend, between delivery and research, between requirements and design. On the surface it reads as temper and attitude. Dig down and it is a structural game.
What the prisoner’s dilemma is
The prisoner’s dilemma is a classic model in game theory. It was framed in 1950 by Merrill Flood and given its final form by Albert Tucker.
Two criminals are arrested as suspects and held separately awaiting trial, each weighing whether to confess.
If neither confesses, each gets six months.
If one confesses and the other stays silent, the confessor goes free and the silent one gets 10 years.
If both confess, each gets two years.
Unable to communicate, the two are stuck.
The same structure is everywhere inside a company:
One manager, several employees; assume the manager is demanding.
If everyone follows orders, pay is equal but everyone is overworked;
If one refuses while the others comply, that one is out and the rest carry on;
If everyone refuses, the manager is out.
But information between employees is opaque, and everyone fears being the one who refused while the rest complied, so they all keep carrying the load. Back to QA and dev: each side picks the move that is safest for itself right now — write a little less, self-test a little less, get the blame off your own desk first — and the result is both sides stuck in inefficiency, with no one ahead. That is the core of the dilemma: individually rational choices add up to a collectively bad outcome.
The bird’s-eye view shouldn’t live only in the boss’s head
There is a line in The Godfather: “The man who can see the nature of a thing in one second and the man who spends a lifetime unable to see it are destined for completely different fates.” Systems thinking is roughly the habit of looking at one concrete thing with a whole-system view.
On the work itself: an engineer tends to fixate on how a particular piece of code or feature gets implemented; a level up, what matters is whether that feature fits the market, keeps customers, and sells — and how much cost, time, and resource it takes to build. In our line of work it is even more direct. Adding a relay hop to a route may be the optimal path technically, but who absorbs the transit fee, whether the cross-province ratio blows past its ceiling, whether net settlement income (净结入) ends up positive or negative — leave those out of view and “optimal” is only a local optimum.
On team climate: we cannot require everyone to have the range, the perspective, and the accumulated judgment to “focus on the core” or see from the top down, nor can we expect every matter to be thought through from every angle. But great ministers rise from the provinces and great generals from the ranks — fail to think today, fail to think deeply, look busy every second, and you find at the end that either what you shipped has drifted far from the user’s original need (saving the sesame, losing the melon), or you fixed A and introduced a hazard in B. At that point, explaining “it’s by design” to QA, “limited resources, we had to compromise” to the boss, and “objective constraints” to the customer slides straight into the prisoner’s dilemma of mutual blame, excuses, and collective giving-up.
And inside that dilemma, employees, boss, and company all lose — and keep losing, until the situation is genuinely improved. The logic is plain: the work is exhausting, the product unremarkable, the company at a competitive disadvantage, margins thin, pay low, satisfaction lower, so the next version and the next product are worse still — a loop that runs until the company is gone.
Over that stretch, what the company and the boss lose is a quantifiable economic loss. But what does the engineer lose? The prime years of a career — the career lifecycle. And the career lifecycle is exactly the starting point for whether an engineer grows into a senior engineer or an architect. Lose at the starting line and there is not much left to discuss.
Defining a system
In Thinking in Systems, Donella Meadows defines a system as:
A system is a set of elements interconnected in such a way that they produce their own pattern of behavior over time — organized to achieve something.
By that definition, you can break any system into four layers:
- Environment: where it runs — hardware, operating system, usage scenarios.
- Elements: the core components the system is made of.
- Interconnections: how the elements connect, what their relationships are, which paths the inputs and outputs take.
- Purpose: what the product is ultimately for. The code and the features are not the purpose — helping the people who use this thing solve a problem or reach a goal is the real root of system design and product engineering.
Run our own CDN scheduling through it. The environment is nodes across many regions and carriers, with line quality and billing conventions that differ everywhere. The elements are nodes, ports, bandwidth, and business traffic. The interconnections are the scheduling logic that decides which stream goes to which node and which port, so the cheap ports run full and the expensive ports only reach the 95th percentile. The purpose is not “get the scheduler running” — it is to hold the user’s access quality steady while keeping cost under control. Optimize any one layer in isolation and you pay it back on another: peak-shaving scheduling drives cost down but can push experience below the line; chase experience alone and the settlement invoice at month end will teach you a lesson.
Systems thinking in engineering and ops
Bring the framework down to daily work. When you design a feature or a route, you have to hold several faces at once: the feature itself, interfaces and integration, reliability, performance, interaction, security. Skip a face and that face eventually comes back to find you in production — a gap in reliability design is a 3 a.m. incident; a gap in security design is the day the “high-defense” box gets punched through.
Testing is especially hard in systems with an ML component. Hard in several places: algorithm performance, system performance, reliability, security — none of them settled by running one pass of test cases. Each has to be verified again and again under real traffic, real peers, real spikes. Which is exactly why we push back on “QA understands nothing”: the “phenomena” QA sees are precisely the layer the system exposes under real conditions.
Hard problems are where the value is — but don’t pass the difficulty to the user
Build products long enough and you will hit, and leave behind, hard problems and pain points. Some are industry-wide, some are your own defects. But from the top: where the industry has pain, the product has a selling point; while the defect stands, the product still has room to improve. Solving a real technical problem is a plus for the engineer’s growth, for the product, and for the company.
Watch the other direction, though — don’t design things that are punishingly complex to use or stacked with technical difficulty for its own sake. Both engineers and PMs are prone to falling for that kind of feature; some of it is necessary, some optional. Even when it has to exist, keep the interaction and the interface simple: the complexity belongs inside the system, not handed to the user. There is a side benefit here too — think the whole system through first and it is easier to pull out the most critical, highest-leverage parts and reduce drift as the project moves; and you need not build everything from scratch, since consistent interaction patterns and existing components save a great deal of time. Once the core skeleton stands, the patching and adding that follow get much easier.
We don’t run a blame culture
This last one is the line we come back to in every post-mortem:
- What is observed in the field is a phenomenon, not aimed at any one person or any one module.
- A phenomenon is not a failure. A bad phenomenon means we have room to improve and a weak spot — and room to improve is what gives us a future.
- Tracing a phenomenon’s cause down to a step and a person is not about blaming or punishing that individual — punishing a person is beside the point; what matters is why that step turned out that way.
- The competitive environment is exactly this harsh: only solid quality, reliability, and performance will satisfy a user and get them to pay, and only then does the company have a future.
There is no shortcut out of the prisoner’s dilemma. It does not ask who will carry the load harder — it asks that information be made transparent and that every step can see its own place and purpose in the whole system. Once everyone is looking at the same picture, the mutual-blame game loses its soil, because at that point cooperation is the better move for each person too.