跪拜 Guibai
← Back to the summary

AI Agents Dominate Coding but Fail Everywhere Else — Here Are the Five Constraints Blocking Them

In the past, many agent business results or pre-research projects have been implemented, but in practice, agents in other fields have not liberated human productivity on a large scale as imagined. Making a demo is fine, but actually solving users' business problems is still difficult (users are very willing to try new things, but the retention rate is likely very low).

So the question is, why are agents so useful in the coding field? This is the first question we need to explore, and then deduce what content is still lacking in other fields?

The following content is my personal deduction and judgment

Radar Chart.jpg


1. The Degree of Knowledge Formalizability

The training corpus for coding is simply too high-quality... If you have done NLP before (what an ancient-sounding term now) and have cleaned corpora yourself, you probably understand what I'm talking about.

Programming languages are the most thorough knowledge compression project ever undertaken by humanity.

The semantics of x + 1 do not depend on context, timezone, or the reader's cultural background. Function signatures, type checking, scope rules—these things form a directed graph where every edge is explicit. When a Transformer's self-attention runs on this graph, it doesn't have to guess "what does this word actually mean in this sentence" like it does when reading Wikipedia.

As an aside, code pre-training has an interesting side effect: models trained on code also perform better on mathematical reasoning and logic problems. Not because the model gets smarter—but because the token relationships in code are hard constraints, "there are only these few ways to transform between this symbol and that symbol." After processing millions of functions, the model develops a statistical intuition for "what can be combined with what." This is very difficult to learn from pure text pre-training.

Outside of programming, this falls apart.

A doctor writes "suspected," "cannot be ruled out," "recommend further examination" in a medical record. A common misconception is that this represents imprecise knowledge. It doesn't. In the face of a nonlinear, multi-variable system with huge individual differences, "probably," "possibly," "it depends on the situation" is the most precise form of knowledge.

Law is the same. "Duty of reasonable care"—from its inception to today, no two people have ever given a completely identical definition. It is vague not because the legislators were bad at language, but because they knew rules cannot exhaust all situations.

Forcibly compressing this kind of knowledge into deterministic rules loses not "precision of expression." What is lost is the information itself.

Let me draw a comparison:

Programming Medicine Law Corporate Management
x + 1 always equals x + 1 Same symptoms + same medicine = potentially different outcomes Same clause + different judges = potentially completely opposite rulings Same decision + different companies = one may become king, another bankrupt
Context-independent Strong context dependency (genes, age, psychological state...) Strong context dependency (jurisdiction, precedents, judge's inclination...) Strong context dependency (corporate culture, interest landscape, timing...)
Compressible Incompressible Incompressible Incompressible

Furthermore, when I discussed this with someone recently, they pointed out a key point I missed. GitHub is not just a place "with a lot of code."

High-quality repositories on GitHub inherently carry all the signal structures needed for reinforcement learning.

A well-maintained open-source project's commit history is not a pile of text. It is:

Requirement → First attempt → CI reports red → Fixed → Error again → Changed approach → Passed → merge
                                                    │
                                    Comes with expert judgment in PR review
                                                    │
                                              Comes with community voting via star/fork

A merged PR is essentially not a text sample. It is a set of behavioral trajectories with attached reward signals: a complete chain of action → outcome → reward. The model doesn't learn syntax from isolated code snippets; it learns reasoning from tens of millions of causal chains of "doing this → getting this result."

Now think about the structure of "big data" in other fields.

Medical records. Hundreds of millions of them. But each one is a snapshot—admission status, medication records, discharge outcome. There is no counterfactual of "what would have happened if Plan B was chosen instead." There is no attending physician writing in the record, "I chose drug A over drug B because a certain indicator from three years ago made me suspect this patient is insensitive to beta-blockers." The most critical decision-making logic is in the doctor's head, not in the medical record.

Court judgments. Tens of millions of them. The citation network barely provides a bit of structure—"This case refers to precedent X." But there is no outcome label for "did this contract later lead to a dispute?" There is no explanatory note for "why was this clause worded this way instead of that way at the time?"

Data in ERP systems. Every row tells you "what happened." Not a single row tells you "why this decision was made" or "whether it was later proven right or wrong."

It's not that these fields lack data. It's that the data's structure is wrong.

They are passive logs of events. Not active carriers of learning. You can do pre-training on them, letting the model learn formats, terminology, surface patterns. But to do reinforcement learning—to let an Agent self-improve through trial and error—you need the causal chain of action → outcome → reward. And this chain is broken in most industry data.

Models are stuck at the level of "imitating human output." They cannot cross the threshold of "learning from their own mistakes"—because there is no data to tell them what counts as a mistake.


2. Verification Speed and Objectivity

"In the real world, is there a 'compiler' to judge right from wrong for you?"

Compilers and testing frameworks are, in my opinion, the two most underrated things in the programming world.

You write a line of code in Cursor, and the compiler tells you in milliseconds: right, or wrong. A binary signal. Unit tests are the same—assertion passes is green, fails is red.

Don't underestimate the two attributes of "milliseconds" and "zero noise."

When humans evaluate human work, there is always bias, fatigue, and contextual deviation. RLHF attempts to turn human preferences into training signals—but every human annotation injects the annotator's cultural presuppositions, task understanding biases, and even their afternoon blood sugar level into the model. And humans are too slow. A senior engineer reviewing a piece of code takes at least a few minutes. A compiler takes milliseconds. A million-fold difference.

The core mechanism for an Agent to become stronger is reinforcement learning—it tries an action, the environment gives a reward signal, it adjusts its strategy and tries again. The signal must be fast enough, clean enough, and large-scale enough for this cycle to spin up.

Programming naturally has a compiler as a referee. AlphaGo has the rules of Go as a referee. Math problems have correct answers as a referee. This is why RLVR (Reinforcement Learning with Verifiable Rewards) has exploded in these three fields.

Outside these three fields, where is the referee?

A doctor prescribes medication. How to verify? Wait. Clinical outcomes over weeks to months. The signal is so muddied by patient genetics, lifestyle habits, and medication adherence that it's almost impossible to attribute—did this patient get better because the medicine was right, or because they also quit smoking? No one can be sure.

A lawyer drafts a contract. The judgment of right or wrong might have to wait until someone actually sues years later. And the same clause might yield three different conclusions in the hands of three different judges.

In these fields, the concept of "correctness" itself is inherently non-binarizable. It's not that "an objective referee hasn't been invented yet"—it's that the concept of an "objective referee" does not hold for this problem. It is multi-interpretive, context-coupled, and scale-continuous. You cannot use a discrete reward function to approximate a quantity that is essentially non-binarizable.

Without pure feedback, the closed loop of reinforcement learning is broken. This is why, except for programming, mathematics, and board games, almost all AI applications are stuck at the "generate a draft for a human to revise" stage. It's not a lack of desire to automate. It's the lack of a referee.


3. Coding Allows Near-Infinite Trial and Error

Pure signals let the Agent know what is good. But what truly makes it stronger is the ability to make mistakes safely.

Before defeating Lee Sedol, AlphaGo played millions of games against itself. Every move could have been a blunder. Every game could have been a loss. But the simulator didn't care. Put AlphaGo in an environment where "losing one game means being pushed off a cliff" to train—it would learn nothing.

The cost of trial and error in programming is almost zero.

git checkout -b experiment   # Create a branch, mess around freely
git branch -D experiment     # Made a mistake, delete the branch. Main branch is unharmed

Docker is the same—AI writes an infinite loop, a memory leak, rm -rf / inside it, shut down the container, nothing happened.

The mathematical expression of this property is: exploration and delivery are two completely decoupled phases. The Agent can trial-and-error millions of times in an isolated environment, find the optimal solution, and then deliver it to the real world.

Most human activities cannot decouple these two things.

The physical world has no ctrl+z. Companies like Pony.ai have been running driverless taxis in cities like Shenzhen and Wuhan, making it seem like the physical world can also "undo" things—but look at its cost structure. Behind it is a huge insurance policy and a complete legal team. This is "wrapping the cost of trial and error with money and lawyers." Do ordinary factories, hospitals, and law firms have these conditions?

Organizational decision-making is even more troublesome. You try a layoff plan—if it's wrong, the laid-off people are gone, and those who remain are changed by fear. Brand reputation takes years to accumulate and can collapse in a single event. There is no version number, no rollback, no revert commit. Every "pilot" is a live operation—you can narrow the scope of the pilot, but that only shrinks the blast radius, it doesn't make the cost of trial and error zero.

This is why RLVR (Verifiable Rewards + Reinforcement Learning) is almost impossible to carry out outside of programming and mathematics. The premise of verifiable rewards is: the operating cost of the verification environment is zero. AlphaGo needed an infinite Go board. Programming Agents need infinite sandboxes. In most fields of human activity, that "parallel universe of free trial and error" does not exist.


4. In the Real World, Power and Responsibility Are Complex

This constraint is not a cognitive problem.

In programming, responsibility is silently absorbed by the technical system:

Compiler + Type System   → Absorbs syntax responsibility. Code has a syntax error? Rejected directly. No human approval needed.
Test Suite + CI          → Absorbs logic responsibility. Regression failed? Automatically flagged red, merge is blocked. No human judgment needed.
Git + Docker             → Absorbs trial-and-error responsibility. Messed up? Instantly restored. No one needs to step up and say "I screwed up."

Outside of programming, all three layers are gone. AI gives a suggestion, a human needs to judge if it's right. AI makes a decision, a human needs to verify if it's feasible. AI makes a mistake, a human needs to bear the cost.

And no one is willing to pay this cost.

I'm not being pessimistic. This is a very rational calculation.

The real reason doctors don't use AI to prescribe medicine is not that AI is inaccurate. The accuracy of a certain imaging diagnostic AI has already surpassed that of human radiologists. The problem is, if a medical accident occurs, the doctor goes to jail, the doctor's license is revoked. AI doesn't go to jail.

The reason lawyers don't use AI to send contracts directly to clients is not that AI is unprofessional. It's that missing a key clause that costs tens of millions, being sued for malpractice—that's on the lawyer. AI doesn't pay malpractice insurance.

The reason corporate middle managers passively resist AI process optimization is not that they are conservative. It's that after "optimization," their team has fewer people, a smaller budget, and less power—but the department's KPI is still on them. AI doesn't carry their annual performance review for them.

This incentive structure is fatal. When AI makes the right decision, the benefits are distributed to everyone. When AI makes the wrong decision, the cost is entirely concentrated on the person who signs off. And that person may fundamentally not understand the AI's decision-making process, cannot control it, and cannot explain it during a regulatory inquiry. This is a risk-reward structure with negative convexity. The optimal response for any rational person is the same: don't sign.

This is an equilibrium that inevitably emerges under given game-theoretic constraints.

This fatal flaw is not within the solution space of technology. It requires legal personhood—making code bear criminal responsibility is conceptually untenable. It requires the assumption of free will—no legal system can be built on the attribution that "this algorithm chose to do evil." It requires a societal consensus on punishment—and punishing a non-human entity provides no moral satisfaction or deterrent effect.

These are not ethical problems that can be solved with increasingly larger model scales!


5. Cross-Domain Comparison: Scanning with Five Constraints

Arrange these five constraints into a table, and a scan makes it clear:

Constraint Programming Chip EDA / Formal Verification Quantitative Finance IoT / Industrial Control Medical Diagnosis Legal Services Corporate Management
Knowledge Formalizability ✅ ✅ △ △ ❌ ❌ ❌
Verification Speed & Objectivity ✅ ✅ △ ❌ ❌ ❌ ❌
Trial-and-Error Reversibility ✅ △ ❌ ❌ ❌ △ ❌
Power & Responsibility Absorbed by System ✅ △ ❌ ❌ ❌ ❌ ❌

A few observations:

Chip EDA / Formal Verification is the only field close to programming. Logic synthesizer = compiler, formal verification = test suite, simulation environment = sandbox. This is also why AI is progressing so fast in chip design (Verilog generation, place-and-route optimization) and mathematical proofs (Lean 4 + AlphaGeometry).

Quantitative finance looks like programming, but actually has two problems. First, the signal is impure—profit does not equal a correct strategy; it could be random noise. Goodhart's Law will kill any strategy adopted at scale. Second is responsibility—a licensed fund manager can be held accountable. If an AI strategy collapses, whose license gets revoked?

IoT / Industrial Control has three problems. An API returning 200 OK doesn't mean the valve actually closed (the "false success" problem). Physical damage is irreversible. The chain of responsibility among equipment vendors, software vendors, and operators has never been clearly sorted out.

Medical and legal AI in serious scenarios basically require a human gatekeeper. Knowledge is incompressible, corpora lack causal chains, signals cannot be binarized, trial and error is irreversible, and responsibility rests entirely on humans. This is also why the most pragmatic current AI application form in these two fields is stuck at the level of "assisted retrieval + initial screening and tagging." It's not a lack of desire to move up. It's that every layer upward requires a human to cushion underneath.

Corporate management is the hardest one. This row is completely blown up. But you will also observe that AI penetration in corporate management is happening—meeting minutes, email summaries, code generation—all fragmented "point-like efficiency boosts." Once you try to string these "points" into a "chain" (end-to-end process automation), those collapsed constraints will pop up the ultimate question of "who bears the responsibility" for all relevant personnel. And then things get stuck.


6. A Few Scattered Supplementary Thoughts

6.1 Every historical "impossible" was later disproven. Will this time be different?

Translation requires cultural understanding → DeepL and GPT changed the industry. Go requires intuition → AlphaGo crushed the human champion. Protein folding requires scientific insight → AlphaFold cracked a half-century-old problem in two years.

Every time someone asserts "XX is a uniquely human capability," the list of historical counter-examples grows longer. So I am very cautious about judgments like "XX field is essentially non-agentifiable." Empirical descriptions of difficulty ("currently not possible") are honest; essentialist claims ("never possible") require proof at a philosophical level—and no one has ever successfully provided it.

But the matter of responsibility is different from the previous ones. Knowledge, intuition, insight—these are all cognitive-level things. Recognizing patterns, establishing associations, finding optimal solutions in a search space. We once thought these were uniquely human, and later discovered statistical learning could approximate them. Responsibility is not a cognitive attribute. It is a sociological construct. This is not a problem of "current AI isn't smart enough, it will be solved later"—it's a problem where "the prerequisites for solving this problem are not in technology's toolbox."

6.2 The Penetration Direction of Agents in Other Fields

I think it's highly likely not "total replacement," but gradual erosion. First, take root in sub-tasks with the lowest cost of error, clearest boundaries, and fastest verification, then penetrate upstream.

Did you notice? The last action in every chain is the same thing—a human takes the bottom line, a human bears the responsibility.

6.3 Responsibility Allocation Is Not a Technical Problem

One line of thought is to "solve it with insurance"—buy insurance for AI decisions, and the insurance company pays if something goes wrong. Waymo is already doing this. But this solution has two hard flaws:

  1. Insurability. The actuarial basis of insurance is quantifiable risk. In most fields, AI decision errors haven't accumulated a mature enough loss distribution. Insurance companies don't know how to price it.
  2. Scalability. Waymo can do this because it has Alphabet behind it. Ordinary hospitals, law firms, factories—try asking them to spend millions a year on insurance for AI.

Insurance just wraps a financial buffer around the outside of responsibility. It doesn't change the fundamental constraint that "AI is not a legal subject." It just turns the "who bears it" problem from "I won't sign" into "I can't afford the premium."


The above is entirely personal judgment. Corrections are welcome in the comments section.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

用户800314581770

Thumbs up