Why superintelligence != superhuman

Published · last edited · 573 words

AI is GREAT. Claude Fable 5.1 is GREAT. GPT-6 Astra is “AGI.” Every day I see something or other written about AI doing our jobs or automating this or that or being oh-so-incredibly GREAT at something nobody thought it had to be a right to be great in.

So based on all this hype, the question should be something like, “why ISN’T Fable running the world yet???”

I think in the days of yore (*ahem* 2022), everything used to be a human process. In CS, people wrote code, read code, wrote docs, read docs, gathered information, and made decisions based on that information. In another field like journalism, people saw things, wrote articles, read articles, edited articles, and you get the idea.

In short, a process used to look something like this:

People
🙂 —> 🙂 —> 🙂 —> 🙂 —> ✅
Minutes/hours/days/units of effort
1️⃣ —> 2️⃣ —> 5️⃣ —> 3️⃣ —> ✅

4 steps of human work, totalling 11 units of effort.

But say that automation can automate some of the steps to make it a LOT faster:

People + Robot
🙂 —> 🤖 —> 🤖 —> 🙂 —> ✅
Minutes/hours/days/units of effort
1️⃣ —> 1️⃣ —> 1️⃣ —> 3️⃣ —> ✅

Now, we got 2 steps of human work and 2 steps of robot work, totalling only 6 units of effort.

Which sounds wonderful, but what if the robot isn’t reliable? What if it’s not trustworthy? What if it needs more supervision? So add some units of effort to the setup time... 

People + Robot
🙂 —> 🤖 —> 🤖 —> 🙂 —> ✅
Minutes/hours/days/units of effort
2️⃣—> 1️⃣ —> 1️⃣ —> 3️⃣ —> ✅

Still not too bad. But what if the product isn’t great? What if it needs checking? Add some to the last step before the output…

People + Robot
🙂 —> 🤖 —> 🤖 —> 🙂 —> ✅
Minutes/hours/days/units of effort
2️⃣—> 1️⃣ —> 1️⃣ —>  5️⃣ —> ✅

And we’re at 9 units of effort, only a slight improvement from the original 11, and that’s assuming the robot automates almost everything perfectly, on the first try, and there’s no unanticipated consequences from this process.

And that’s my issue with vibe coding and AI, right now. I see these models getting smarter and smarter, and at the same time, their outputs get more and more opaque. I asked Opus 5 to do one thing, look away for five seconds, and it comes back with 11 files and 3000 lines of code. It works like magic — until I add the next feature, then it breaks a little. And on the next one, it breaks a little more. Until all that is left is a glorious, chaotic mess of beautiful soup and broken promises.

“I’ll fix it next time,” Claude says with a whisper. “I promise, one more change, one more try, and everything will be perfect.”

…

“I’m sorry, that didn’t work out,” Claude apologizes profusely. “But now I see the answer, and the path forward is clear. One more prompt, a thousand more tokens, and the sun shall rise upon a new age.”

…

And I watch, as the planet turns, and the context compacts again only to grow towards its next compaction, and Claude toils on, again and again, tracking circles on the ground trying to set foot on the moon.

Dramatic recreation aside, this is the biggest issue I see with AI adoption. The model works, it gets smarter, but the work still ends where it always did — with the humans. Today's frontier models may have raised the floor at which we work, but it has yet to make us superhuman.