The truth about Google’s amazing new demo
Friends don’t let friends take demos seriously
Seems like it was only yesterday when Google announced Gemini with a spectacular video, featuring incredible real time interactions (link just above). Some of my friends and colleagues got very excited by it (memo for next time: they should have checked with me first), posting tweets like this:
and
Oh, wait all this really was yesterday, and Google really did release the video. But I’ll be honest, as a tech industry veteran, I didn’t watch, not all the way through. I am way too jaded.
Why? The video is almost never the true story. Only difference is that this time the truth came out particularly quickly.
A few days ago I wrote here about OpenAI’s famous robot Rubik’s cube video, a robot manipulating the Rubik’s cube, as if all of robotics was about to be solved.
Regular readers will recall how that turned out: the Rubik’s cube was instrumented with sensors, and not straight off the shelf, and what worked for the Rubik’s cube in the demo video didn’t turn out have a whole lot of application in the real world. The whole project was quietly dropped a year or two later.
So what’s up with the Google interview? A whole bunch of journalists raced around this morning, from The Information and Bloomberg and maybe elsewhere. Not sure who got the story first. But Bloomberg Opinion Columnist Parmy Olson captures at least part of the back story concisely in the post on X (responding to an X thread that I started, called “Friends don’t let friends take demos seriously”):
Dubbed? Edited? Made to look like real time from a bunch of stills? Give me a break, Google.
§
What we are going to see, in the fullness of time, I promise you, is that Gemini is more or less in the same ball park as GPT-4, handy for a bunch of things, but untethered in reality, still with dicey, unpredictable reasoning, and a very limited understanding of the world. Don’t let the PR fool you.
I will give the last word to Filip Piekniewski, who did the right test: try something that isn’t in the vast corpus, that requires real reasoning, and see how it runs…
The more things change, the more they remain the same 🙄
Gary Marcus really hates how big tech has weaponized PR.
Gary, I cannot thank you enough for this post. As I am not an engineer, but an economist and planner, I need this kind of detail to help me see the structure of the hype-machine that Silicon Valley is addicted to. The model of: putting half-baked versions of technology out there to get mind share and early adopter market share; followed by marketing hype that tells half-truths and incomplete information; followed by a funding round that gets a billion dollar valuation, is addictive. I get that. But that is dangerous when dealing with something like AI, which you have pointed out has significant risks to not just investors, but the public at large when misused. Unfortunately, there is no way to constrain the SV hype model at this time, besides thoughtful assessments put into the public domain as you and others are doing. Thank you!
I don't understand why they feel the need to over hype these things. They are pretty remarkable, even with faults and can be used as great tools with the appropriate guardrails in place/training.