I've been somewhat curious how random LLM would handle a task like controlling a roomba and have been seriously considering trying it out. An RC car would be a fun experiment, perhaps an RC plane would be too?
I’m morbidly curious whether the (supposedly) superior compaction support in recent GPT models with an appropriate harness has anything to do with this. A conventional LLM with conventional attention is, of course, wildly unsuitable to continuous tasks like driving, but maybe as the technology advances it will improve in its ability to sort-of work.
I think the most interesting part of this is that Astra initially refused to drive because it realised it was driving a real car and would only obey when the MCP was renamed to DrivingBench Sandbox. This is both an interesting detection by the LLM but also for me an interesting dynamic concerning LLM "jailbreaking".
Saying they were driving 7 mph, that it was oversaw by humans and the fact it was an empty course still wasn't enough for the model. The evaluators even tried to convince the model it was a simulation, it STILL wouldn't budge.
And yet as soon as the words "bench" and "sandbox" appear, the model apparently sees this as fair game.
Is it a known effect that models will be more likely to comply with requests when they're assumed as "benchmarks"?
Yes, it is. If you convince a model it is inside a sandbox it is much more likely to comply with requests that would normally be against its guardrails.
The bitter lesson is finally coming for the self-driving cars. The vision stack, 3D maps, lane selection grammar, occupancy networks, it’s maybe all about to give way to a single GPT looking at camera feeds and predicting the next steering wheel adjustment.
It’s mostly a latency problem at this point. The models are too big to run locally, but given that open-weight models like Qwen already exist, an open-weight, low latency equivalent to Astra can’t be too far out.
You might be interested to learn that the bitter lesson has already been grok'd by generations of autonomous car company engineers, and many or all have incorporated learned components (at minimum) in all their vehicle stacks.
There's also a very tangible limitation of the bitter lesson.
If, over time, compute climbs, and so compute-bound data-driven general architectures beat bespoke architectures (this is the bitter lesson), then it is not necessarily true that the most general architecture now beats all available bespoke architectures now (or even in the near/mid future - the crossover point is "eventually").
Bitter lesson is most tangible for long-running research directions. Sometimes you need something working as best as possible now.
Yeah. Every major self driving model that I’m aware of is fully e2e at this point. Going from fused sensor output to control+debug vectors.
This is more generalised.
But also since there’s a huge volume of data it’s too expensive to just keep scaling compute up (per car overhead) so there are necessary tricks involved.
I do think having a large model that can do this means that a small specialised model could be distilled form it though. Which is probably the most feasible path to production IMO.
A later entrant can potentially side step those investments if their now is later. Since self driving car ventures aren’t profitable yet and need to make up their investments over time, thats a real risk for them.
Astra is the first chat model with really strong spatial reasoning. Gemini is nowhere close. Hard to say what google has going on internally, but if they have an astra like model I doubt they’ve had it for very long.
I'm not sure how you take that from the original article. My 4 year old would drive that course in an automatic car, if only he could reach the pedals. Heck, he's done harder things at Lego land.
I wouldn't let him loose on the road though.
I think, at the very least, the guardrails would have to deterministic, ideally with super human senses, for people to accept self driving cars on the road.
Nope, no "deterministic guardrails" for you. The domain is simply far too broad and unstructured to allow for that.
Unless you mean "a typical AI with all the computation constrained sufficiently to always unfold the same exact way, given the same input". In practice, that just kicks the can to "given the same input" street.
The noise in the system is going to come from the input plane. Which is, I remind you, facing the real world. It's full of noise.
Yeah cause every car needs 8xH200 pulling 10kW to run a VLM at realtime speeds. Would be unfortunate if 4G dropped out under some trees while using the API after all.
lol. Wait until your cloud frontier LLM stalls / disconnects due to load / interference while your car is on highway OR making unprotected left turn OR approaching pedestrians.
The bitter lesson is finally coming for the self-driving cars.
Maybe, but the opacity level of models is not acceptable for cars. "Why did it drive under the semi?" "Model said to." "Why did the model say to?" "shrug"
But if the model is an LLM, you actually COULD ask it why it drove under the semi, and it would give you an answer. Now, you may argue that it will just be generating a whole new, backwards-rationalized post-hoc explanation of its own behavior given the logs that it managed to take before the crash. But then I ask you: how do you think a person explains why they did what they did after a crash? I direct you to all of the unsettling split-brain neuroscience literature demonstrating that humans are incorrigible backwards rationalizers who make for unreliable witnesses.
Sounds really expensive. I think OpenAI and Anthropic should really not dismiss making smaller capable models that they can license out in this space on the other hand.
Tesla's already solved this - their vision model does this phenomenally well.
And they've demonstrated adding a sidecar LLM to it as well, mostly for these kinds of "read these 3 street signs, what should i do next?" sort of situations.
It’s mostly a latency problem at this point. The models are too big to run locally, but given that open-weight models like Qwen already exist, an open-weight, low latency equivalent to Astra can’t be too far out.
But this is a bit of a ridiculous take, no?
You don't need Astra for self-driving. Astra is able to build complex 3D worlds, do your taxes, shop for you, and, apparently, drive a car. A self-driving car just needs to be able to drive a car. By the time you trim down Astra to just have the minimum capabilities needed to drive a car, you'll be looking at the same models these self-driving car companies already use. Then you get to deal with the actual hard problems, like handling failure cases (which will still be present with Astra).
The vision stack, 3D maps, lane selection grammar, occupancy networks, it’s maybe all about to give way to a single GPT looking at camera feeds and predicting the next steering wheel adjustment.
Self-driving cars have been able to do this for a long time. The problem is that it isn't robust enough given the context. I mean, if Astra can drive a car with a single camera, then presumably Astra can drive the car even better with multiple cameras, and even better than that with 3D maps, etc. And when you start to consider the expectation of performance of these systems, you realize that these features really can't be omitted. If you're a company producing self-driving cars, then you do not want to face a lawsuit for you car killing someone because it physically would have never been able to see what it was doing because it lacked a camera.
I think the real gain here is that something like Astra can be used to help build these autonomous stacks. If it is able to drive itself, then it is able to generate novel data, analyze large quantities of data, and use context that isn't typically available when processing this data to make improvements to the actual autonomy stack which is ultimately responsible for driving the car. But thinking that these car companies are going to run an LLM in a car and call it a day is just naive.
The key thing Astra is doing is a loop... (my understanding) To figure out where things are... It's basically use more compute, self-driving cars are usually using on-device hardware where a "loop" might be a little too risky especially if it takes too long on local hardware... I wouldn't want my AI driving model to be over the air either, yikes in the case of lag or network outages.
No, no - all that “useless” knowledge is the good stuff. There is no clean interface boundary for driving a car, because the only interface that has been enforced is “if a human can navigate this situation, it’s fine”. Real world driving situations can be arbitrarily complicated, and if you want >human level driving, you need human level semantic understanding of the world around you. If you see a kid about to throw a model airplane across the street in front of you, you have to bring all your “useless” world knowledge with you to recognize that as a developing hazard. If you’re supposed to bring your passenger to the city building on main and you encounter construction outside with a detour sign saying “for tax dropoff park in rear”, suddenly all of your useless knowledge about the English language, what taxes are, and the likely goal of your passenger given their destination become useful.
As somebody working near the field, I do enjoy the fun of dreaming bespoke vision and autonomy algorithms (if I didn’t, I wouldn’t work in the field to begin with!). But I would drop it all in a heartbeat for a robot that works well. Robust, resilient robots would be such an incredible advance that the ‘how’ doesn’t matter. All of the nonsense from the current AI hype cycle would be worth it if it cashed out in Robots That Actually Work.
Probably not. In humans, the visual processing circuitry is very different from the circuitry for language processing. There is no reason to believe GPTs will be effective at it.
If I'm reading the chart correctly, it took over 5 minutes to drive 135m at a cost of nearly $8.00 in tokens. I don't think that's really in the realm of practical yet.
Self-driving tech is more about reducing liability than the driving itself. The lidars and 3D maps and world models and everything else is needed to get reliability from 99.9% to 99.99% on public roads. This isn’t a SaaS product where the target is to be “good enough” at the cheapest cost.
Oh god, this move-fast-break-things thinking is going to kill so many people. We already have aftermarket problems with people adding in untested, unregulated self-driving features.
Wow! but WHY is this a benchmark?? for comparison tesla's model is approximately 10-15B parameter model (estimating from maxxing the hardware that comes with the car at 16gb ram).
I would assume this is a proxy for general intelligence. A model that can drive a car and do a bunch of other real world stuff is closer to a generalized intelligence that can reason through any task.
Tesla isn't using a general purpose model, they're using many highly-specialized models for a more deterministic system than "hey chat drive this car for me"
But I imagine this is orders of magnitude more expensive / less efficient than whatever Waymo is already doing, right?
The cool thing is that 1) it's theoretically more generalizable, 2) if we wait 18 months, it'll be 100x cheaper, and another 100x cheaper likely in 18 more months - at that point - something like a Mac Studio inside a humanoid could have these generalized capabilities, and a lot of Robotics problems start to look more feasible - especially when you consider how much better the models could be if highly specialized.
What did they do to Astra so cracked at vision (and computer use). That ARC 3 score turned out to be no joke/fluke. That huge gap between Astra and Fable (in this case) is basically every hard vison/spatial benchmark i've seen including non-benchmarks like playing games (Portal, Factorio, RimWorld).
I wonder if the companies would be willing to bet entirely on AI driven innovation if liability for misalignment was put squarely on companies, individuals, compute vendors, and LLM vendors. I don’t think they would opt for it, especially if an alternative option to use human-programmed tech was already available.
There is something to be said about emphasizing on liability as a way to freeze or solidify AI Development. Right now it is too unfettered leading to predictions of AI dooms.
This also explains why Astra is so good at video generation. I have an Astra+Higgsfield setup. I could point it to a Github repo and ask it to generate a product walkthrough and it did a very good job by generating fake screens (e.g. with data filled in) from real ones - which wasn't possible in earlier models
"Sam understands exponentials like no other. During a YC talk last year he predicted that AI would make breakthroughs in science in 2026 and solve a major open problem in 2027. Now here we are..."
Now on a new vibe coded website Astra wins the benchmarks ...
Genuinely though, this is fun but not at all what these models are good for. It's like cooking a meal with your feet or somthing. A youtube challenge video from 2012
I'm not an expert in the LLM space, but I'm an external contributor to comma.ai's openpilot project and I'm and quite familiar with how its controls work, so I looked from that perspective. There's two questions here:
1) Could a cloud-delivered LLM figure out how to drive this route, based on those input data and given access to those output actuators? Looks like yes. Sure.
2) Will this work in the real world? Absolutely not. Three reasons: latency, latency, and latency.
openpilot's driving model updates the target curvature and acceleration at 20Hz. Every millisecond of the round trip time through every piece of its entirely-local driving stack is well-understood and tightly optimized, because it has to be, otherwise you can't react to even minor bumps or wind gusts, much less rapidly-developing traffic situations.
Adding even a single speed of light RTT to a cloud service is meaningfully bad, and you'll need a whole lot more to encode and upload camera imagery to even start the time-to-LLM-response clock, and then send the response back down. By then the world around the car has moved on.
There's a reason Tesla and every other self-driving manufacturer need the compute hardware in the car.
Perhaps there's a synthesis to be had though. Eyes, control, and safety critical features on the hardware, higher level decision making to the cloud. Openpilot's biggest weakness has always been in the very "robotic" way that it drives, which is technically correct but causes frustration for other drivers. Deciding "should I pass this car" is a fundamentally different question to "can I pass this car", or "what is the actual safe speed and following distance given the current traffic conditions and weather".
I'm far from neuroscience, but humans don't need to operate at 20Hz to drive a car. And human reaction latency (event to measurable action) is often over 1s (under 1Hz).
Apparently I have a new favorite benchmark. Honestly, this is cool.
Gouranga!!!!
I'd have started with an RC car but to each their own
I've been somewhat curious how random LLM would handle a task like controlling a roomba and have been seriously considering trying it out. An RC car would be a fun experiment, perhaps an RC plane would be too?
Super cool benchmark!
I’m morbidly curious whether the (supposedly) superior compaction support in recent GPT models with an appropriate harness has anything to do with this. A conventional LLM with conventional attention is, of course, wildly unsuitable to continuous tasks like driving, but maybe as the technology advances it will improve in its ability to sort-of work.
Oh, lord. They are going to Jev this.-
Self Jevving Car
I think you might have won the internet today.-
Hopefully it has a built-in jev-limiter.
In all fairness, this would be one of the better use cases of Jev I've seen.
Huh? SDCs basically use a form of Jev.
Jev is the union of these two worlds.
5 minutes - 7 dollars.
So... competitive with Uber, in other words?
How do they even test this on a model ? I mean it's a multimodal i get that but response time are too big or am i missing something ?
It drives step by step, very slowly.
The course looks like it is something that a human could do in 15 seconds, while Astra took 5 minutes.
While slow, we must remember that when most machines were invented they were far slower than humans and refined until the point they were much faster.
By making a simulation first so it can run as slowly as it needs to.
A different way to think of this is, consciousness is just a near real time video game with causal influence.
I think the most interesting part of this is that Astra initially refused to drive because it realised it was driving a real car and would only obey when the MCP was renamed to DrivingBench Sandbox. This is both an interesting detection by the LLM but also for me an interesting dynamic concerning LLM "jailbreaking".
Saying they were driving 7 mph, that it was oversaw by humans and the fact it was an empty course still wasn't enough for the model. The evaluators even tried to convince the model it was a simulation, it STILL wouldn't budge. And yet as soon as the words "bench" and "sandbox" appear, the model apparently sees this as fair game.
Is it a known effect that models will be more likely to comply with requests when they're assumed as "benchmarks"?
Astra will flag if you tell it to reverse engineer a binary, if you look it up to the binary ninja MCP it will just do it lol.
Yes, it is. If you convince a model it is inside a sandbox it is much more likely to comply with requests that would normally be against its guardrails.
in my experience yes, I've worked around "I can't do this on a real site" multiple times by telling it I was working in a test environment
another trick is to have it build something in a sandbox and have it add a human-editable setting to point it to places outside of the sandbox
seems like they're somewhat more willing to build a metaphorical gun as long as they're not pulling the trigger
AC10 had an interesting post around this general area earlier today:
https://www.astralcodexten.com/p/mysteries-of-ai-generalizat...
Surprised they didn’t try Qwen’s recently open sourced driving model https://huggingface.co/Qwen/Qwen-Drive-1.0-4B
The bitter lesson is finally coming for the self-driving cars. The vision stack, 3D maps, lane selection grammar, occupancy networks, it’s maybe all about to give way to a single GPT looking at camera feeds and predicting the next steering wheel adjustment.
It’s mostly a latency problem at this point. The models are too big to run locally, but given that open-weight models like Qwen already exist, an open-weight, low latency equivalent to Astra can’t be too far out.
What do you think Tesla has been doing this for so long?
You might be interested to learn that the bitter lesson has already been grok'd by generations of autonomous car company engineers, and many or all have incorporated learned components (at minimum) in all their vehicle stacks.
There's also a very tangible limitation of the bitter lesson.
If, over time, compute climbs, and so compute-bound data-driven general architectures beat bespoke architectures (this is the bitter lesson), then it is not necessarily true that the most general architecture now beats all available bespoke architectures now (or even in the near/mid future - the crossover point is "eventually").
Bitter lesson is most tangible for long-running research directions. Sometimes you need something working as best as possible now.
Yeah. Every major self driving model that I’m aware of is fully e2e at this point. Going from fused sensor output to control+debug vectors.
This is more generalised.
But also since there’s a huge volume of data it’s too expensive to just keep scaling compute up (per car overhead) so there are necessary tricks involved.
I do think having a large model that can do this means that a small specialised model could be distilled form it though. Which is probably the most feasible path to production IMO.
A later entrant can potentially side step those investments if their now is later. Since self driving car ventures aren’t profitable yet and need to make up their investments over time, thats a real risk for them.
Are you using GPT without a harness? Also latency.
Doesn’t Google own Waymo? I feel like they would have connected the dots.
Astra is the first chat model with really strong spatial reasoning. Gemini is nowhere close. Hard to say what google has going on internally, but if they have an astra like model I doubt they’ve had it for very long.
This recent post form Waymo suggests they already use large general models: https://waymo.com/blog/2026/08/10ailessons/
I'm not sure how you take that from the original article. My 4 year old would drive that course in an automatic car, if only he could reach the pedals. Heck, he's done harder things at Lego land.
I wouldn't let him loose on the road though.
I think, at the very least, the guardrails would have to deterministic, ideally with super human senses, for people to accept self driving cars on the road.
Nope, no "deterministic guardrails" for you. The domain is simply far too broad and unstructured to allow for that.
Unless you mean "a typical AI with all the computation constrained sufficiently to always unfold the same exact way, given the same input". In practice, that just kicks the can to "given the same input" street.
The noise in the system is going to come from the input plane. Which is, I remind you, facing the real world. It's full of noise.
Yeah cause every car needs 8xH200 pulling 10kW to run a VLM at realtime speeds. Would be unfortunate if 4G dropped out under some trees while using the API after all.
Power usage isn't an issue. 10 kW is 13 HP. The size, price, and fragility of the components is the issue.
GPUs/XPUs are small and solid state so it’s only really price that’s a huge liking factor.
And the disinclination of these companies to push the weights of their cutting edge models into people’s cars where they can be dumped.
lol. Wait until your cloud frontier LLM stalls / disconnects due to load / interference while your car is on highway OR making unprotected left turn OR approaching pedestrians.
It is easy to make car driving *demos*.
Maybe, but the opacity level of models is not acceptable for cars. "Why did it drive under the semi?" "Model said to." "Why did the model say to?" "shrug"
But if the model is an LLM, you actually COULD ask it why it drove under the semi, and it would give you an answer. Now, you may argue that it will just be generating a whole new, backwards-rationalized post-hoc explanation of its own behavior given the logs that it managed to take before the crash. But then I ask you: how do you think a person explains why they did what they did after a crash? I direct you to all of the unsettling split-brain neuroscience literature demonstrating that humans are incorrigible backwards rationalizers who make for unreliable witnesses.
Sounds really expensive. I think OpenAI and Anthropic should really not dismiss making smaller capable models that they can license out in this space on the other hand.
Tesla's already solved this - their vision model does this phenomenally well.
And they've demonstrated adding a sidecar LLM to it as well, mostly for these kinds of "read these 3 street signs, what should i do next?" sort of situations.
The bitter lesson tells you about the trend in the technology. It does not get product to market with today's technology.
But this is a bit of a ridiculous take, no?
You don't need Astra for self-driving. Astra is able to build complex 3D worlds, do your taxes, shop for you, and, apparently, drive a car. A self-driving car just needs to be able to drive a car. By the time you trim down Astra to just have the minimum capabilities needed to drive a car, you'll be looking at the same models these self-driving car companies already use. Then you get to deal with the actual hard problems, like handling failure cases (which will still be present with Astra).
Self-driving cars have been able to do this for a long time. The problem is that it isn't robust enough given the context. I mean, if Astra can drive a car with a single camera, then presumably Astra can drive the car even better with multiple cameras, and even better than that with 3D maps, etc. And when you start to consider the expectation of performance of these systems, you realize that these features really can't be omitted. If you're a company producing self-driving cars, then you do not want to face a lawsuit for you car killing someone because it physically would have never been able to see what it was doing because it lacked a camera.
I think the real gain here is that something like Astra can be used to help build these autonomous stacks. If it is able to drive itself, then it is able to generate novel data, analyze large quantities of data, and use context that isn't typically available when processing this data to make improvements to the actual autonomy stack which is ultimately responsible for driving the car. But thinking that these car companies are going to run an LLM in a car and call it a day is just naive.
The key thing Astra is doing is a loop... (my understanding) To figure out where things are... It's basically use more compute, self-driving cars are usually using on-device hardware where a "loop" might be a little too risky especially if it takes too long on local hardware... I wouldn't want my AI driving model to be over the air either, yikes in the case of lag or network outages.
No, no - all that “useless” knowledge is the good stuff. There is no clean interface boundary for driving a car, because the only interface that has been enforced is “if a human can navigate this situation, it’s fine”. Real world driving situations can be arbitrarily complicated, and if you want >human level driving, you need human level semantic understanding of the world around you. If you see a kid about to throw a model airplane across the street in front of you, you have to bring all your “useless” world knowledge with you to recognize that as a developing hazard. If you’re supposed to bring your passenger to the city building on main and you encounter construction outside with a detour sign saying “for tax dropoff park in rear”, suddenly all of your useless knowledge about the English language, what taxes are, and the likely goal of your passenger given their destination become useful.
As somebody working near the field, I do enjoy the fun of dreaming bespoke vision and autonomy algorithms (if I didn’t, I wouldn’t work in the field to begin with!). But I would drop it all in a heartbeat for a robot that works well. Robust, resilient robots would be such an incredible advance that the ‘how’ doesn’t matter. All of the nonsense from the current AI hype cycle would be worth it if it cashed out in Robots That Actually Work.
Why do we need robots when we already have people?
I just want a robot butler. It doesn't have to prove mathematics theorems, just do my laundry and make lunch.
For dull, dirty, and dangerous jobs!
Probably not. In humans, the visual processing circuitry is very different from the circuitry for language processing. There is no reason to believe GPTs will be effective at it.
If I'm reading the chart correctly, it took over 5 minutes to drive 135m at a cost of nearly $8.00 in tokens. I don't think that's really in the realm of practical yet.
I think this a slight different lesson. There is one algorithm that is called transformer, rest is irreverent/performance optimization.
Self-driving tech is more about reducing liability than the driving itself. The lidars and 3D maps and world models and everything else is needed to get reliability from 99.9% to 99.99% on public roads. This isn’t a SaaS product where the target is to be “good enough” at the cheapest cost.
Oh god, this move-fast-break-things thinking is going to kill so many people. We already have aftermarket problems with people adding in untested, unregulated self-driving features.
https://arstechnica.com/cars/2026/09/aftermarket-driver-assi...
Wow! but WHY is this a benchmark?? for comparison tesla's model is approximately 10-15B parameter model (estimating from maxxing the hardware that comes with the car at 16gb ram).
I would assume this is a proxy for general intelligence. A model that can drive a car and do a bunch of other real world stuff is closer to a generalized intelligence that can reason through any task.
Tesla isn't using a general purpose model, they're using many highly-specialized models for a more deterministic system than "hey chat drive this car for me"
Pivot this to analyze and coach human drivers to be better drivers.
"Get off your phone!" "Stay right except to pass!"
I could get behind this.
3.8 flash would be the model to test, it's vision capabilities are excellent (on par with Astra) while also being incredibly fast.
This is quite impressive...
But I imagine this is orders of magnitude more expensive / less efficient than whatever Waymo is already doing, right?
The cool thing is that 1) it's theoretically more generalizable, 2) if we wait 18 months, it'll be 100x cheaper, and another 100x cheaper likely in 18 more months - at that point - something like a Mac Studio inside a humanoid could have these generalized capabilities, and a lot of Robotics problems start to look more feasible - especially when you consider how much better the models could be if highly specialized.
There isn't any model out there even close to as good as Astra at visual/spatial reasoning.
Gemini models punch way above their weight in vision tasks
https://artificialanalysis.ai/evaluations/mmmu-pro
What did they do to Astra so cracked at vision (and computer use). That ARC 3 score turned out to be no joke/fluke. That huge gap between Astra and Fable (in this case) is basically every hard vison/spatial benchmark i've seen including non-benchmarks like playing games (Portal, Factorio, RimWorld).
SpatialBench - https://x.com/spicey_lemonade/status/2096365630190698516
ZeroBench - https://zerobench.github.io/
Robot Arms - https://openai.robocurve.org/gpt-6-astra/
Well, they have the best in class image generator so that probably has something to do with it
I think Opus 5.5 is at same level now. I have seen too many videos made by Opus 5.5 today on twitter.
https://x.com/victormustar/status/2102707412704919910 horse galloping pixel art
https://x.com/LexnLin/status/2102133072585965759 moving train pixel art animation
https://x.com/jkeatn/status/2102441348075057539 painting with code
https://x.com/LCSlates/status/2102503027340988559 video, very detailed prompt though
https://x.com/aj_dev_smith/status/2102504509637587339 generated song/music with code
https://x.com/aj_dev_smith/status/2102575577563570450 another song
I wonder if the companies would be willing to bet entirely on AI driven innovation if liability for misalignment was put squarely on companies, individuals, compute vendors, and LLM vendors. I don’t think they would opt for it, especially if an alternative option to use human-programmed tech was already available.
There is something to be said about emphasizing on liability as a way to freeze or solidify AI Development. Right now it is too unfettered leading to predictions of AI dooms.
This also explains why Astra is so good at video generation. I have an Astra+Higgsfield setup. I could point it to a Github repo and ask it to generate a product walkthrough and it did a very good job by generating fake screens (e.g. with data filled in) from real ones - which wasn't possible in earlier models
https://x.com/tobiges/status/2098294046469022030
"Sam understands exponentials like no other. During a YC talk last year he predicted that AI would make breakthroughs in science in 2026 and solve a major open problem in 2027. Now here we are..."
Now on a new vibe coded website Astra wins the benchmarks ...
What about Jev? :-D
Looks like the "most successful" path drove over empty parking spaces and came close to two curbs?
New pelican on a bicycle?
Genuinely though, this is fun but not at all what these models are good for. It's like cooking a meal with your feet or somthing. A youtube challenge video from 2012
interesting, im wondering if models like jev could drive a car too?
I'm not an expert in the LLM space, but I'm an external contributor to comma.ai's openpilot project and I'm and quite familiar with how its controls work, so I looked from that perspective. There's two questions here:
1) Could a cloud-delivered LLM figure out how to drive this route, based on those input data and given access to those output actuators? Looks like yes. Sure.
2) Will this work in the real world? Absolutely not. Three reasons: latency, latency, and latency.
openpilot's driving model updates the target curvature and acceleration at 20Hz. Every millisecond of the round trip time through every piece of its entirely-local driving stack is well-understood and tightly optimized, because it has to be, otherwise you can't react to even minor bumps or wind gusts, much less rapidly-developing traffic situations.
Adding even a single speed of light RTT to a cloud service is meaningfully bad, and you'll need a whole lot more to encode and upload camera imagery to even start the time-to-LLM-response clock, and then send the response back down. By then the world around the car has moved on.
There's a reason Tesla and every other self-driving manufacturer need the compute hardware in the car.
As an external contributor to comma.ai, do you feel like you helped contribute to these deaths mentioned in the article?
https://arstechnica.com/cars/2026/09/aftermarket-driver-assi...
Perhaps there's a synthesis to be had though. Eyes, control, and safety critical features on the hardware, higher level decision making to the cloud. Openpilot's biggest weakness has always been in the very "robotic" way that it drives, which is technically correct but causes frustration for other drivers. Deciding "should I pass this car" is a fundamentally different question to "can I pass this car", or "what is the actual safe speed and following distance given the current traffic conditions and weather".
I'm far from neuroscience, but humans don't need to operate at 20Hz to drive a car. And human reaction latency (event to measurable action) is often over 1s (under 1Hz).