I linked to the tweet that has the video because if you are not signed in you cannot see the whole thread of tweets.

I can see the individual tweets in the browser while not signed in though.

The doom demo is also in the article, for anyone that doesn't want to go to X.com. :)

Link to a raw MP4 of the video, from the parent article: https://framerusercontent.com/assets/rlL7ImEbISFoYt3IJEHHfvj...

It's in the parent article under a section named "Doom" in case that asset URL ever changes.

i think its just telling the model that it cant output a shoot action

I'm not sure the authors realize this is way more than "just a cool demo": if this holds up, it's going to be huge for game QA work.

Instrument your game to output properties of entities near the player and the output is the various control inputs - moment to moment gameplay gets solved. Maybe augment with a tick-by-tick controlled stepping mode if particularly twitchy - an LLM can take care of the higher level reasoning then.

It wasn't until the demo videos that I realized the post wasn't satirical.

Flip both, Cario Vmodei.

Well now I’m rooting for them!

thought the same lol

Wild that it doesn't generate text. I wonder how its technology compares to Tesla's FSD stack.

Signed up for the beta! :) would love to put this through some real-world shootouts against traditional LLMs to see where this type of model really excels.

I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.

https://typesafe.ai/blog/antibenchmaxxing

But also effectively this is a classification model. It excels at specific certain types of workloads, and obviously will fail at others. Not really sure how one benchmarks this tbf. I can see their argument on why this requires a novel specific eval for whatever your usecase is. A consistent "global" benchmark might be hard to do

Looks like a great model for NLP.

The whole video seemed generated to me

definitely not AI-generated - this is my real wardrobe

we also thought the voice at the end was AI-ish, but apparently that's a real voice actor but slightly sped up

Pretty sure that's an intended joke.

Reminds me of this: https://www.reddit.com/r/ITcrowd/comments/tg05j1/i_cant_beli...

Cool project!

That's what I was insinuating through "better encoder"; the model creating more efficient representations of ASTs using something like JEPA

Yeah it would be quite trivial to try and implement an auto regressive AST generator for STLC with Jev provided that you had bounded variable names and integers.

As you said, if it worked, they would have demoed it haha

I could see Jev being great at finding key symbols in codebase before a code generation/code review task. I sent you guys an email (to hello@) about using Jev in Code Review for www.ellipsis.dev.

im not seeing it.

youd ask it to pick a location on the ast to add something from the grammar?

i dont see how this stays confined well enough? make a new output space every time? does that end up auto-regressive?

There was no "AI Winter"

join the discord! we love forward thinkers

I could put this to use today.

I think we'll see a bunch of different architectures over the next five years.

The doom demo shows the program state / query.

DSPy seems like the right comparison and this is the first comment I've seen mentioning it.

Thanks for putting this together. I'm surprised the cost saving is so little though. I expected much more based on the post.

Super intrigued by this - large scale automation using LLMs is quite annoying due to deprecation cycles of models from frontier labs and cost of running your own being prohibitive when you have a blend of them.

The thing is, in this climate it's hard to believe such tech will remain secret for long.

So, assuming this is not vaporware, this would raise the tide for everyone because it shows what's possible.

That's still ultimately privacy by contract (where you have to trust the inference providers to uphold their end of the deal), rather than privacy by design.

I always setup guard rails so that only zdr providers are used.

They don't like adding stealth startups :(

It can be shoehorned to work with OpenAI's newer Responses format.

They have various benchmarks, e.g. how much time it takes them to do wikipedia page -> page games. Jev seems to take the same or fewer hops but in ~10x less time and for ~10x less money.

It's totally reasonable to compare against LLMs doing chain of thought if it gets comparable performance.

It's not an LLM though it's a frontier model on structured data

so what? put an LLM on Cerebras and get its responses faster, and put Jev on Cerebras and gets its responses even faster

i've seen it play minecraft as well, what im not sure here is what is the thing that produces the JSON and keeps track of the objects

can jev play battlefield six for example

Congrats ! Really excited for the team.

If Jev is skipping generation entirely for a narrow structured task, of course it's faster

I think this is reasonable if people are actually using LLMs to solve this type of narrow structured task, which they are. The evidence is that every LLM provider has some method of forcing the output to conform to a json schema in their documentation.

Comments under the x post.

they’re talking about two totally different things, right?

it's our output tokens that are free (under the system one / jev column)

The output tokens are just responses to your inputed questions and their probability. So relatively few output tokens. No unstructured text back in the response.

This is not confusing

What is it about the rendering of this page that is so... off? It almost looks like the entire thing is a <canvas> element.

edit: looks like a framer export where there is a text stroke being applied :|

Bingo. It's a Psychology term for the part of our brain that reacts instinctively rather than thoughtfully and logically

I don't know if they changed it since your comment, but it's all just text to me

We deliberately chose not to publish performance against public benchmarks. In fact, we plan to only have one-off evals when we make product updates.

lol, I bet they would publish them if their score on those benchmarks were good.

"is this the real thing or is just fantasy"

yup just joined the waiting list with a very similar use case in mind

This ought to work better than SpamAssassin, I'm sure

it is just a model, no harness yet ;)

it is a structured data model, but technically not a language model (it doesn't generate language)

If you provide it an AST of the english language, yes.

you could, but it the model is not optimized for text

this is complex, but generating text is highly complicated and requires mode dropping to make long cohesive text

Strings trigger us

was wondering same

Feed the generated note back into the input for the next query and you have ... autoregression?

It looks like a specialized encoder-only(-ish) transformer with scalar and ordinal output heads. Acausal in effect, maybe? Probably not even autoregressive?

I'd use this as a tool an LLM can use for specialized tasks. It's not AI in itself.

If this is true means, AI Stock bubble burst. (For good)

1. I am extremely on the same page 2. I do think that subconscious is not only much smarter than we give it credit for, but also much more robust than the "jagged frontier" of current LLMs

(shilling my blog post on that jaggedness: https://www.completeskeptic.com/p/lies-damned-lies-and-bench...)

I think that's to demonstrate its speed

I was confused at first too, but it makes more sense when you read about their primitives. E.g. https://docs.typesafe.ai/primitives/noul

The demo is showing System One producing its output in parallel very quickly and for little cost compared to an LLM generating its answers token-by-token. The "noul" type is used to evaluate a yes/no question and return the probability that the answer is yes.

So this demo is showing System One offering much more nuanced responses and specific probabilities compared to an LLM's more crude responses (e.g. LLM shows "true" or "false" compared to "0.9" or "0.07" probabilities that the answer to some question is true).

very accurate!

the one nuance I'd get into is I'd call it "zero-shot" over "instruction-tuned" (the latter often implies a particular distribution), but very safe for sharing

architecture is close to the chest for now, but we have talked about writing a paper

I don't want to shill my blog too much, but I will say data is probably far most interesting than architecture: https://www.completeskeptic.com/p/the-bitterest-lesson

It's unclear if the context extends as the conversation grows?

It is Bernoulli (mentioned in the comment I replied to). I just failed to guess that myself :)

I assume this isn't really for consumers/individuals currently?

Unless they're hackers, no. It's not really a chat interface, it's meant for consumption by machines and composing into higher level systems (pairs great with LLMs).

Is there somewhere to read more about that? I assume customers would just have some tests or talk to you.

We're going to release some more info on evaluations over time, and yeah, join the waitlist! We offer faster access in exchange for good memes

If this is true, then AI Stock Bubble burst (for Good)

Non-hallucinated ones at that.

Creativity is not required for AGI, that's maybe the only thing that is not required for AGI actually.

What a sad world would you live in if you don't keep creativity for the humans.

we have played with this! the fascinating thing we've found so far is that adversarial examples for our model are quite different from that of LLMs so that they work even better together

This is basically what they have. https://github.com/fastino-ai/GLiNER2

That was the first thing that come into my head. OK I can train very simple model, that can generate json's for specific tasks, so what? How we can be sure that this "limited use cases" not just overfitting for particular outputs (or even distillation?)

Except this, this thing looks like revolution.

Last year I also had a rather similar idea, but dropped it before I went very far in working on it. I wonder if you and I had similar ideas?

1. Start with an LLM, so that your model understands natural language.

2. Replace RoPE with a tree embedding scheme, and causal attention with a sparse attention on the graph structure. (You could use full attention... but it's cheaper to use graph attention.)

3. Chop off the final unembedding layer, replacing it with a projection down to two scalars, one for logits and one for confidence.

4. Each option of a choice is represented by a number of tokens in leaf position; average these tokens' logit outputs to get the option's logit. Average all of the confidences from all of the options to get the choice's confidence.

5. Train the logits by KL divergence from a true distribution (or NLL on samples from a true distribution).

6. Train the confidences on a subset of the data in which you know the entire true distribution.

The hardest part is getting real world data for workflows, but I wildly speculate that you can get by with only ~50,000 documents if you first adapt domains using synthetic data.

Yeah saying it can't hallucinate is crazy. It can still forward a billing query to the dev department incorrectly. It can still get an obvious yes/no question completely wrong

oooooh it can play Doom!

forget LLM benchmaxxing sidequests, I'm sold on the real benchmark

That's kinda the goal. Imagine all the automation in the world being able to embed intelligence directly inside it - factories could route based on more complicated questions, hardware could anticipate your needs. Customer support could be done without humans 90% of the time.

Funny how it can do everything but not chat. Sort of how when I was a kid I thought of a medicine that could cure any disease except the common cold.

this is interesting, so not an LLM but can be used in these use cases that LLM's have been shoehorned into

https://docs.typesafe.ai/concepts/use-case-map

exactly right!

Only if you think that everyone cares about self-driving. Lots of niches require structured domains; self-driving is just one that has a lot of capital thrown at it.

That's a great point! It quite looks like the System 1 model of Physical Intelligence

that's right, but because these models are probabilistic, it's also possible to be confidently wrong (and all future models will be smarter still and still have that possibility)

Yeah. Yet another reason why open-weight models are better. If I want to use the logits, I can.

RLVR generally upweights tokens along the whole thinking trace that led to a correct answer, whether each token was "correct" or not. RLVR doesn't train a model to output an 80% likelihood, it just trains it to produce correct answers, and not to produce incorrect ones.

System One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it:

System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.

It's pretty darn smart. If you did want to hack on it in earnest and find out for yourself, send me an email - nathan@typesafe.ai

I'm certainly not resigned to that, at least for classification.

Even non-frontier models are absurdly good at this in a broad sense.

Which would make it hard to judge "a model that will never produce unreliable outputs in the first place" against something that is already really, really good and exceptional in domain-specific areas with the tiniest amount of elbow grease.

Speed and cost look good though (for now)!

Right and so maybe we should stop saying "can't hallucinate" when it can by definition.

It’s not what people are looking for, but what they wrongly claim.

In one universe that is true, in another one not.

It's not a binary thing. You can get closer or further away from that standard.

And humans also behave differently in different contexts. A conversation at the pub has more such hallucinations than a formal deposit in court. For the latter, a good lawyer will look at her shoes, when you ask him what colour her laces are.

Yes, and for the LLM you can do it in multiple passes.

So you can bolt the fact-check / source-check pass onto whatever other system you have, without having to redesign the underlying system.

No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.

What we would want to see if a confidence value that is in line with the actual correctness. If the value is 0.9 for 1000 different answers, then approximately 900 of those answers should be correct.

I read "hallucinations" as "generates novel output with no grounding/source". i.e. "it just made something completely up".

I believe their "accuracy" metric (sonnet 5 level) is where "right/wrong" is measured.

What about the LLM calls though that are done midchain? In the Home Assistant video the multi-intent prompt gets split using what looks like a traditional llm model, which I'm assuming is vulnerable to classical hallucinations.

That's really funny when you consider that generative models also don't hallucinate if you check up on them on every token generated?

From a quick look at this it looks like it could easily generate natural language text by following a structured representation like UMR (Uniform Meaning Representation) or the similar representation the Abstract-Wikipedia folks will be working on for generic encyclopedic text (which will be heavily informed by Universal Dependencies). These are basically linguistically principled and frame-based counterparts to a programming language AST, that can be then converted to natural language (in a broadly language-independent way, to the extent that semantics and pragmatics make that feasible) via some sort of NLG rendering.

(To be clear, this one raw model does not support outputing a full AST directly - it wants to output "choice" among fixed options, "score" on a sliding scale, or a true/false answer (all of these with confidence scores attached), so building the AST/structure would be a code-driven (or even perhaps outside LLM-driven in some more challenging cases) multi-step affair where the model would essentially be playing a "game" of building the structured output step by step and getting a revised partial state back. But one could expect this to lead to interesting results.)

https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...

I'm biased but I wouldn't call it misleading

- @CompleteSkeptic

Very strange.

And furthermore, because the model is forced to answer in a boolean (if in boolean mode), if the user input is outside of the range of a boolean, it's forced to hallucinate. It can't abstain.

Let's say classifiers don't hallucinate. To make a fair comparison we should constrain LLMs to the same classification task. In that case, no, LLMs also don't hallucinate.

- Give Jev and LLM the same input

- Lock down both to approved/rejected/unknown (LLM restricts on decoding)

- Both can be wrong, but neither can hallucinate (invent an another option).

id say yes. a linear classifier that classifies between red and yellow balls will hallucinate on blue.

linear regressions hallucinate in the simpson's paradox.

the model output can be quite confident and not representative of reality

hallucinations are not wrong answers, that's why we use a different term

To be fair - you’re crafting a deliberately bad model input for a contrived example.

A hallucination in the context of LLMs is generally understood as an incorrect answer presented as factual. If you claim that "x can't hallucinate" in the context of LLMs, you're saying that x always gives accurate answers. It does not matter whether the answer is type safe. If its value is incorrect, it's a hallucination.

I very much agree with this

From the intro blog

"Hallucination and type-safety are intrinsically related"

I'm not entirely sure why we're conflating type safety with, I guess, value or output safety.

"Would you say a linear classifier hallucinates?"

No, but it can be (and often is) mathematically correct and functionally incorrect. It doesn't help to say "a linear classifier can't hallucinate" when you get even 99% accuracy. That's 100% a semantic play, and it doesn't help when the picture of a dog is labeled cat and the response is "yeah but that's not a hallucination, only stupid LLMs do that"

The inputs are natural language, they're just also structured into a tree. The first example on that very page shows natural language instructions:

    questions = {
        "refund_requested": Noul(
            instructions="Does the customer request a refund?",
        ),
    }

It's nothing like a traditional LLM and so should not be compared to one.

Frontier LLMs are expensive jack of all trades. You can absolutely compare them to purpose-built tools on any domain they touch. Engineering is all about assessing tradeoffs.

Well, not quite a week: https://x.com/harshagundal/status/2100044305536889015 - apparently it took him 2 hours.

Large language models are not the only type of model.

It is frontier in the sense it is exploring an unexplored domain. I do agree on questioning the comparatives though. Speed/cost is indeed relevant for problems that can be framed as structured decisions only. The question is, would defining a structured decision model be a structured decision model itself? This would significantly increase the application domain.

I feel like the power of the approach presented here is that it gives a model a proper "language" to describe computations directly vs moving tape silliness.

I foresee this to be the path moving forward - giving AI models understanding of the computation directly(as well as compositional rules) This feels like a short path towards total software in many areas.

Agreed. It's a wildly dishonest presentation of their product from many perspectives, which is a shame because it might actually have some good use cases.

The comparison between LLM speed and Jev speed is misleading, because they're using autoregression to generate all of the type names, all of the schema, etc. A closer comparison would be if the LLM was purely outputting the raw numbers. Even then, comparisons to LLMs are pointless because you could train a transformer on the same sort of task that Jev is doing and get even better performance yet again, and a smaller model. I suspect this is some form of stripped down diffusion language model.

You really have to do a lot of hand holding here, and map out your problem space manually, and very carefully, to get any sort of accuracy. For example:

Keep each Score to one dimension. If a description says “punctual and smart and experienced”, the question is measuring three things, and an input that is high on one and low on another can’t be placed. Confidence drops and the score means less. Split it into one Score per thing and combine them in code

If you don't perfectly represent the distributions of possible answers then you'll likely get garbage results. As far as probabilistic state machines are concerned, I'd say creating the distributions of possible answers, and their hierarchy, is the actual hard part.

One of their examples is:

- "state": "I have asked three times now. Can I please just talk to a real person?"

- "Is the customer asking for a human agent?"

Imagine the users request is: "I want your human agent to call me tomorrow at 5pm."

Human conversation is fuzzy, getting useful reliable results out of this is going to be a challenge. Of course, you could add follow up checks like: "Do they want that now, or later?" -> if later -> "Do they want that tomorrow, or the day after?" and so on... But now you're building an LLM out of if statements. I am skeptical of whether this model has much utility for fluid language interpretation - I suspect it'll only be useful for scenarios where you've tightly constrained the answer space but want to use fuzzy language to describe it. Like:

- Question to human: "Would you like a support agent RIGHT NOW?"

- Their response: Yes | Yeah | Mhmm | ye sure (any possible yes signal)

Model input: "Did they ask for a support agent?"

Still... a tiny LLM could accomplish this sort of thing without problem. And that doesn't stop someone from saying: "No, not right now. But tomorrow." - and the tomorrow would get missed. I think this is why people haven't really tried this approach much already.

Also their Doom demo is on structured state, not on images. Meaning, the enemies must be being served to the model as coordinates (or the exact angle of projectiles that hit the player), otherwise it'd have to scan every pixel of the 360 degrees to know whether an enemy is in front of the crosshair or not. You can see from the map below that it's also choosing travel checkpoints/destinations through walls. So they've severely cooked this to make it look far more capable than it is in practice, and any speed advantage that is offered here is not factoring in the shortcuts it is taking, the training on the map, and the fact that it can cheat because the structured state it is using is not bound by obstructions.

Here is their docs by the way: https://docs.typesafe.ai/ - so you can understand how it works.

Has LLM become so synonymous with Generative Transformer that other high-parameter count models that interpret language need a different name?

For all we know this might be a non-language-generative transformer e.g. a transformer where the decoder produces confidence scores rather than language. Please provide more likely architectures if you know them, I'm genuinely curious.

AFAICT it is the same interface as you describe, but the underlying inference algorithm is fundamentally different, hence the speed gains. There is an application I am currently working on right now where this typed output predictor is the performance bottleneck. I'd be very interested to see how this performs.

Yeah, I thought about constrained generation as well. I've actually done something similar with local models before. And you can even get a "confidence" score by looking at the logits (something along the lines of logprob("YES") + logprob("Yes") + logprob("yes") - logprob("NO")...

There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.

That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.

edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...

Forgive my ignorance. Tabular foundation model?

So... a classifier model?

Will need hands on to truly tell, but the doom demo seems very promising. If it can play that with text descriptions of where stuff is by distance and degrees in a 3D context then many GUI automation tasks should be easily doable

Not in the traditional sense of a coding agent, but we think there's a ton of opportunity in using it for context management ("do we _really_ need to pass all these tokens to the agent?"), semantic linting ("how does this score against this AGENTS.md: <...>"), etc.

What's the difference compared to just taking an embedding and feed forward a simple net trained for the task?

This smells like a tool a more broadly capable LLM would take advantage of extremely well.

Great question! Yes, this works much like the doom player. Sensor data (LIDAR, velocity, etc.) becomes the state. You use the score primitive to operate the controls ("What level of braking should be applied" 0: None, 1: just slightly slowing down, 2: there's a suspicious cat on the side of the road you don't trust, ...

Full disclosure, I am not they :=)

But the real problem in self driving isn't the decision making but object description. That is, computer vision if with cameras.

Decision making isn't that of a bottleneck I suppose.

Example I have worked: Personal delivery app, that tracks packages from various senders using incoming emails.

I am using the single prompt approach with GPT5.4, which is free, but it’s not reliable. Using Jev I’d decompose the prompt into a bunch of smaller questions, then I’d combine the answers in software. I’m super excited to try Jev out.

Congrats on the release!

Finetuning a language model for decision classification (with probabilities) is already well-understood. What specifically changes in the training objective with RLCD? Are its benefits isolated from Jev’s new architecture/parallelism?

Interesting concept, I can't see a reason to use a generalist classifier over an api rather then just training my own? If it was open weights I would probably mess around with it.

thanks a ton!

constrained decoding (OpenAI-style structured outputs) make models dumber unfortunately - the short+dense version is that simply masking logits is insufficient because if ever a model was assigning probability to an invalid token, the model is by definition confused. you'd be better off erroring IMO

If I’m understanding correctly, this will work well for self driving cars?

I’m not understanding what this is. It’s a faster cheaper LLM?

Congrats on the launch! What's different between Jev and Microsoft's Guidance package? https://github.com/guidance-ai/guidance Is it a diffusion generator under the hood?

Is there a downloadable technical report somewhere?

From the person in the video regarding issues with benchmarks in general, and for LLMs. Also their approach. Good article.

https://substack.com/home/post/p-215252866

Don't your numbers suggest DeepSeek V4.1 Flash, for $100 more, gets you to slightly better agreement?

Current models go beyond the simple embedding because you start to encode groups of concepts in the context-aware part of the model (attention heads or any other method). So it is never simply words/tokens in isolation anymore.

inputs are structured program state. there is an example at around second 30 of the doom demo

(though ideally everyone gets off the waitlist and can try it out for themselves )

You can get rigid output format from "classic" LLMs https://docs.vllm.ai/en/latest/features/structured_outputs/ though model support is limited.

Would like to have something like in the original post but open weights.

I wonder how well it can play chess, or go.

I had a different initial confusion - it seems this company has no relation to the company formerly known as Typesafe https://en.wikipedia.org/wiki/Akka.io

What even is "system one"

I definitely agree it's underexplained in type safe.ai's materials.

I have to assume it's a reference to the fast, heuristic, intuitive "system 1" process in humans, as opposed to the slow, procedural, reasoning "system 2".

This theory is recognized, among others, in Daniel Kahneman 2002 Nobel prize on Economics.

I don’t see this as an option in their website

You could theoretically ask “what is the next appropriate character?” and add the entire ascii charset but i doubt it’d work well and you’d be implementing autoregressive churn across network latency…

strings (and all sequential data structures) are not allowed at all - this is how we make sure all outputs can be computed in parallel (thus no output token cost)

"While Jev gives up string generation, it’s optimized for structured outputs and can’t hallucinate"

Ouh! Any open weights models that can do this yet?? If not, how much longer? I have a Mac Studio coming soon.

what is the…epistemic status, for lack of a better way to put it, of the probabilities? what do they mean? what (probabilistic) guarantees do we have about, say, the responses to

- is the capital of france paris?

- it is august. is it raining in paris?

(forgive the examples; they're probably not semantically the sort of thing jev is trained to work on. but i figure the point translates to various kinds of questions that come up in "inner loop of agentic pid controller" contexts)

a normal text-generating model if asked to produce a number will also do that just fine. i assume in jev's case it was actually rled to essentially learn to express priors over things using its implicit world model, which definitely ought to help, but can we say more?

The technology and the results are very handwavy. What is RLCD exactly ? What are scores on benchmarks compared to LLMs ?

This website does not inspire confidence at all, it all sounds like a marketing piece. I wish it was true, some kind of text-prompted classifier with LLM performance would be cool, but I can't trust it with what we are given.

1. yes a general model 2. no training at all 3. but it is focused on "System 1" tasks (more human judgment, less math reasoning)

It's very generalized. Can't wait until everyone can see it.

The impressive part is that it is low latency enough to serve high quality answers at game speed through the model instead of a pre generated ad-hoc machine.

this might finally be smart enough and fast enough for jarvis. hard to feel like iron man when your assistant takes 8 seconds to decide to pause your music

The eval is baffling me

we assume there is a correct compute graph (a “workflow” represented in code) and use the predictions of the largest, smartest, and most expensive external models as reference probabilities. ... Rephrased: every model gets the same workflow. We test how they compare to the average of the smartest models (in this case, Astra and Fable).

They assume there is a correct graph, but they don't compare to that, they compare to the average of the smarts models? So the smartest models are getting it wrong but you compare that anyway as a benchmark? So the outcome is "how much of a Fable am I getting" etc. Why not compare the actually correct thing?

But then even on this hand constructed eval, the first plot is showing Jev at less than Sonnet 5 accuracy. It is barely better than Luna. There are two Opus 5's and two Sonnet 5's without explanation. What is the plot showing?

I gave up.

Hm, would be good to understand the architecture better. Is this answering just from a world model informed prior? How informed is it by the information in the prompt? I can't see this maintaining calibration across all domains and all types of structured output.

Is there anything published on how it maintains calibration? Or when you say "outputs calibrated probabilities" you mean "as calibrated as frontier LLM models, just cheaper" - which is a different claim; as LLM's aren't particularly well calibrated

yes and can do many of those in parallel

you can still chain them

They said that it works with up to 255 options.

Funny how the authors are asserting that "doing the right task > data > compute > algorithms" while simultaneously releasing AI model for calibrated decision making, which if they work, would mean that "compute > doing the right task"

love it. send me an email and i'll try to get you moved up on the list? nathan@typesafe.ai

This will be insane for tool usage, and probably where the major economics for day-to-day usage will be.

The goal is going to be to use llms to distill operations down to some dsl, and pass it into something like Jev.

This makes me think of Expressions of Change [1], a project that aimed to make updates to a program a first-class primitive in a programming language. A model like this can't output code directly, but perhaps it would be well suited to select from the small set of discrete operations on code envisioned by the EoC author?

[1]: www.expressionsofchange.org

the edit is right - jev would be cheaper, faster, and more self-consistent (in general)

we actually use astra (and fable) in this way for our evals: evals.typesafe.ai

someone on the team cooked hard on that and it shows example traces comparing our model to opus/sol

Yeah. Computer use is essentially a model navigating the OS-provided accessibility tree. I imagine a model trained on it would operate the computer exactly as we saw it control Doom.

https://developer.apple.com/library/archive/documentation/Ac...

This is amazing. I really could use this.

I like the idea of System one models but all LLMs so far work as system 1 thinking because humans generate speech subconsciously with system 1.

System 2 thinking requires consciousness which AI does not have, so even reasoning models are still system 1 thinking as system 1 in humans has reasoning with heuristics.

Its limited but most people navigate the world with it completely, so it's enough for AI.

I can't see how this is different from a fine tuned LFM2.5 encoder

I think this is a great direction -- for some kinds of users. And this makes me wonder if the 'vs' framing is misleading.

Yes, I think it's a mistake that many organizations are cramming LLMs inside of automated pipelines where the extreme generality/flexibility of the model is at odds with the fact that you're using it for a very specific task that gets repeated over and over, and needs a very specific structured output to be successful. But specifying your task carefully (as well as deciding what counts as your input state representation etc) seems like a form of programming. Something (a person or a model working in a relatively unrestricted way) will need to produce a configuration/specification for this system.

So rather than Jev vs Claude I imagine that using Claude/ChatGPT/whatever interactively to define / refine your Jev config which then runs in prod might be the happy combination?

Seems that way.

Can this be used in conjunction with a text-generating LLM for better quality code generation?

defining the workflow such that the operation is a set of relevant questions

I think the doom demo uses a text representation of the world and it's basically, "projectile coming your way" -> "Strafe". "Enemy ahead" -> "shoot. So it works well when spawned in a room of enemies (as we see in the video).

If self driving is red means stop, green means go, and stay in your lane - then it would work great, but having to actually think and test which maneuver is optimal for a given situation while weighting safety, road rules, random unexpected actions and getting to your destination, I think it's a much bigger problem. A bigger model specifically trained on that maybe would do great, but then the output is not the constraint anymore.

But I haven't tried the model, so I 'm just ballparking and could be very wrong.

The model doesn't have image input capabilities (yet, it seems from the post), so for the Doom demo, a harness is extracting a bunch of structured information from the game (map layout, enemy locations, player ammo, health, etc) and providing it as a massive JSON blob to the model so it can make its decisions. This model _could_ be hooked up to make the decisions for a self-driving car, but it would need to be fed a structured blob of the situation around it, so all the computer vision problems of self-driving are still there. And that's before you get into the confidence and accuracy of this model.

Driving is more complicated than Doom, and it doesn't look that great at Doom to me.

This is potentially huge and can crash the Big Two's stock prices or block their IPOs completely.

Golem, if you read this, add me on battle.net (europe) Sansviande#2540 and let's talk. Give me 1 minute.

“Crime against humanity” buddy please

I definitely agree with this, but the transition is gonna suck for some.

If we could come up with a system to classify the probabilities across a large number of candidate words (or components thereof) then this could actually be good at producing text, one element at a time. We could call these elements 'tokens' and picking the right one could be called something like 'decoding'. Crazy idea but hear me out...

On a more serious note, it will be fascinating to see how this different spin on modelling inference will create new paradigms or slot into existing ones.

Thanks for the early access! I was testing the Lisp idea out in the playground, but I don't think the model is smart enough right now to generate actual code. I tried having Jev finish generating the code for a Fibonacci number function, but it kept wanting to create a literal number instead of refer to a variable which is a number. This happened both when I gave Jev the current program as a string and when I gave Jev the program as structured data.

Maybe I'm just not doing a very good job at prompting Jev, but I think right now it's not quite capable enough to generate Lisp code.

Link: https://console.typesafe.ai/playground?share=shr_148e1248984...

You know you're too old when you see the company name and think! Oh I wonder what Martin Odeskey , Jonas Bonér and co are up to. Wait, didn't they become lightbend... Altho this comment takes away from what these guys are doing which legitimately sounds interesting.

Any relation / inspiration to GLiClass?

API example[0] makes it clear how it'd be used:

  from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
  
  with TypeSafeClient() as client:
      response = client.system_one(
          state={"document": "I was charged twice. Please fix this ASAP."},
          questions={
              "billing": Noul(instructions="Is this ticket about billing?"),
              "tone": Choice(
                  instructions="What is the customer's tone?",
                  criteria={"calm": None, "frustrated": None, "angry": None},
              ),
              "urgency": Score(
                  instructions="How urgent is this ticket?",
                  criteria=["can wait", "this week", "today"],
              ),
          },
      )
  
  print(response.nouls["billing"].noul)
  print(response.choices["tone"].choice)
  print(response.scores["urgency"].score)

[0]: https://docs.typesafe.ai/sdk/python

An example with manual combinatorial exclusion in “not_for” field made me cry, this is a wild hybrid of code logic, textual definitions, and AI blackbox. It’s a cool idea, but the “glue” layer is too boilerplate-ish

If it is so cheap, why such a limited release?

We've already started using it for some pretty powerful decision tree stuff. We're just scratching the surface. We shipped an extension for Swamp[1] a few minutes ago and the combination is great!

The one downside is that the context window is very small (32k.) So some initial ideas we had for initial evaluation of code reviews won't fit yet in the window.

1: https://swamp-club.com/extensions/@swamp/typesafe-ai

For what it’s worth, I didn’t get that impression, and even noticed a couple typos ;)

I really appreciated the hand-written release. Thank you.

you will need to collect data for every decision/usecase and then train a model. But this can be used for different use cases with just a prompt.

Founders response to a similar question on X: https://x.com/CompleteSkeptic/status/2100067328620896408?s=2...

pasting it here: zero-shot + general == programmable

I would assume any extreme scale narrow task could then be fine-tuned for, but we'll see - I suspect putting it all in shared cognitive core has bit maintainability/generalization benefits

While I understand that accelerating development isn't necessarily the target for this, and it's not at all intended to generate code the way many of us are...

I think this could be pretty decent in CI? There's a lot of "flakes" I've mediated that this could have handled much more efficiently. Maybe observability as well, triggering elevated logging and other initial measures?

ok I've searched, it may not, but it can do "driving car" and "trade 0-DTE Option" much better.

Is this a Markov/Diffusion model with some sort of external Engram memory? If so, this could be extremely interesting.

Looks promising. I'm building an AI video editor and multi tool calls take >30s using Gemini. This would be a a game changer if Jev can take that down to single digits at p95.

This looks and feels a lot like productionized conformal prediction

I can see the value in this but looks like there's going to be trouble in communicating the difference between this and a regular LLM, and also proving the potential cost savings in using this to replace existing systems that are using LLMs with frameworks like langgraph, as this can't be a drop in replacement and would require a significant amount of re-architecting/reengineering of systems to get the type system to work

Huh this looks fantastic. The Doom demo really sold for me that this could be a great tool for accelerating QA at my gamedev studio. Signed up for early access.

Agreed but is it much easier to deal with if you need to have all of these sub processes integrated? How does one know when you need to reword a request? What if Anthropic then has a type error, then debugging that just got harder.

That's fair, but it highlights how this would actually be used. It doesn't really seem like a competitor to other models but instead a way to make these real systems more enjoyable to deal with.

This video makes a better job at explaining what it is about vs the marketing ones. Thanks for sharing.

Rereading some things and because there's no official benchmarks, I misspoke about the ballpark comparison., but the open model's still a useful foundation to work with

thanks, it's definetly relevant

Most people think it's cool but use it once.

The regular-ceiling-lights-as-alarm service gets positive feedback. Weird how smarthome companies never market that, seems easy win.

Need is a big word. Convenience is also a factor.

We have 8 light buttons in our living room/dining room/kitchen space. It is very convenient to us that we have 1 button for turning all of them on/off at the door to upstairs (at night turn off all lights and go to bed upstairs, in the morning come downstairs and turn on all lights) - but also have 1 on/off button near our back door for when we leave/come home.

Next to that: on/off toggles a schedule where the lights are bright and cold-ish by day, and low and warm by night without us having to manually adjust each light every hour or something.

Again, need is a big word. But it's very convenient and pleasant.

That really helped figuring out what this thing does, thank you.

That's a really nice demo and way more helpful than their website, thanks for sharing it!

Thanks, much clearer that way. The speaker is unusually eloquent and to the point for a very nerdy demo, they should put that on their home page.

Super cool! Instantly joined the waitlist.

It might be boring, but I can see exactly how I could use this right now to improve my agentic rag.[0] In two months I am supposed to deal with a giant corpus, while still maintaining responsive chat UX. I have been working my butt off to make our first big client happy. This could really help solve the chunk ranking problem.

[0] assuming the policies are compatible with sensitive production workloads, some time in the near future.

Very cool. LLMs have been borderline unusable as functions for the longest time, very excited for this direction.

Could you use this to build a proactive memory formation and retrieval system for LLMs that runs lightning fast?

Last 32k of connect + Summary of current task: Did we learn something useful here (true/false)? What is the category to file it under? Then notify the LLM to file it away.

What class of memory might be useful here? Model gives probability to each item in the list. Short description of all memories ordered by tagged class is used in the next round. Are any of these memories useful in the current context, such that they will inform the model and help in its task (yes/no)?

I’m sure there’s some fine tuning to be had, but this sure seems like the basis for a substantially better proactive memory system that works around an existing LLM conversation.

If I’m understanding what this does and how this works (generic input, intelligent classification with probabilities, rapid and cheap), this is absolutely nuts.

This is actually pretty cool. I think the undertalked about part of this for TypeSafe is that they can always "extract"/distill the frontier of this type of task from the newest LLMs for cheap. Jev seems seems to be GPT-6-Astra/Fable 5.1 but I imagine a bunch of training data is from earlier models?

Then, you can serve it faster/cheaper than the frontier LLMs. It's basically distilling a small but extremely common use-case from LLMs and serving it. Then RLCD comes into play to update weights when a new model comes out, etc.

Any thoughts on what the next potential "cheap" win to be distilled from frontier LLMs is? I'm going to need to play around with this.

afaik Jev's context window is 32k

my sons name is also Jev

Can HN have a tag for open-weight vs closed-source models please? The progress is nice, but if it is not released at least in papers or open-weight? These are just ads?

I think I missed why is this faster? What I’m reading here is it’s similar to constrained decoding but I’m not seeing the explanation of why it’s able to get those results.

Seems like "some" of LLMs tasks are now Jev tasks.

insane doom demo i wonder what the limits of its intelligence are? i'm guessing it's not great at reasoning tasks, it seems breaking down the problem helps significantly, but how much does a problem need to be broken down for reliable performance? also this would be huge if it could run locally but it seems like there's no intention to do that at the moment

Hallucinating as we use the word really only applies to generative AI. Non generative AIs can't hallucinate, they can just be wrong.

System One seems to be more of a "class" of models, as it's a good classifier but can't do what traditional LLMs with chain of thought do.

As a zero-shot classifier, I expect that effectiveness is dependent on the data trained upon.

Jev input … > Unstructured data (e.g. text) with an emphasis on structured program state.

What pre-training data/model is Jev based on? Surely result effectiveness is dependent (outside of one’s own input as “state”) on that?

LLM seems to have become synonymous with Generative Transformer architecture.

While this model may share much with GPT-style models on the encoder side, it clearly has a different decoder architecture. So is a high-parameter count language model an LLM even when it doesn't have a GPT-style decoder? The definitions are in flux.

Cool I guess. Definitely not worth the 1000+ points though.

Wonder if this could lead to better recommendation algorithms.

Crazy, looks like this was just published a few hours after the TypeSafe post!

I like it. What is it?

This is too much for me. ML playing doom was a thing since before LLMs, decisions tree were always insanely and no one ever used then anyway, i can't see anything new in this yet everyone is treating this as a revolution. This technology was always there and quite easily accessible all along.

I'm trying to understand what difference does this make over LLMs.

LLMs are universal simulators, their latents model the world. So I bet if you compare their logprobs with probabilities output by this model, it will be highly correlated.

Someone should do this quick experiment. I bet there won't be enough of a meaningful difference.

Great to see something new..

However I don't understand how are they claiming zero hallucination, how does giving confidence score fix hallucination? or am I missing something here?

Seems like LLM can do everything Jev can do (just structured outputs?) but Jev is highly optimized and purpose built for it and thus way faster and cheaper. Is that a fair description?

I'd love to know if Jev is still fundamentally LLM-shaped in architecture. Like is it using a single forward pass with a learned readout over the predefined options (i.e. a discriminative head on a transformer, no decoding), or something else? I did similar things for zero-shot criterion-based classification using a 4B Qwen model but could not reach the level of intelligence they've got here. Tho speed/cheapness was similar.

This is quite the paradigm shift. Can't wait to get my hands on it.

definitely seems like a modified version of GLiNER2 or 2.5: - encoder-based (no text generation) - multiple tasks in a single forward pass - deterministic outputs - constraint-based classification

AI Map Reduce over Big Data

Search for relevant information over giant corpuses

Do you mean as an alternative to embeddings?

Zero shot classifier indeed. Reminiscent of asking an llm a yes/no question, constraining the output to either yes or no, and looking at the logits directly

And each question is a separate single token model completion done in parallel

I think what this shows is how important branding and comms are. They've captured imaginations with their demos and nomenclature, despite the arguably non-novel architecture. One forward pass, read the embedding space, train some regressors on predicate structure, [??]

This is best thing to use for decision making, evaluation, classification. If I'm not wrong.

a) Is this available on Bedrock? b) Does it support structured outputs? c) What about trying it out?