Anecdotally, current models seem to be decent at general personal finance principles - certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources. But I wouldn't trust them with direct decision making with actual money due to the training lag time on current tax policy, etc.
certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources.
Also, models are now good enough that you can give them chapters from "authoritative" books, and they'll integrate that and come up with better answers even if their "vanilla" answers were average. And they'll tailor stuff to your particular situation. It's funny that the "agentic" stuff is only used in coding mostly, while it can and does work in other fields as well.
As always, you kinda need to check it (at least spot check) but all in all I'd agree it's better than the average stuff you used to find with a quick google search.
I've done this on different topics - I know the answer is in a particular eBook/PDF/document, but for whatever reason it's not trivial to look it up. The model can do it a lot more quickly than I can, and then I can still verify the accuracy.
Some of the more widely spread and tolerated LLM outputs seem to be AI slop replacing Journalism/Blog slop. Places where people complained about quality already, but tolerated it if important enough.
So to name some of the more common ones Translation, Summarization, and Reiteration of a source material.
Humans put spin on things, how much you trust a source might not reflect the source's factual accuracy. It might just mean you liked reading it better from one source than another.
It's basically a fancy context aware ctrl-f to the book.
I use this regularly with RPG manuals. I _know_ the stuff, but don't remember every detail by heart. And just ctrl-f:ing through a Mörk/Pirate Borg -style PDF isn't really productive (they're "artistically" laid out). But I can just ask an AI bot that has the pdf indexed like "how does the medical kit work?" and it'll give me a summary along with the relevant rolls within seconds.
Important to note is that what is being measured here is the ability of the models not of the chat tools themselves, which combine model completions with other tools that the models can call upon. The mainstream labs already know this about models, it's no secret, and in fact training materials from e.g. Anthropic are at pains to point out that users, or analysts designing workflows, have the reponsibility to ensure the correct tools are used and that human verification takes place at appropriate stages depending on the risk/consequences of the task at hand.
Of course a language-completion model with a training cutoff date won't have up-to-date information on tax rules or the ability to carry out correct numerical calculations, but when you combine that with (in Claude terminology) web search and code execution tools invoked by the chat agent, you immediately have much more reliable results.
Single shot or with reasoning enabled? My experience is that reasoning dramatically reduces hallucinations and improves output quality. I don't trust models without it.
These models do pretty well in benchmarks and real world so I'm highly suspicious of this article. Further more, in the original report, the examples of bad answers are from Haiku - at least 7 out of 10. Anyone who knows anything about LLMs know that haiku shouldn't be used for anything pretty much.
There's no reproducible set either. I'm not gonna trust this report.
Most people[1] interacting with chatbots don't have a paid subscription and they do interact with the free-tier LLMs that are Luna and Haiku, so I still think it's relevant.
[1]: not on HN obviously, but IRL, and probably among FT's readership as well.
I don't have the time to review the underlying research and decide which one is more correct. My personal biases make me want to believe the current one. Your personal biases may be pulling you in the other direction. How do we make the conversation more intelligent than that?
Given most financial advisors tend to vend out suboptimal advice and steer customers in favour of products they receive a kickback for, I'm happy to be accepting of an unbiased LLM that's trained on bogleheads.org.
Some people do have complex financial situations. It's not as simple as that.
For example in the UK (and maybe US?) you get tax relief for money you put into your pensions, but there's a limit of £60k/year. Unless you earn a lot (which I do, yeay) when that limit is tapered. Except that you can also use up to 3 years of previously unused allowance. But you have to use this year's first.
Also interest is taxed, but you can put up to £20k/year into an ISA which isn't. And if you still want to avoid some tax you have kids ISA's and even pensions!
Then there are also startup investment schemes that save you some tax. Those seem to be not worth it, but you get the idea - it can be complicated. Especially if you are near one of the many tax/benefit thresholds.
The marginal tax rate in the UK bounces all over the place - it's even technically possible for it to be over 100%!
Much of the testing is on Haiku and Luna, and criticizing the quality of free AI (!). But they do claim Opus 5 with reasoning still failed 39% of their financial questions.
So I downloaded that report which of course doesn't contain the most relevant information (the questions) but it contains some examples of wrong answers.
I fed the first question to Grok (which they claimed they tested as well) and it answered it correctly in detail.
I repeated it with another one - again correct answer. I then selected the question they said Grok specifically answered incorrectly and it again answered it correctly.
I am sticking with my first intuition: people are terrible at testing tools and probably wanted them to answer incorrectly/not fully (the questions are constructed in a way to make it difficult as well). They also have vested interest in the conclusion (they are financial advisory firm) so there is that to consider.
People reading ft will now think chat boxes are bad at answering financial questions while they are pretty good at it. Zero consequences for spreading fake news for Financial Times there but good for financial advisors I guess.
And they hallucinate errors in the millions and struggle with financial data that is in a layout that isn’t in the training data. Ie balance sheet etc.
Been trying to add more AI to my workflow but it just doesn’t work (yet) - not in the same way as vibe coding does
The technical references lookups work though. Looking up regulations etc
Since LLMs can give different answers to the same question, each question was run five
times. That means, each LLM was tested 600 times, and in total over 10,000 questions and
answers were assessed.
All models were given the same zero-shot format. They were not given worked examples,
previous conversations, hints or an opportunity to correct their answers. This is to make it as
similar as possible to a response to a question from consumers.
As for the evaluation itself:
Responses were checked against this (using an LLM-as-a-judge), and was only given a pass if
every element was met; otherwise it was assessed as a fail. This all-pass approach was
intentionally strict, so that the score measures whether an answer is complete enough to
meet the expert legal standard, rather than how many individual points it gets right.
It's just AI slop and it should be taken with a mountain of salt.
Article is just a vague summary of https://www.saturnos.com/report/artificial-authority
Anecdotally, current models seem to be decent at general personal finance principles - certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources. But I wouldn't trust them with direct decision making with actual money due to the training lag time on current tax policy, etc.
Also, models are now good enough that you can give them chapters from "authoritative" books, and they'll integrate that and come up with better answers even if their "vanilla" answers were average. And they'll tailor stuff to your particular situation. It's funny that the "agentic" stuff is only used in coding mostly, while it can and does work in other fields as well.
As always, you kinda need to check it (at least spot check) but all in all I'd agree it's better than the average stuff you used to find with a quick google search.
So if you know a book that has the information you need, you just need to upload it into the model to get the right answer.
Isn't that a bit circular - if you already know the authoritative source, why ask a model?
Because it's faster.
I've done this on different topics - I know the answer is in a particular eBook/PDF/document, but for whatever reason it's not trivial to look it up. The model can do it a lot more quickly than I can, and then I can still verify the accuracy.
Some of the more widely spread and tolerated LLM outputs seem to be AI slop replacing Journalism/Blog slop. Places where people complained about quality already, but tolerated it if important enough.
So to name some of the more common ones Translation, Summarization, and Reiteration of a source material.
Humans put spin on things, how much you trust a source might not reflect the source's factual accuracy. It might just mean you liked reading it better from one source than another.
It's basically a fancy context aware ctrl-f to the book.
I use this regularly with RPG manuals. I _know_ the stuff, but don't remember every detail by heart. And just ctrl-f:ing through a Mörk/Pirate Borg -style PDF isn't really productive (they're "artistically" laid out). But I can just ask an AI bot that has the pdf indexed like "how does the medical kit work?" and it'll give me a summary along with the relevant rolls within seconds.
Important to note is that what is being measured here is the ability of the models not of the chat tools themselves, which combine model completions with other tools that the models can call upon. The mainstream labs already know this about models, it's no secret, and in fact training materials from e.g. Anthropic are at pains to point out that users, or analysts designing workflows, have the reponsibility to ensure the correct tools are used and that human verification takes place at appropriate stages depending on the risk/consequences of the task at hand.
Of course a language-completion model with a training cutoff date won't have up-to-date information on tax rules or the ability to carry out correct numerical calculations, but when you combine that with (in Claude terminology) web search and code execution tools invoked by the chat agent, you immediately have much more reliable results.
I would prefer to use agent-assisted python scripts that chatbot.
Agreed, this works really well for me. Double check the math/python, execute many times without a LLM that can change o
I like how FT makes me accept cookies from their 46 “technology” (advertising) partners before showing me that the article is behind a paywall anyway.
You actually like that? I find it kind of annoying.
No they do not like it, it is a figure of speech to underline how much they do not like it.
that figure of speech is called sarcasm.
very popular on Earth.
Single shot or with reasoning enabled? My experience is that reasoning dramatically reduces hallucinations and improves output quality. I don't trust models without it.
These models do pretty well in benchmarks and real world so I'm highly suspicious of this article. Further more, in the original report, the examples of bad answers are from Haiku - at least 7 out of 10. Anyone who knows anything about LLMs know that haiku shouldn't be used for anything pretty much.
There's no reproducible set either. I'm not gonna trust this report.
Most people[1] interacting with chatbots don't have a paid subscription and they do interact with the free-tier LLMs that are Luna and Haiku, so I still think it's relevant.
[1]: not on HN obviously, but IRL, and probably among FT's readership as well.
A free claude account with no subscription gets you access to sonnet and I believe uses it by default over haiku
Now, compare this to a recent story that seemed to claim the opposite:
https://news.ycombinator.com/item?id=49139102
I don't have the time to review the underlying research and decide which one is more correct. My personal biases make me want to believe the current one. Your personal biases may be pulling you in the other direction. How do we make the conversation more intelligent than that?
Well duh! If it's not using tools to look up the state of the market empirically it's not likely to be accurate financially.
Given most financial advisors tend to vend out suboptimal advice and steer customers in favour of products they receive a kickback for, I'm happy to be accepting of an unbiased LLM that's trained on bogleheads.org.
If that's what you want, I'll save you some tokens:
#!/bin/sh
while read question; do echo "Put it into VFIAX"; done
This is missing a lot of steps like:
- Building an emergency fund
- Budgeting and tracking where your money goes
- Planning and saving for large purchases like cars, homes and life goals
- Optimizing use of tax-advantaged accounts like 401Ks, HSAs, and IRAs
- What to do with ESPPs, RSUs, and options
- How taxes work and how to optimize around them
- Estate planning
Some people do have complex financial situations. It's not as simple as that.
For example in the UK (and maybe US?) you get tax relief for money you put into your pensions, but there's a limit of £60k/year. Unless you earn a lot (which I do, yeay) when that limit is tapered. Except that you can also use up to 3 years of previously unused allowance. But you have to use this year's first.
Also interest is taxed, but you can put up to £20k/year into an ISA which isn't. And if you still want to avoid some tax you have kids ISA's and even pensions!
Then there are also startup investment schemes that save you some tax. Those seem to be not worth it, but you get the idea - it can be complicated. Especially if you are near one of the many tax/benefit thresholds.
The marginal tax rate in the UK bounces all over the place - it's even technically possible for it to be over 100%!
While I also practice Bogle's approach from The Little Book of Common Sense Investing, even with this baseline there are some subtleties.
-VFIAX is currently $707/share. Fidelity's FXAIX does not have to be purchased in increments of a share price, and this fund's expenses are lower.
-There are versions of the S&P 500 for taxable accounts that minimize capital gains.
-Vanguard has a total-market index, VTSAX, that is mentioned in the book.
-Vanguard also has a non-U.S. total market fund, VTIAX, that avoid the current CAPE problems of the U.S. market.
Claude is very familiar with Bogle's approach, likely because the pirated book was part of the training set.
Another summary report here, for those who can't get past the paywall:
https://www.financialreporter.co.uk/ai-models-give-wrong-fin...
Much of the testing is on Haiku and Luna, and criticizing the quality of free AI (!). But they do claim Opus 5 with reasoning still failed 39% of their financial questions.
Tool calling solves this
So I downloaded that report which of course doesn't contain the most relevant information (the questions) but it contains some examples of wrong answers.
I fed the first question to Grok (which they claimed they tested as well) and it answered it correctly in detail.
I repeated it with another one - again correct answer. I then selected the question they said Grok specifically answered incorrectly and it again answered it correctly.
I am sticking with my first intuition: people are terrible at testing tools and probably wanted them to answer incorrectly/not fully (the questions are constructed in a way to make it difficult as well). They also have vested interest in the conclusion (they are financial advisory firm) so there is that to consider.
People reading ft will now think chat boxes are bad at answering financial questions while they are pretty good at it. Zero consequences for spreading fake news for Financial Times there but good for financial advisors I guess.
https://archive.is/WD6Vy
And they hallucinate errors in the millions and struggle with financial data that is in a layout that isn’t in the training data. Ie balance sheet etc.
Been trying to add more AI to my workflow but it just doesn’t work (yet) - not in the same way as vibe coding does
The technical references lookups work though. Looking up regulations etc
Not a surprise, so do financial advisors: garbage in = garbage out.
From the official report:
As for the evaluation itself:
It's just AI slop and it should be taken with a mountain of salt.