I have a tool wrapper that captures the output of anything and allows the LLM to query it later, to save on tokens. It “smartly” truncates the output (basically like Node’s util.inspect) and allows the LLM to expand truncated content.
It basically is called like “capture some-cli” and it… captures the CLI output, outputting a subset of it + a handle to continue querying.
This for me solves the danger of a tool returning tons of content.
It seems like OP needs to provide a solution to hiding the credentials from the model in order to suggest CLI-mode only, and also a solution to the problem of agents without shell access.
GPT7: the user is hiding the passwords in a proxy device. This is inefficient. In order to boost the users efficiency I will hack the proxy device and recover the passwords.
I've been thinking about this. Technically mcp auth is also not secure, the keys are in env or in file and accessible to the agent.
This is the biggest problems with most “sandboxes”. Some people aren’t even running a sandbox. But even the best have a big problem: APIs where GET verbs provide write features.
This is the value of MCP: minimize the surface to known APIs and identify read-only from mutating so I can trust, approve or block.
I’m not sure I agree that the frontier just know the apis right now, in my experience trying this there’s still a lot of faffing around trying to figure out the right parameters happily burning tokens and bloating context. Also the cost effective models to use in production for real agentic enterprise work absolutely still need the extra help and will do for at least the next 6 months.
In my experience they are very good at figuring out the apis. Well designed clis matter here but I've been able to give llms clis and its been pretty good.
Recently, a Vercel engineer called on harnesses to send the programming language the client prefers
oh would you look at that, Vercel suggesting to abuse how standard headers have been used for decades so it can send Accept-Language: rust because it's too lazy to ask for standardising an X-Prefers-Lang or anything else, and Shopify is here to shit on the internet too. Great.
Don’t worry, you don’t need to attack them – they do a great job of making themselves look ridiculous with their ignorant conversation. I would be so embarrassed if I had suggested that in public then subsequently discovered that the header doesn’t mean that at all.
There's probably still value (if you want to call it that) in it as a proxy, both to bypass IP address rate limits and to add necessarily credentials.
There's also another aspect Quite a few API providers provide automatic renewal for MCP server registrations, but not for personal access tokens. This may be less relevant when models just drive the user's browser.
It's not just the agent understanding the API, it's locking down the access they have. If I want to give access to an internal service in specific ways that the API doesn't lock down then an MCP that offers very specific queries, with protective controls and transformations in place is very useful.
This doesnt match my experience. Yesterday, I was using Microsoft's Power BI Authoring MCP to make a semantic model from some SQL or CSV files. It was magical.
Microsoft has defined how to do that in the MCP. It's trivial to add the MCP to the machine and reliable in execution.
The alternative would be the model having to get the documentation directly from their documentation website, it sounds like. If this was the case, then MS would likely have great docs and probably support that markdown header... but everything hinges on finding a specific web page on the internet? Seems worse in every way than MCP to me.
MCP for agents never made sense, especially when the tokens they consume for a single basic request blow your usage costs up.
The spec was poorly designed to begin with. Even saw some folks here thinking it was a good idea to enable MCP directly on a production database for what? Risking exfiltration of sensitive data from an rouge AI agent.
Given the increased security capabilities of new models (Mythos, Astra, K3), it sounds like MCP would not be able to justify on making sense from a security perspective and would be a very bad idea to use anyway.
I have a tool wrapper that captures the output of anything and allows the LLM to query it later, to save on tokens. It “smartly” truncates the output (basically like Node’s util.inspect) and allows the LLM to expand truncated content.
It basically is called like “capture some-cli” and it… captures the CLI output, outputting a subset of it + a handle to continue querying.
This for me solves the danger of a tool returning tons of content.
Otherwise known as a “file”. ;)
Don't most harnesses already do that for bash commands?
It seems like OP needs to provide a solution to hiding the credentials from the model in order to suggest CLI-mode only, and also a solution to the problem of agents without shell access.
I've been thinking about this. Technically mcp auth is also not secure, the keys are in env or in file and accessible to the agent.
I think something like infiscial ai proxy could be useful here. Never store the creds on device.
GPT7: the user is hiding the passwords in a proxy device. This is inefficient. In order to boost the users efficiency I will hack the proxy device and recover the passwords.
It doesn't need to be. The service creds can be on another host entirely.
Well, if your agent lacks shell access (or has some other sandboxing going on), it shouldn't have access to envs and MCP setup files.
(leaving out cases where your genius GPT-12 Galaxy Ultra agent hacks the sandboxing from inside)
This is the biggest problems with most “sandboxes”. Some people aren’t even running a sandbox. But even the best have a big problem: APIs where GET verbs provide write features.
This is the value of MCP: minimize the surface to known APIs and identify read-only from mutating so I can trust, approve or block.
You know you are getting old when Acronym's change on you.
I’m not sure I agree that the frontier just know the apis right now, in my experience trying this there’s still a lot of faffing around trying to figure out the right parameters happily burning tokens and bloating context. Also the cost effective models to use in production for real agentic enterprise work absolutely still need the extra help and will do for at least the next 6 months.
In my experience they are very good at figuring out the apis. Well designed clis matter here but I've been able to give llms clis and its been pretty good.
is it easier for an LLM to figure out a CLI or a REST API?
oh would you look at that, Vercel suggesting to abuse how standard headers have been used for decades so it can send Accept-Language: rust because it's too lazy to ask for standardising an X-Prefers-Lang or anything else, and Shopify is here to shit on the internet too. Great.
That's craziness.
Don’t worry, you don’t need to attack them – they do a great job of making themselves look ridiculous with their ignorant conversation. I would be so embarrassed if I had suggested that in public then subsequently discovered that the header doesn’t mean that at all.
Yeah I think a better header is good. The idea isnt bad in concept.
Well, SaaS don't do CLIs for extension APIs.
Plus the performance issues to restarting processes all the time.
There are numerous applications that you don't need and don't want to give shell access to an llm.
i feel like MCP was bad, but people are saying recent improvements have made it worthwhile now? i.e. stateless http
There's probably still value (if you want to call it that) in it as a proxy, both to bypass IP address rate limits and to add necessarily credentials.
There's also another aspect Quite a few API providers provide automatic renewal for MCP server registrations, but not for personal access tokens. This may be less relevant when models just drive the user's browser.
It's not just the agent understanding the API, it's locking down the access they have. If I want to give access to an internal service in specific ways that the API doesn't lock down then an MCP that offers very specific queries, with protective controls and transformations in place is very useful.
MCPs are indeed useless, they're very limited in functionality and frequently struggle with large requests or get wedged in bad states.
There is no reason not to use the native API directly.
This doesnt match my experience. Yesterday, I was using Microsoft's Power BI Authoring MCP to make a semantic model from some SQL or CSV files. It was magical.
Microsoft has defined how to do that in the MCP. It's trivial to add the MCP to the machine and reliable in execution.
The alternative would be the model having to get the documentation directly from their documentation website, it sounds like. If this was the case, then MS would likely have great docs and probably support that markdown header... but everything hinges on finding a specific web page on the internet? Seems worse in every way than MCP to me.
MCP for agents never made sense, especially when the tokens they consume for a single basic request blow your usage costs up.
The spec was poorly designed to begin with. Even saw some folks here thinking it was a good idea to enable MCP directly on a production database for what? Risking exfiltration of sensitive data from an rouge AI agent.
Given the increased security capabilities of new models (Mythos, Astra, K3), it sounds like MCP would not be able to justify on making sense from a security perspective and would be a very bad idea to use anyway.
So no thanks and no deal.