I'm not sure why, exactly. But I don't pay any attention to news like this from Google. I don't know if there's some marketing which has me writing them off or if it's something else.
What I do know is that the Gemini integration into sheets is surprisingly incapable of performing basic tasks. This is where I expect Google to really shine. I expected Sheets + Gemini to be magical like Google Photos was. I hardly try anymore besides some basic math questions when I don't feel like inputting the formula myself.
The other thing I know is Google's propensity to sunset products. For many things, it's not a huge deal. And it may not be for this. But, why? When there are alternatives - both open and closed.
On your Sheets + Gemini integration point, I've genuinely tried to give the Gemini integration into Google Docs & Google Sheets a chance. It is so incompetent that it is fully useless to me. I have not gotten a single correct solution each time I tried to use it, even something I consider table stakes. I often write my work reports in Vim in Markdown format, but they need to go to the corporate Google space. No matter how hard I tried, no matter how many prompts I have, it was completely unable to manage the command to "convert the Markdown format markers into native Google Docs markers". And I want to note, this was 2 pages of extremely simple Markdown with no "advanced" patterns, like tables or quotes, I think all I used was heading-marks, bolding, italicizing, and code blocks. This is something I would expect even GPT 3.5 to succeed in, and even more so Luna, but somehow it destroyed the formatting throughout half the document. This leads me to believe that they apply the absolute cheapest model they have there, or they have the model a harness which can barely be considered working.
I found it absurd when I found out that they suddenly made this Gemini integration an additional paid plan recently, there's absolutely no way I can consider that in good faith.
sorry if this isn't it. there is a hidden global setting that defaults to off that lets docs play nice with markdown. it's in file > settings i think. super annoying even if this is no help
Not sure if this is connected, but I've found Gemini pretty hopeless within its own notebooks. I've been doing some job applications, and added cv and docs into a notebook. I then create a new chat to say "here is a job description, help me write a cover letter" or some such.
After about 3 messages in any given chat, a follow-up to "rewrite that with a more friendly tone" will result in a letter for a completely different job from another chat within the notebook.
My understanding is that this one was never really "official", and that the guy who released it wasn't following standard procedures. That's not even the official Google github account.
It depends on where they decide to go with it, I guess.
Kubernetes, Go, Tensorflow, Chromium, gRPC are some examples that obviously went incredibly well.
Yeah I'm actually less hesitant to try out Google open source projects than I am new Google products. I have no idea if this is accurate or just my impression, but I feel like I've been burned by the "killed by Google" meme almost exclusively on their software products, whereas there are plenty of open source efforts from Google that I think of as stable.
In addition to the ones you listed, I'd add the V8 runtime, Jax, Protobuf. Even some of their projects that wound up declining in market share (Angular, Tensorflow--both losing share to projects that wound up at Meta, ironically) are still actively maintained and pushed.
But I'm sure there's also a huge graveyard of open source projects they abandoned that just never hit my radar. Still, at least with their open source stuff, you can fork in the worst case.
Or if it becomes commercially valuable, then you can expect Google to take a third direction (embrace, extend, extinguish) : a la their current approach with android
I'd evaluated both Google's Agent Substrate (that underlies Ax) and their Scion project. I really enjoy how Scion operates with existing tools really well. Ax/Agent Substrate is much more a greenfield independent effort, it's own thing.
I think Scion has so much more mature a disosition: you could write OpenCode plugins that enhance the runner, and use that locally, and use it in Scion. With Ax/Agent Substrate, you are opting in to a pretty huge stack that is just Agent Substrate, that is their runners, their harness, their substrate. I do think their actor model is pretty neat! It's neat having the agent have such primacy! But it feels so much less integrative, is such it's own thing. Scion, to me, is much more interesting an effort, that similarly helps scale out agentic workloads.
you are opting in to a pretty huge stack that is just Agent Substrate, that is their runners, their harness, their substrate
The website makes me think the contrary: It is described as “low opinion” and explicitly mentions that the running tasks don’t even have to be AI agents. Can you explain in what ways you’re more locked in than the website suggests?
Scion at the same time talks much more about concrete agents, giving me the opposite initial impression.
You need a Kubernetes cluster, ko (brew install ko), a container registry your cluster can pull from, and a reachable Agent Substrate Control API (in-cluster default: api.ate-system.svc.cluster.local:443).
make deploy AX_IMAGE_REPO=<your-registry>
This deploys Redis, then builds and deploys the control plane images with ko. Everything lands in the ax-system namespace.
Indeed, there's much less, and that's lower opinion. But you also can't run normal workloads. You have to build for Agent Substrate / Ax.
What's nice about Scion is that it runs existing systems. It runs Claude, it runs Code, it runs Pi, it runs OpenCode. By contrast, "low opinion" means build something new, from scratch, atop this brand new platform.
Note that both of these are designed to work at some scale. Agent Substrate specifically is somewhat coupled to Kubernetes, is my impression, but honestly that's fine with me. Scion can run on Docker, Podman, Apple Container, Kubernetes, or Cloud Run. It's good that we be able to run these relatively quickly, but (especially with LLM assistance) the idea of running some substantial dependencies / services to run these things does not seem like a bad thing. If anything, I'd prefer having some well known services underfoot to these all being recreated afresh.
The reality with releases like this is that I'm 90% sure most Google bigwigs have never heard of it, and it's misleading to label it as "Google's" in the title.
Yes, it was developed by Google employees, that does not imply it has the full backing of Google, or Deepmind, or GCP. Notably, the website doesn't seem to claim this either.
Fair enough, I missed that specific line. The point still stands that I wouldn't expect this to have GDM leadership backing. (If you're planning to use this at all, that matters for how much faith you should have in the product.)
I have no insider knowledge but https://x.com/rakyll is working on it and she is tweeting about it and I got the impression there is a quite a team behind it. It looks like an effort in GCP.
Yes, and one thing to understand about Google is that no AI framework or tool is guaranteed to survive unless it has the explicit backing of Google Deepmind.
"Effort in GCP" is a red flag. (See Gemini CLI, which was shut down in favor of Antigravity CLI.)
And anyone who bothers to just do a side-by-side feature comparison can immediately see how many features antigravity is still missing compared to Gemini CLI even today.
Yeah, anyone should think of this as an experimental side project that may be forgotten in 15min and make careful decisions about using them in production environments.
We have used a few of Google's (smaller) open source projects, and in the last 2-3 years most of them are getting fewer updates if any updates at all. Some became very bad tech debt and we had to spend a lot of time migrating them.
Of course, that is the nature of open source projects (written in the license terms), and there is nothing to complain. But it's important to point out these days Google's open source project are not any more trustworthy than a one man's project in terms of support and maintainability. Personally I would stay away from them as far as possible. Especially if you look at what happened to Android, Gemini CLI etc.
(To be honest, even if it were officially supported by Google, that barely means anything. https://killedbygoogle.com/)
I really don't think any of these SOTA labs are doing agentic engineering correctly. Skills are the universal language of all agent harnesses. If you abstract the taste and prescription out of the skills and into guidance docs, then leave the skills as basically just workflow scaffolding, you can build task-specific workflows that work with any harness like Claude Code, Codex, Antigravity, etc. Technically, you only really need 2 skills, work and review, and with these you can build infinitely complex workflows including self-improving loops. I built this out and have been using it for months. It's been extremely nice. https://github.com/DanMcInerney/orchflows
Overly complex; yaml files, heavy framework. Same mistake as Claude Code's Dynamic Workflows. Why not just use the dehydrated skills as the workflow skeleton and use custom guidance docs to hydrate the skills with taste and preference depending on the domain of the task? Now you can build a library of small workflows that compose into larger workflow, and you can export any workflow as a single skill to be used in other harnesses. For example, I have a code.md. It's really small, just a bit of taste preference. If I'm using it to hydrate orch-work for coding tasks, then maybe I want to create a code.api.md which hydrates for further specificity if the task is about creating APIs. Then when new models come out, I can just delete code.api.md and leave it as code.md for /orch-work to read from within a workflow because newer models won't need as much prescription.
Part of what's happening is this is running on Kubernetes, which is oft described as "Overly complex; yaml files, heavy framework" but has value regardless, as perceived by being an industry standard. All the things you describe are well and good, but do not address how one runs many of them reliably (from an infra stand point)
The OP is not really a workflow manager, it’s a workspace manager that facilitates creating controlled environments where your skills can run. Everything you said is compatible with (and complementary of) the OP project.
With that said, I’ll somewhat disagree with you. I’ve been down the path you’re talking about and while it is incredibly flexible and powerful, it became too difficult to maintain, and too inconsistent between workflow runs, and a pretty hefty waste of tokens to use AI on things that could instead be handled by deterministic scripts. I ended up creating an orchestrator for myself that uses skills as the primary way to tell agents how to execute a step in a workflow, but also directly orchestrates running scripts and managing state in a deterministic way rather than leaving it all up to agents.
You really, really need different skills depending on the model.
If you are using Qwen 27B you need very prescriptive skills.
If you are using Astra you usually want very minimal skills (because it will follow them but be unnecessarily constrained in some contexts)
If you are using Fable then it depends - it will take the skills as general guidelines but ignore them a lot more than Astra does. Sometimes this is good, sometimes not at all.
Right. That's what modular guidance documentation is for. You could have astra.code or qwen.code.api. All reusable in different workflows. Prescription doesn't belong in the skill itself.
Future of platforms is operators in k8s to abstract the developer need to the underlying systems. On local it maps to kvm, on gke it maps to their stuff, on AWS to RDS. It's "interfaces" on a platform level so devs can just ask for a thing.
Overall I agree though, this is a bit of an abuse of that concept.
EDIT: I'm sure op is familiar with this workflow but I'm being overly verbose to clarify what I think they mean and my thoughts.
Google already has gVisor running in Kubernetes as a product (GKE Sandbox), which provides the security guarantees necessary for secure sandboxes (regular k8s isn't great in this respect). They also have pod snapshots running at scale (which run on gVisor), so you can spin up process(es) and snapshot the memory and fs of a pod at a point in time, ship it to a blob in GCS, and then rehydrate those snapshots very quickly (or fork into new instances), which allows for the fast/cheap startup and suspend times and the instant scaling they advertise here. One of these snapshots can be created in one cluster and spun up in another.
Not sure if this is an extension of tech they already have had in their systems, but I've experimenting with it to build my own orchestrator and it's been a pretty neat set of tools and abstractions so far.
I can launch Astra to deploy changes to my homelab via creating Forgejo issues from my phone. That kind of system is pretty hard to set up without kubernetes.
Even if you confine yourself to a dev workstation, having 5 agents concurrently building testing deploying code makes your computer loud and/or hot.
I have been happy with Google's Antigravity harness and Jules so looking forward to playing with this. Thanks for sharing. Simultaneously I am looking to also revisit local offline models.
While I feel like I have a decent understanding of the model landscape I'm feeling a bit lost at which agentic harness to leverage for local models. Hermes, Cline, Aider, Qwen Code, Goose, Pi, OpenCode, something else? I live in the terminal so Desktop UX is a bonus but not a must have.
Can I modify the antigravity settings/program to point to a local model? Where should I spend my energy?
What's going on with Goose? Seems like Block donated it to some consortium; I can't tell if that's a good signal or a bad one. With so many "contenders", if Goose is going into maintenance mode it'd be helpful to know.
I'm stuck on Windows, so oh-my-pi has been really nice. The others I've tried such as kilo do alright but tool calling can mess up a bit.
Only complaint is that connecting the agent harness to my local model took more work getting configured right than I'd like, but that's been true of most harnesses I've tried as well. Most assume you're using a cloud model and local model configuration is a bit of an afterthought.
I tried many of these pi extensions but they have tons of paper cuts. Don't remember which one had which issue but here are some I ran into:
* commands run by me (! prefix) are also sandbox blocked
* agent has no way to _request_ unsandboxed execution (e.g. if `kubectl whatever` is rejected by the sandbox, the model should have the chance to request permission)
* does not understand shell composition patterns (e.g. if `git status` is allowed and `git log` is allowed, then `git status && git log` should be allowed automatically)
* sandbox only supported on mac or linux. not both
All of that can be fixed by yourself. That's certainly the spirit of pi. But if you want strong defaults and batteries included (like omp promises) then that's just annoying.
- instruct model to write a markdown file with a phased plan to implement whatever feature or change I want
- start a new context, instruct model to implement one phase of the file
- review changes manually, then start a new context and have it do the next phase
- repeat as needed
I've never seen omp touch a file outside of the directory I start it up in, and the few times where I've been unhappy with a change git has been there to revert.
This could easily be a case of survivor bias but I've not had an issue with letting it go yolo yet.
Sometimes agents create huge half-minified one-off python/bash scripts to do some data processing. I'd prefer to neither review nor yolo these. Sandbox restricts reads/writes to designated directories, so at least there's no `rm -rf /` in there.
Since local models are largest constrained by context window you want to have a tiny system prompt. I know Hax: https://github.com/OleksandrChekhovskyi/hax was designed with local models in mind, but I haven't used it.
So the agent-substrate checks a _ton_ of boxes. Almost all of the things it offers should be table stakes for everywhere we run not only agents but most software.
(For context I built something very similar to this the past 2 weeks for my homelab, trying to solve many of these problems. This comment is an edited version of an unreleased blog post I wrote last week.)
- Run code in secure microVMs or gVisor. Docker is not good enough. Qemu is not good enough. A secure environment for running untrusted code is the bare minimum. I don't see Firecracker in the repo yet, but that's ok the idea is there.
- Fast resumption. In my homelab, time-to-first-message is around 11-12 seconds. That's half setting up the pod, and half resuming the CLI (e.g. `codex resume ..`). Why resuming? In my homelab agents are commonly blocked waiting for CI or waiting for me to approve an action, in this case I stop their container to keep resource usage low. Then for resumption, you definitely don't want to waste the agents time by giving a new ephemeral disk and forcing them to re-clone and re-build. For microVMs this is not actually straightforward, for example Firecracker only allows block devices, so re-attaching an agents disk workspace requires a custom storage interface
- Zero Trust. Codex CLI permissions for example are extremely broken. "Can I run this 500 line long command? or allow any command starting with first 100 chars always?" More reasonable grants are needed.
I don't understand yet how they will surface Zero Trust notifications. In my homelab it's a Forgejo comment linking to an auth service, and a ntfy.sh iOS notification which opens up the auth service.
I don't get why they to restore the RAM of the agent env. Maybe to fully optimize resumption. Idk, I don't have that much RAM in my homelab, my agents use a ton, testing stuff in Chromium making screenshots for me. I can't keep RAM for 100 workspaces from the past 24 hours in RAM.
MITM gateway is very cool.
I'm curious how they will integrate with microVMs. I just wrote yesterday[1] about how there are NO GOOD OPTIONS for this atm. Kata is decent but the attack surface it introduces makes me uncomfortable.
But anyway, even if this project is abandoned out of the gate by Google, we should be happy, it sets the bar where it should be. I'm excited to learn how they solved these problems differently than I did.
Restoring memory is useful if, when you resume an agent VM, you want the apps to be in the exact state when they were suspended.
But for most things, I find resuming with memory is more trouble than it's worth. If you always resume from memory, you lose the ability to control the state of a VM. It's much easier to define which services should run than to define which active RAM state should be purged
Similar to why "did you try turning it on and off again?" is good for system reliability.
Could someone explain to me what the general workflow is now that people are converging to? I haven't really been catching up with the AI ecosystem but I was looking into agent sandboxes and VM's recently and there's a ton of these startups and tools now. Is giving the agent a temporary scratchbox really that valuable?
I've been still just like, making VM's with proxmox, then putting my agent in the machine and letting it run free (with my dotfiles setup script making dev env pretty much free, though I could also just make a VM snapshot). What's wrong with that? Is that not the scalable solution for enterprise rn?
"Is giving the agent a temporary scratchbox really that valuable?"
Yes, but, wrong layer here. Giving the agent a computer use (a la bash) is what folks are after. A temporary sandbox with lots of control knobs and security bits is how you do that in (as you noted) an enterprise.
Compared to enterprise yours is missing egress control and secrets management, if you make the isolation watertight you cripple the agent's performance, and then the careful game of whack-a-mole begins when you stand up local package mirrors, authentication brokers etc. etc.
I don't think there is really any convergence going on. The agentic ecosystem is continuing to multiply on a daily basis and everyone and their grandma has written a new agent framework--people are stepping over each other to get these new projects out the door.
That said, I think Google's ADK ecosystem and this new AX platform is promising--I would expect Google to maintain this and other tooling around this for years to come.
To the Googlers out there: is Google using this at any capacity for internal projects?
Hey, Dialogflow still appears to be alive and kicking after over a decade. Given how much money they have spent marketing their agent plaform I could see this lasting a while.
And you aren't even being snarky. This is a legit concern whenever I see some new Google initiative that feels like a wart. A nice friendly useful loving wart but its days can be numbered
Google already dumped their original agent framework on the Linux foundation after they realized noone really liked it and the overzealous managers who immediately bought into the hype would go apeshit if they dropped support so fast. Agentic tools have a lifetime measured in months.
Gmail for Your Domain/Google Apps for Your Domain/Google Apps/Google Apps Premier Edition/Google Apps for Business/Google Apps for Work/G Suite/Google Workspace
Google tries a lot of stuff. It doesn't always work out for them, and sometimes they give up on it. I don't think that's a complete loss for the rest of us. We get to see what didn't work in the real world, and if it's really valuable, someone else can pick up the idea and build on that knowledge. I say this as someone who misses Google Reader.
I think the negative impact lots of humans regularly experience from Google's approach to business is far more than the benefit of the very, very specific learning that "this idea doesn't make enough money for Google the way Google did it".
The problem is that you'll end up wanting to run 2 or 3 (or 20, 100, 10,000) agents at once and that gets very hard with a single VM.
There's also an argument that you should be using a separate sandbox for each code operation a LLM performs (or at least each set of related operations). That's even harder to do with conventional VMs.
I just use git worktrees in a single container. Albeit I don’t have more than 3 agents going at a time generally due to my own context switching limitations.
It's a different level of isolation, worktrees help agent work on different code repository in parallel but things get wonky once you consider processes and environments variables
The way I do it is each worktree gets its own .env, with dev credentials to whatever it needs access to, and my coworkers have theirs set up to have an isolated Postgres database per worktree, but I didn’t bother with that in my setup— maybe I will, though. It’s a simple script to create a worktree and properly prepare it.
I'm working on something in the "cloud VMs for agents" space[1], so I have some battle scars and opinions!
IMO, you want the flexibility to create either: (a) permanent devbox VMs, and (b) per-task VMs
Agent sandbox platforms tend to be tuned for the latter, which sometimes involves VMM hackery for fast boot, snapshotting VM filesystem and RAM, etc.
Some workflows are a lot simpler if the multiple agents share a VM. These are workflows where agents must share state. A simple one we have: making related changes in our public OSS repo and our private repo, and then testing the change.
And other times you want to split up the tasks onto isolated VMs so they don't interfere with each other (ie run two dev servers without database or port collisions).
For my example where I modify both the OSS and private repos, I sometimes have one agent coordinate both changes (maybe with subagents), or if the work is modular and separate, I have disjoint agents do the work separately.
That said, most of the time, I only want an agent to work in one repo. I could give it multiple repos at once and instruct it to work in just one, but that risks it forgetting my instructions and it can load more context into the agent's window.
I think my main point was to have flexibility about the topology of VMs, repos, and agents.
From what I've seen, the vast majority of agent sandboxes with funding aren't for developers to use when coding, but for production applications that want to have LLMs do work. It's just a different model - APIs are better than a great terminal experience for a coding harness.
I've been working on https://lullabot.github.io/sandbar/latest/ which works with Proxmox for VMs (and lima for locals or regular linux hosts over ssh). There's a diagram in https://lullabot.github.io/sandbar/latest/why/#recommended-w... with what we're currently recommending. Though, after some feedback, I'm in the process of integrating a colleague's web-based review tool as it turns out many preferred fully reviewing locally instead of using draft PRs.
It's got some opinions in terms of default tools for our team and industry so it may not fit yours. Forgive some of the AI-isms in the docs, I want to get the UX and feature set to a solid place before doing a full review.
The goal is "ticket in JIRA -> solution in production" without human intervention. Right now agentic frameworks are multiplying to bring us closer to that solution like JavaScript frameworks did ~10y ago. We still haven't uncovered the "React" of this space yet, the one that business decides is good enough to standardize on.
We started building that but it quickly turned out to be too narrow. Often we want agents to do task that have no input ticket and often the output is not a code change (Slack bot, incident investigatior, scheduled daily tasks, ...)
Correct and that is why bash coding agents like pi.dev got things completely backwards with their anti-minimalist bloated core tools.
No, giving the agent access to every single command on the system is not minimalist. It is actively detrimental if you want to do more than just attended coding with the agent.
I don't mind that as long as that system gets regularly wiped. If you don't wipe it, you can't reasonably measure the actual output and it's pets Vs cattle all over again, only with agents.
I love how I don't even know how many years after they were created mainstream languages haven't yet figured out that the only sandboxing that works is default deny, like Tcl or Lua.
Especially with autonomous agents, it's the only way to sanity.
I've worked on that as well, and agree with you. You do need the "build this thing" flow, but that just shifts the bottleneck. You also need a whole infrastructure around it, where the jira-to-production pipeline isn't the interesting part.
What do you mean by the "React" of this space? The overall base layer has more or less converged on Kubernetes + MicroVM, which makes total sense. But if you mean how to interact with it as well as how permissions should work, yeah I agree.
Basically everyone has a sandbox of some sort to run agents inside. Everyone has a registry of some sort for tools. Everyone has a way of running agents inside a sandbox and giving it some tools.
Now the stuff people are coming up with is: how do you do authorization in this model? do you need a full sandbox all the time or can it be a workflow? how do you specify an agent is it a prompt or does it have some kind of control flow structure? How do you coordinate among many running agents?
I would say thats where we are now is there’s loads of people all solving the same problems a bit like when CoreOS, Kube etc. were all competing.
Note you don't need a sandbox if you are not doing code execution. There are a lot of applications where inference only is sufficient e.g. web scraping websites that don't change frequently or OCR on scanned documents.
Code execution (usually TS/JS or Python) is useful most when you are dealing with truly open ended problems. It's the opposite of the use cases of most enterprise SaaS.
It is really easy to restrict and contain an AI agent as long as you don't give it access to a terminal. If you only give it tools to read files and access the internet, then it is much easier to just restrict the tools themselves rather than setting up a whole isolated sandbox.
Yes, but what I'm saying is that it is much easier to put a limit on the read_file tool or the http tool rather than sandboxing your whole environment.
You only really need to sandbox when you provide access to tools that are almost impossible to filter correctly, such as a bash tool or a tool for arbitrary code execution.
Do you actually need it, or do you just fear you need it?
What's the actual realistic threat model for median developer or median user here?
By realistic, I mean that leaking your grandma's recipes or your SSN or your million dollar idea to some pastebin is neither likely nor going to meaningfully make things worse for you, or be useful for any malicious actor. Surely this is not what everyone is worried about?
I spent today doing forensics on ten compromised WordPress sites sharing one hosting account.
I used two agents: One with network access to collect the evidence, and one with everything except the model endpoint cut off, which did the analysis.
The second agent's entire input was attacker-authored. So PHP droppers, obfuscated loaders, database rows, filenames, blah blah.
In this case I'm more worried about hostile input attacking the agent, and I need to contain the damage. My sandboxing solution does that by restricting access to the source data, making it read-only. The work dir can only transfer data via patch and apply (like a git workflow), so even my workspace can't be modified until I approve each change. And then restricted network means that any compromise ain't going noplace.
The second agent couldn't even install PHP or contact any CVE site to check if it was looking at a known attack, and that was by design. All it could do is write up a report about what it observed, not make assumptions about what it is. I could then take its (much smaller) clean output and pass that to a third agent with network access.
This is forensic work, so of course not your median dev's bread & butter. But the attack surface is only just starting to be plumbed. Compromising input can turn your agent into their agent, planting things as easily as planting worms was back in the early internet days when people connected without a firewall.
Security by obscurity is just a bet on weights, a belief that the economic motivations for attacking are insufficient. That worked before, but developments in ML calls to account all the debt we’ve accumulated through that practice.
Nothing at all. You'll know when you've outgrown it.
what the general workflow is now that people are converging to?
Graph-based workflows where agents pick up work as it becomes available, structured output, while you manage the work queue and outcomes. Maybe? IDK really, it's all moving quite fast.
https://chatgpt.com/ is a good starting point. Enable voice control and ask the robot to draw a pelican riding a bicycle. Much wow, very revolutionary. More to come.
I think at this point there is also lots of wild enthusiasm and not that much proof mass-agent anything + orchestration is actually financially viable or even useful.
I've been still just like, making VM's with proxmox, then putting my agent in the machine and letting it run free
I outgrew this when I wanted to bring different sets of skills and templates to different machines, wanted to be able to share a small number of credentials, different agents in different machines, different egress rules etc. I wrote https://github.com/pjlsergeant/byre which gives you a TUI and some machinery for doing this easily on top of Docker or Podman.
I have no idea, I just live in my terminal at this point, on linux. Don't use Visual Studio or Jetbrains products anymore at all. I use visual studio code to view the occasional diff and run sql queries. Other than that, zero desktop apps, just terminal & cli tooling. Its great! I have about 20 terminals open at any given time though. I have no idea how some of my colleagues stay productive as they are messing around with all these workflow tools, desktop apps etc.
Yup. I'm also working more and more in a terminal typing English sentences to my computer. It looks like a very smart adventure game UI from the 80s.
A little git log, show, diff almost always in another terminal.
A customer of mine wants to standardize his developers on a Jetbrains IDE for python but I think that he is late by one year. Furthermore he is using the subsidized plans for Claude, not paying for token, so it makes sense to keep using the Claude TUI.
Doing the same more or less, replacing vscode for vim. Usually three/four panes open, one codex running in a container so it doesn't share my host filesystem, one pane for git diffs/git, one for general shelling and one for neovim basically. One of these per project/task/worktree I'm working on, usually 1-2, maximum 3 at the same time.
Not sure what all the other folks are doing, but the industry/ecosystem tends to over-engineer every single thing instead of just working on the thing, while I just want proper code, proper design/architecture, and proper high-quality results.
It's kinda true. I use the terminal approach too, but I wouldn't trust every type of user with this long of a leash. There's a place for instituting systems that share context, manage authentication, and is permissions-aware.
For my workflow I need both, a permanent-ish VM and emphemeral sandboxes. Both have their place and pros and cons.
VMs are better for personal assistant work, GUI clicktesting, investigating bugs in your personal dogfooding dev instance and anything you haven‘t yet made repeatable and fast to set up.
Sandboxes are better when you need resource isolation or security and have a graph of tasks to work through. My agents often starve each other on one VM, so if they don‘t need any of the above it‘s just easier to isolate them.
Everyone is working in this area, including me [0], but either option really isn‘t that convenient to use yet. It‘s a bit of a „isn‘t Dropbox just FTP on a VM“ moment right now.
It‘s a bit of a „isn‘t Dropbox just FTP on a VM“ moment right now.
Since this is not the first mention of Dropbox I've seen in HN threads in the last 48 hours:
Let's not forget that Dropbox was at its best when it was "just" a streamlined ftpd over sshfs or whatever - when it was just "a folder that syncs". That didn't last long, the downfall started with them killing their most useful accidental feature[0], which started them on a path of enshittification[1], which they followed swiftly and diligently into complete irrelevancy they enjoy today.
So if the agentic tooling is now enjoying its "Dropbox moment", I implore people working on these tools, don't overdo it.
--
[0] - The "Public" folder initially supported direct linking, meaning you could publish static web sites by simply putting them in Dropbox/Public/, you could update the files there and changes were immediately "live". Notably, this was the heyday of phpBB and similar discussion boards, back between the rise and subsequent fall of free image hosting - so the ability to put images in your Dropbox/Public/ and hotlink them in a discussion was extremely useful and popular way to use the service.
[1] - They didn't just kill direct links, they replaced them with what I consider to be OG enshittification pattern - captive page that asks you to press a button to download. Yes, same one every "synced drive" service offers now, to enable various functionality that's 99% harmful to the user with the link.
Maybe I knew how they explained the change back then but I forgot about it. If I must take a guess now, maybe their legal office had a word with marketing about the risk of becoming a publisher, with responsibility for what their users shared publicly on the internet.
Perhaps. This was also before CDNs were a thing, so they may have decided they can't afford to be one, as more and more people discovered just how versatile the Public/ folder is.
Either way, this was the peak of Dropbox; after shutting down direct Public/ links, it was still useful for its main job as seamless cross-machine, cross-platform "folder that syncs", but gradually lost market share as OneDrive and Google Drive became more broadly useful (and had the advantage of being first-party on their respective platforms), and then Dropbox the company itself lost focus and tried a bunch of failed pivots in the direction towards cloudification, away from "just syncing files".
End result for end users? We now have zero options for bullshit-free, file-first, seamless "folder that syncs" experience for non-tech users (techies that like fiddling with things have Syncthing). Only cloud-first options remain, and they're full of footguns and enshittified to the core (which becomes apparent the moment you want to share a file outside of the vendor's cloud ecosystem).
Personally I think microVM are the future but nothing wrong with a solid proxmox setup.
Probably we'll converge on a virtualised IO / Storage layer running microVMs beneath for isolation and security. Keep the network and storage layer separate for compatibility running a variety of stuff and a second security boundary.
Not sure about billions, but companies doing evals or RL or training will create really big bursty agent workloads. I think they are the best fit for AX, as opposed to individual dev teams building software, etc.
Task declares the container image and command, compute requests and limits, environment variables [...] Declares listeners the task exposes and an egress allowlist of hosts and ports the sandbox may reach. Use it to restrict an agent to, say, your LLM provider and your Git host.
I'm planning to buy a whole linux mini-PC to run my agents/code servers for more isolation. Codex/Claude Code let you run prompts on code over ssh (same with most IDEs) even on the desktop apps.
I wonder if that's going to be the new standard practice. You get a work laptop and an isolated agent box.
Running access control and network whitelists is always a maintenance challenge and it's easy to make mistakes.
I think it will be, but I don't think you need a standalone machine! If you run things inside a VM, you can get safety and control over access and networks
A standalone machine is nice if you need more compute resources or if you want an always-on machine you can connect to from your laptop, phone, etc.
It doesn't look like Google's AX is quite the plug-and-play fit for running agents on a computer you own, since it requires setting up a K8S cluster, etc.
I think what's needed is something like a zero-setup combo of Tailscale and Firecracker
I'm trying to work towards that with my startup (https://github.com/gofixpoint/amika) but the bring-your-own-computer part doesn't work quite yet.
You might be right, it likely needs a full proper PC setup with the test suite stuff. I was looking at this vendor, https://www.gmktec.com/collections/all there's this whole AI mini-pc market but they aren't quite a full dev machine replacement
I’m running ~5 agents at a time very comfortably on a $280 mini PC with 8GB. They’re all in Docker containers, a couple have sidecar VMs they can own and run. Not having any issue with load.
Interesting, we had developed a very similar framework for our internal agents: https://github.com/apoxy-dev/clrk For us main use-case was intercepting all network I/O including LLM providers, HTTP, and random TCP/UDP calls
Nobody has a use for this, and anybody who can look at this website and work out what it's for is kidding themselves. Even the demo gif playing just has them pausing a task and resuming the task.
If there's a criticism here, it's that they don't mention k8s right up front. It's completely opaque what this tool is. "Orchestration" can mean literally anything.
The web page presents ax as a typical developer tool, but it's actually not for developers.
Kubernetes is the last thing I wanted to see recreated for agents. It’s like Multics of cloud, now for agents. Complexity for the sake of it, powered by your favourite YAML slop bowl.
Interesting approach. How does it compare to LangGraph for multi-step agent workflows? The orchestration layer always seems to be the hardest part to get right in practice.
Drawing on agentic runtime research from Google DeepMind alongside deep experience in large-scale isolation, resumption, and scheduling, AX is being built as an open, declarative control plane purpose-built...
The project seems like an open-source initiative born out of the experience of some Googlers but not being used at Google. So, the title appears a bit misleading - people will be misled.
The best part of all of this is the most if not all people have no idea what the hell is going on when every day a new paradigm/tooling/harness emerges. It's hard to keep up. On the plus side, it's a great equalizer.
Offload the work of an LLM agent to a box in the cloud, so it doesn’t run on your laptop. This has security benefits (no access to your laptop’s files) and you can scale it up (run a lot of agents at the same time).
Then put a “sandbox” around these agents, that word has many meanings. In this case they fence the network traffic, so likely some kind of allowlist for network requests so that the agent doesn’t exfil crap to random websites. They also limit the resource limits of the sandbox, so that is beneficial to the cost of running these agents.
Everything is always better with more YAML, are you perhaps new to this industry?
Next we also need an instruction style guide and CoC. It's important to treat your agents with respect. I almost forgot, the YAML template meta-language to YAML the YAML. Then we will need a foundation employing 12 FTEs to maintain it all and of course to run the certification process. You are certified, right? Statistics show a 10x increased chance of an agent going rogue and hacking competitors if it has been mistreated or been run in an unvalidated sandbox. It goes without saying the sandbox certification process is separate and must be repeated yearly by a trusted third party auditing company.
must look cool for people who want to solve every problem with k8s
it starts with interesting misnomers like "Task" which is not a work item but a sandbox.
"billions of tasks" is a "solution" to problem nobody has (maybe some RL labs? but they solve it other way and with orders of magnitude better optimizations).
freezes design too early – unless they'll actually focus on developing it and make tons of breaking changes it looks shit.
shared state in the same workspace, identity, authority, etc – stuff like that needs to be solved
The real difficult in arranging agents is not to run them, but to identify the state change - to judge whether an agent stops to wait for you, or is stuck, or finished, and whether the two should be handled automatically or someone should be found. You have made this judgment for 12 agents, and you need to know how unreliable it is.
K8s jobs don’t run in a secure runtime. K8s jobs don’t give you dynamic zero trust permissions scopes. Restoring a harness in 500ms is really fast, much faster than naively creating a new job downloading session and ‘codex resume’ etc.
Essentially there is no out of the box solution about orchestrating agents and increasing LLMs sandboxing. That's why everyone and their grandmother are re-inventing the wheel.
At the same time, it's an incredibly complicated problem, with a variable perimeter (OS support, sandboxing primitives support).
I am quite happy about my own solution (because it supports my use case!) but I hope something with a decent dev UX will appear one day. AX definitely is NOT.
IIUC agent-sandbox and agent-substrate (the one ax builds upon) are similar. Agent-sandbox is more k8s-native, while agent-substrate is less so.
Personally I think that this kind of workload is better off not being tied too much into kubernetes. I've worked with crossplane and other controller who put a lot of load on the k8s-apiserver and etcd and can easily slow the whole machinery down / grind them to a halt.
We have built similar abstractions directly on top of Kubernetes [1]
I was looking at this project a couple of months ago, and I did not understand why not use Kubernetes instead of rebuilding the abstractions. The reason is that Kubernetes already provides other abstractions to run services and batch job, gang scheduling, gpu and other accelerators enabled workflow.
We want to make dealing with agentic infrastructure easier so you can focus on your work. AX is designed with an uncompromising focus on ergonomics, rapid iteration, and joyful workflows for both application developers and AI researchers.
On the the other hand, the readme quickstart section says
You need a Kubernetes cluster, ko (brew install ko), a container registry your cluster can pull from, and a reachable Agent Substrate Control API (in-cluster default: api.ate-system.svc.cluster.local:443).
Call me old-fashioned but I don't find this "easier". Maybe it's easier in the same way that Kubernetes itself is easier than managing VMs and container deployments at massive scale without such a tool. But there's a vast chasm between what this tool is being sold as and what it actually is.
As with everything in software engineering, any average dev can fire up a few containers with Docker, but running anything in a real environment takes a whole team of people who actually understand the platforms involved in order to deliver a fully functional service, hopefully via properly designed code (TF) for ease of reusability. Most devs simply don't care to think about that part, and then throw it over the fence for "someone else" to deal with. Just as long as they can say "DONE!" (I created a thing!), that's all that matters.
Google operates at such a scale with a wide surface area of serious production considerations that even “ergonomic” solutions internally feel extremely heavyweight externally.
If you are approaching anything looking like an enterprise environment, you are likely to have Kubernetes nearby. ko is a weird requirement. ax requires agent substrate, so it makes sense that it's needed. Agent Substrate also has experimental support in kagent, so it makes sense that it is growing.
I don't really like the oversubscription of agent pods though, as you can no longer trust the k8s pod identity as being from a singular workload. Haven't seen a solution to this for ax yet and it is a barrier to adoption for us.
One thing I’d say, is that I find it progressively more interesting/fun to rollout your “everything”. I mean… hello security, but by the time I’ve read somebody’s documentation I’ve already halfway done making thing exactly how I want it.
I'm not sure why, exactly. But I don't pay any attention to news like this from Google. I don't know if there's some marketing which has me writing them off or if it's something else.
What I do know is that the Gemini integration into sheets is surprisingly incapable of performing basic tasks. This is where I expect Google to really shine. I expected Sheets + Gemini to be magical like Google Photos was. I hardly try anymore besides some basic math questions when I don't feel like inputting the formula myself.
The other thing I know is Google's propensity to sunset products. For many things, it's not a huge deal. And it may not be for this. But, why? When there are alternatives - both open and closed.
This is an Apache 2.0 open source project
On your Sheets + Gemini integration point, I've genuinely tried to give the Gemini integration into Google Docs & Google Sheets a chance. It is so incompetent that it is fully useless to me. I have not gotten a single correct solution each time I tried to use it, even something I consider table stakes. I often write my work reports in Vim in Markdown format, but they need to go to the corporate Google space. No matter how hard I tried, no matter how many prompts I have, it was completely unable to manage the command to "convert the Markdown format markers into native Google Docs markers". And I want to note, this was 2 pages of extremely simple Markdown with no "advanced" patterns, like tables or quotes, I think all I used was heading-marks, bolding, italicizing, and code blocks. This is something I would expect even GPT 3.5 to succeed in, and even more so Luna, but somehow it destroyed the formatting throughout half the document. This leads me to believe that they apply the absolute cheapest model they have there, or they have the model a harness which can barely be considered working. I found it absurd when I found out that they suddenly made this Gemini integration an additional paid plan recently, there's absolutely no way I can consider that in good faith.
sorry if this isn't it. there is a hidden global setting that defaults to off that lets docs play nice with markdown. it's in file > settings i think. super annoying even if this is no help
Making Kubernetes a centre of everything. This one, they won’t sunset, because it helps selling GCP services.
Not sure if this is connected, but I've found Gemini pretty hopeless within its own notebooks. I've been doing some job applications, and added cv and docs into a notebook. I then create a new chat to say "here is a job description, help me write a cover letter" or some such.
After about 3 messages in any given chat, a follow-up to "rewrite that with a more friendly tone" will result in a letter for a completely different job from another chat within the notebook.
This might be a blessing in disguise though, as the thing you've tasked it to do is something you should not offload to an LLM.
I told this exact thing to my gf few days ago and somehow she was mad at me for it
k8sification of AI was always inevitable, if only as a form of salary justification.
Ah yes, k8s8n.
You know you're on the right path when Kate Satan turns up.
Question: What is Google's track record for where their open source releases end up over time?
Genuinely not knowledgeable here
Well, they're not above forking their own project to patch security holes and never upstreaming the fixes.
https://grapheneos.social/@GrapheneOS/117282080803799576
Maintain it briefly then slowly let it die.
more like create a big launch for promo, then maintain it briefly then slowly let it die
That's not fair. Sometimes they kill it off quickly.
https://killedbygoogle.com/
A recent example was the Google workspace CLI, which was maintained for just a month: https://github.com/googleworkspace/cli
My understanding is that this one was never really "official", and that the guy who released it wasn't following standard procedures. That's not even the official Google github account.
yet they "maintain" the pinned state on the org page
https://github.com/googleworkspace
It depends on where they decide to go with it, I guess. Kubernetes, Go, Tensorflow, Chromium, gRPC are some examples that obviously went incredibly well.
Yeah I'm actually less hesitant to try out Google open source projects than I am new Google products. I have no idea if this is accurate or just my impression, but I feel like I've been burned by the "killed by Google" meme almost exclusively on their software products, whereas there are plenty of open source efforts from Google that I think of as stable.
In addition to the ones you listed, I'd add the V8 runtime, Jax, Protobuf. Even some of their projects that wound up declining in market share (Angular, Tensorflow--both losing share to projects that wound up at Meta, ironically) are still actively maintained and pushed.
But I'm sure there's also a huge graveyard of open source projects they abandoned that just never hit my radar. Still, at least with their open source stuff, you can fork in the worst case.
Or if it becomes commercially valuable, then you can expect Google to take a third direction (embrace, extend, extinguish) : a la their current approach with android
Don't forget `cgroups`, eh? That was a huge Linux contribution.
or Android Open Source Project?
I can understand why it was chosen, but I'm not a fan of writing a bunch of yaml.
I assume they expect agents will be writing most of those
I'd evaluated both Google's Agent Substrate (that underlies Ax) and their Scion project. I really enjoy how Scion operates with existing tools really well. Ax/Agent Substrate is much more a greenfield independent effort, it's own thing.
I think Scion has so much more mature a disosition: you could write OpenCode plugins that enhance the runner, and use that locally, and use it in Scion. With Ax/Agent Substrate, you are opting in to a pretty huge stack that is just Agent Substrate, that is their runners, their harness, their substrate. I do think their actor model is pretty neat! It's neat having the agent have such primacy! But it feels so much less integrative, is such it's own thing. Scion, to me, is much more interesting an effort, that similarly helps scale out agentic workloads.
https://github.com/googlecloudplatform/scion
The website makes me think the contrary: It is described as “low opinion” and explicitly mentions that the running tasks don’t even have to be AI agents. Can you explain in what ways you’re more locked in than the website suggests?
Scion at the same time talks much more about concrete agents, giving me the opposite initial impression.
Not GP, but you start with Kubernetes…
Indeed, there's much less, and that's lower opinion. But you also can't run normal workloads. You have to build for Agent Substrate / Ax.
What's nice about Scion is that it runs existing systems. It runs Claude, it runs Code, it runs Pi, it runs OpenCode. By contrast, "low opinion" means build something new, from scratch, atop this brand new platform.
Note that both of these are designed to work at some scale. Agent Substrate specifically is somewhat coupled to Kubernetes, is my impression, but honestly that's fine with me. Scion can run on Docker, Podman, Apple Container, Kubernetes, or Cloud Run. It's good that we be able to run these relatively quickly, but (especially with LLM assistance) the idea of running some substantial dependencies / services to run these things does not seem like a bad thing. If anything, I'd prefer having some well known services underfoot to these all being recreated afresh.
The reality with releases like this is that I'm 90% sure most Google bigwigs have never heard of it, and it's misleading to label it as "Google's" in the title.
Yes, it was developed by Google employees, that does not imply it has the full backing of Google, or Deepmind, or GCP. Notably, the website doesn't seem to claim this either.
This /looks/ at least more official. Most unofficial Google projects have a disclaimer in the repo.
It is on Google's github https://github.com/google/ax and the title comes from there.
While this one seems like an official Google project, some other projects are not, even if it is on github.com/google
A random example E.g https://github.com/google/filament#disclaimer
This is not an officially supported Google product.
Does Filament claim that it's Google's in the repo description? It's a valid title.
Fair enough, I missed that specific line. The point still stands that I wouldn't expect this to have GDM leadership backing. (If you're planning to use this at all, that matters for how much faith you should have in the product.)
it's built on top of https://github.com/agent-substrate/substrate which is not on Google github.
Agent Substrate is being moved to CNCF: https://github.com/cncf/sandbox/issues/523
the repo description https://github.com/google/ax is "Google's open agentic orchestrator"
for comparison/contrast, another very related Google project (one employee) on one of their github orgs that comes with the following disclaimer
https://github.com/GoogleCloudPlatform/scion
The project that ax builds on top of does have that disclaimer, and sits on a non-google organisation https://github.com/agent-substrate/substrate
I have no insider knowledge but https://x.com/rakyll is working on it and she is tweeting about it and I got the impression there is a quite a team behind it. It looks like an effort in GCP.
Yes, and one thing to understand about Google is that no AI framework or tool is guaranteed to survive unless it has the explicit backing of Google Deepmind.
"Effort in GCP" is a red flag. (See Gemini CLI, which was shut down in favor of Antigravity CLI.)
Antigravity CLI is far, far superior to Gemini CLI.
Plant many flowers, keep the ones that bloom and stop watering the ones that don't.
I agree on both accounts fwiw.
It is not, it is intentionally not-interoperable walled garden, unlike Gemini CLI.
Is it superior in terms of license?
And anyone who bothers to just do a side-by-side feature comparison can immediately see how many features antigravity is still missing compared to Gemini CLI even today.
Google has around 200,000 employees. They probably haven't heard of most things Google releases.
It's got a formal announcement on the blog. 4 months ago.
https://cloud.google.com/blog/products/ai-machine-learning/a...
Yeah, anyone should think of this as an experimental side project that may be forgotten in 15min and make careful decisions about using them in production environments.
We have used a few of Google's (smaller) open source projects, and in the last 2-3 years most of them are getting fewer updates if any updates at all. Some became very bad tech debt and we had to spend a lot of time migrating them.
Of course, that is the nature of open source projects (written in the license terms), and there is nothing to complain. But it's important to point out these days Google's open source project are not any more trustworthy than a one man's project in terms of support and maintainability. Personally I would stay away from them as far as possible. Especially if you look at what happened to Android, Gemini CLI etc.
(To be honest, even if it were officially supported by Google, that barely means anything. https://killedbygoogle.com/)
Everyone and their mother are vibe coding their own solutions like this, all the time.
This is bound to cause some confusion with the other tool called Ax for agentic development: https://axllm.dev/ (which is DSPy for other languages)
this is nice, basically virtual threads for kubernetes.
I don't see a meaningful difference to the 100s of other 'agentic frameworks' that promise to be the one to all solution for all your troubles.
Would be about time we get benchmarks for these ... so these can also be gamified just like with the LLMs.
what are your points of comparison ?
I really don't think any of these SOTA labs are doing agentic engineering correctly. Skills are the universal language of all agent harnesses. If you abstract the taste and prescription out of the skills and into guidance docs, then leave the skills as basically just workflow scaffolding, you can build task-specific workflows that work with any harness like Claude Code, Codex, Antigravity, etc. Technically, you only really need 2 skills, work and review, and with these you can build infinitely complex workflows including self-improving loops. I built this out and have been using it for months. It's been extremely nice. https://github.com/DanMcInerney/orchflows
How does your criticism relate to the specifics of what OP posted? https://github.com/google/ax/blob/main/docs/concepts.md#work... This says it has skill registries.
Overly complex; yaml files, heavy framework. Same mistake as Claude Code's Dynamic Workflows. Why not just use the dehydrated skills as the workflow skeleton and use custom guidance docs to hydrate the skills with taste and preference depending on the domain of the task? Now you can build a library of small workflows that compose into larger workflow, and you can export any workflow as a single skill to be used in other harnesses. For example, I have a code.md. It's really small, just a bit of taste preference. If I'm using it to hydrate orch-work for coding tasks, then maybe I want to create a code.api.md which hydrates for further specificity if the task is about creating APIs. Then when new models come out, I can just delete code.api.md and leave it as code.md for /orch-work to read from within a workflow because newer models won't need as much prescription.
Part of what's happening is this is running on Kubernetes, which is oft described as "Overly complex; yaml files, heavy framework" but has value regardless, as perceived by being an industry standard. All the things you describe are well and good, but do not address how one runs many of them reliably (from an infra stand point)
The OP is not really a workflow manager, it’s a workspace manager that facilitates creating controlled environments where your skills can run. Everything you said is compatible with (and complementary of) the OP project.
With that said, I’ll somewhat disagree with you. I’ve been down the path you’re talking about and while it is incredibly flexible and powerful, it became too difficult to maintain, and too inconsistent between workflow runs, and a pretty hefty waste of tokens to use AI on things that could instead be handled by deterministic scripts. I ended up creating an orchestrator for myself that uses skills as the primary way to tell agents how to execute a step in a workflow, but also directly orchestrates running scripts and managing state in a deterministic way rather than leaving it all up to agents.
You really, really need different skills depending on the model.
If you are using Qwen 27B you need very prescriptive skills.
If you are using Astra you usually want very minimal skills (because it will follow them but be unnecessarily constrained in some contexts)
If you are using Fable then it depends - it will take the skills as general guidelines but ignore them a lot more than Astra does. Sometimes this is good, sometimes not at all.
Right. That's what modular guidance documentation is for. You could have astra.code or qwen.code.api. All reusable in different workflows. Prescription doesn't belong in the skill itself.
I just have a tmux session acting as the orchestrator, and I tell it to report back and direct the other agents working in separate tmux sessions.
Dropbox comment
Why kubernetes? Seems like an overload
Future of platforms is operators in k8s to abstract the developer need to the underlying systems. On local it maps to kvm, on gke it maps to their stuff, on AWS to RDS. It's "interfaces" on a platform level so devs can just ask for a thing.
Overall I agree though, this is a bit of an abuse of that concept.
EDIT: I'm sure op is familiar with this workflow but I'm being overly verbose to clarify what I think they mean and my thoughts.
Google already has gVisor running in Kubernetes as a product (GKE Sandbox), which provides the security guarantees necessary for secure sandboxes (regular k8s isn't great in this respect). They also have pod snapshots running at scale (which run on gVisor), so you can spin up process(es) and snapshot the memory and fs of a pod at a point in time, ship it to a blob in GCS, and then rehydrate those snapshots very quickly (or fork into new instances), which allows for the fast/cheap startup and suspend times and the instant scaling they advertise here. One of these snapshots can be created in one cluster and spun up in another.
Not sure if this is an extension of tech they already have had in their systems, but I've experimenting with it to build my own orchestrator and it's been a pretty neat set of tools and abstractions so far.
Agree. Probably an unpopular opinion, but I strongly dislike YAML.
EDIT: if I HAD to use YAML, I'd prefer KYAML: https://dev.to/mechcloud_academy/goodbye-yaml-hell-meet-kyam...
I can launch Astra to deploy changes to my homelab via creating Forgejo issues from my phone. That kind of system is pretty hard to set up without kubernetes.
Even if you confine yourself to a dev workstation, having 5 agents concurrently building testing deploying code makes your computer loud and/or hot.
I have been happy with Google's Antigravity harness and Jules so looking forward to playing with this. Thanks for sharing. Simultaneously I am looking to also revisit local offline models.
While I feel like I have a decent understanding of the model landscape I'm feeling a bit lost at which agentic harness to leverage for local models. Hermes, Cline, Aider, Qwen Code, Goose, Pi, OpenCode, something else? I live in the terminal so Desktop UX is a bonus but not a must have.
Can I modify the antigravity settings/program to point to a local model? Where should I spend my energy?
goose has now native support for local models
Its docs have zero support
What's going on with Goose? Seems like Block donated it to some consortium; I can't tell if that's a good signal or a bad one. With so many "contenders", if Goose is going into maintenance mode it'd be helpful to know.
Did something change with it for depth of integration? Goose has been able to use local models via OpenAI compatible endpoints for at least a year
I'm stuck on Windows, so oh-my-pi has been really nice. The others I've tried such as kilo do alright but tool calling can mess up a bit.
Only complaint is that connecting the agent harness to my local model took more work getting configured right than I'd like, but that's been true of most harnesses I've tried as well. Most assume you're using a cloud model and local model configuration is a bit of an afterthought.
oh-my-pi has pretty poor permission system in my experience. Either yolo or deny/approve everything. No classifier, no sandbox.
Everything you want to add can be made as an extension, and pi has many ready extension to be added. For example, Sandbox: `https://github.com/earendil-works/gondolin` The sandbox extension is at https://github.com/earendil-works/pi/tree/main/packages/codi... And the extension at `https://github.com/earendil-works/pi/tree/main/packages/codi...` Which can also be found under `/@earendil-works/pi-coding-agent/examples/extensions/gondolin` at the npm modules
In case of omp, not sure if it's already at the node module package but you can just grab it from the links I shared and set it up.
I tried many of these pi extensions but they have tons of paper cuts. Don't remember which one had which issue but here are some I ran into:
* commands run by me (! prefix) are also sandbox blocked
* agent has no way to _request_ unsandboxed execution (e.g. if `kubectl whatever` is rejected by the sandbox, the model should have the chance to request permission)
* does not understand shell composition patterns (e.g. if `git status` is allowed and `git log` is allowed, then `git status && git log` should be allowed automatically)
* sandbox only supported on mac or linux. not both
All of that can be fixed by yourself. That's certainly the spirit of pi. But if you want strong defaults and batteries included (like omp promises) then that's just annoying.
My workflow is generally like so:
- instruct model to write a markdown file with a phased plan to implement whatever feature or change I want
- start a new context, instruct model to implement one phase of the file
- review changes manually, then start a new context and have it do the next phase
- repeat as needed
I've never seen omp touch a file outside of the directory I start it up in, and the few times where I've been unhappy with a change git has been there to revert.
This could easily be a case of survivor bias but I've not had an issue with letting it go yolo yet.
Sometimes agents create huge half-minified one-off python/bash scripts to do some data processing. I'd prefer to neither review nor yolo these. Sandbox restricts reads/writes to designated directories, so at least there's no `rm -rf /` in there.
Since local models are largest constrained by context window you want to have a tiny system prompt. I know Hax: https://github.com/OleksandrChekhovskyi/hax was designed with local models in mind, but I haven't used it.
i built something with a similar philosophy https://github.com/khimaros/hrns
pi.
Always Pi.
What are you using Jules for? I want to like it but it fails too often. If I could use Gemini 3.8 I would be happy but 3.6 rarely succeeds.
Does agy has auto mode?
Yes
This is news to me, do you have a link to docs for the configuration option?
Deepseek Harness is great, at least with deepseek.
Give https://juggler.studio a shot if you want a nice desktop UX with all the extensibility of things like Pi
I'm not a fan of open-core apps.
Hmm, that phrasing is misleading - it's an open-source project: the core is AGPL, the extensions are permissively licensed
So the agent-substrate checks a _ton_ of boxes. Almost all of the things it offers should be table stakes for everywhere we run not only agents but most software.
https://github.com/agent-substrate/substrate
(For context I built something very similar to this the past 2 weeks for my homelab, trying to solve many of these problems. This comment is an edited version of an unreleased blog post I wrote last week.)
- Run code in secure microVMs or gVisor. Docker is not good enough. Qemu is not good enough. A secure environment for running untrusted code is the bare minimum. I don't see Firecracker in the repo yet, but that's ok the idea is there.
- Fast resumption. In my homelab, time-to-first-message is around 11-12 seconds. That's half setting up the pod, and half resuming the CLI (e.g. `codex resume ..`). Why resuming? In my homelab agents are commonly blocked waiting for CI or waiting for me to approve an action, in this case I stop their container to keep resource usage low. Then for resumption, you definitely don't want to waste the agents time by giving a new ephemeral disk and forcing them to re-clone and re-build. For microVMs this is not actually straightforward, for example Firecracker only allows block devices, so re-attaching an agents disk workspace requires a custom storage interface
- Zero Trust. Codex CLI permissions for example are extremely broken. "Can I run this 500 line long command? or allow any command starting with first 100 chars always?" More reasonable grants are needed.
I don't understand yet how they will surface Zero Trust notifications. In my homelab it's a Forgejo comment linking to an auth service, and a ntfy.sh iOS notification which opens up the auth service.
I don't get why they to restore the RAM of the agent env. Maybe to fully optimize resumption. Idk, I don't have that much RAM in my homelab, my agents use a ton, testing stuff in Chromium making screenshots for me. I can't keep RAM for 100 workspaces from the past 24 hours in RAM.
MITM gateway is very cool.
I'm curious how they will integrate with microVMs. I just wrote yesterday[1] about how there are NO GOOD OPTIONS for this atm. Kata is decent but the attack surface it introduces makes me uncomfortable.
[1]: https://srcreigh.ca/posts/auditable-kata/
But anyway, even if this project is abandoned out of the gate by Google, we should be happy, it sets the bar where it should be. I'm excited to learn how they solved these problems differently than I did.
For microVM, smolvm is quite impressive.
For further isolation, I like to use nono inside a smolvm instance.
For microvms, take a look at https://smolmachines.com
I'm not interested in a VM which supports mounting host filesystems in untrusted Kubernetes pods.
smolvm integrates with k8s (https://smolmachines.com/docs/guides/kubernetes-in-a-microvm...) and can mount s3 buckets (https://smolmachines.com/docs/local/machine-lifecycle-cli-re...).
I wouldn’t dismiss smolvm so fast. It brings together many ideas that make the whole very interesting.
It doesn’t provide isolation. It is not even part of the conversation.
I thought the whole point of a microVM is that it does provide isolation ?
What do you mean by "It doesn’t provide isolation" ? How so ?
This post explains how a GPT agent broke out of qemu VM. It could not break out of firecracker.
https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyb...
Why? Firecracker mounts very few host systems into the VM, exposing minimal host code to malicious guests. Qemu and smolvm expose much more.
So yeah, smolvm is more like a docker or qemu alternative, definitely useful but NOT relevant to the discussion of sandboxing malicious code
Restoring memory is useful if, when you resume an agent VM, you want the apps to be in the exact state when they were suspended.
But for most things, I find resuming with memory is more trouble than it's worth. If you always resume from memory, you lose the ability to control the state of a VM. It's much easier to define which services should run than to define which active RAM state should be purged
Similar to why "did you try turning it on and off again?" is good for system reliability.
what about msb? https://github.com/superradcompany/microsandbox
Supports direct filesystem access and who knows what else. It is unsuitable for running untrusted code on Linux.
I'm keeping an eye on another Google Cloud orchestrator
https://googlecloudplatform.github.io/scion/overview/
Scion wraps the harnesses (9x) we all use every day and is closer to OpenClaw on Kubernetes
Can someone clarify the use case for this? What's the benefit over this: https://openai.com/index/introducing-the-agents-api/
you can run it yourself, it's open source, you can use any harness (req. custom image), you can use any token vendor (config)
A DAG?! Holy innovation, Batman!
How does it compare to kagent (https://kagent.dev/)?
As usual, Google makes it "googley" by building an incompatible monolith with the kitchen sink included.
Could someone explain to me what the general workflow is now that people are converging to? I haven't really been catching up with the AI ecosystem but I was looking into agent sandboxes and VM's recently and there's a ton of these startups and tools now. Is giving the agent a temporary scratchbox really that valuable?
I've been still just like, making VM's with proxmox, then putting my agent in the machine and letting it run free (with my dotfiles setup script making dev env pretty much free, though I could also just make a VM snapshot). What's wrong with that? Is that not the scalable solution for enterprise rn?
"Is giving the agent a temporary scratchbox really that valuable?"
Yes, but, wrong layer here. Giving the agent a computer use (a la bash) is what folks are after. A temporary sandbox with lots of control knobs and security bits is how you do that in (as you noted) an enterprise.
Compared to enterprise yours is missing egress control and secrets management, if you make the isolation watertight you cripple the agent's performance, and then the careful game of whack-a-mole begins when you stand up local package mirrors, authentication brokers etc. etc.
I was expecting whack-a-mole as well when designing my sandbox software, but mostly it didn't happen.
As a test, I built a sandbox with only the host-side filtering proxy allowed for networking. 99% of traffic was HTTP. No QUIC at all.
npm, pip, apt, go, curl and git-over-HTTPS all worked on the standard proxy environment variables alone. No mirrors or other coaxing needed.
DNS is disallowed through the chokepoint, but that's no problem because the proxy resolves host-side anyway.
I don't think there is really any convergence going on. The agentic ecosystem is continuing to multiply on a daily basis and everyone and their grandma has written a new agent framework--people are stepping over each other to get these new projects out the door.
That said, I think Google's ADK ecosystem and this new AX platform is promising--I would expect Google to maintain this and other tooling around this for years to come.
To the Googlers out there: is Google using this at any capacity for internal projects?
The same Google that pulls plugs on a whim?
at least this one is pre-named for past tense when that day arrives /s
Hey, Dialogflow still appears to be alive and kicking after over a decade. Given how much money they have spent marketing their agent plaform I could see this lasting a while.
And you aren't even being snarky. This is a legit concern whenever I see some new Google initiative that feels like a wart. A nice friendly useful loving wart but its days can be numbered
Google already dumped their original agent framework on the Linux foundation after they realized noone really liked it and the overzealous managers who immediately bought into the hype would go apeshit if they dropped support so fast. Agentic tools have a lifetime measured in months.
Like Kubernetes?
btw its the same google that has already killed its "gemini cli" and re-introduced it in the form of "antigravity cli"
Google+ would like a word
Google Wave is waving
Google Buzz is buzzing
It’s clear that maintaining everything in perpetuity is how you become #2
Yahoo would like a word with you
What about:
Gmail for Your Domain/Google Apps for Your Domain/Google Apps/Google Apps Premier Edition/Google Apps for Business/Google Apps for Work/G Suite/Google Workspace
Google gears is grinding
Gears has been superseded, not canceled
Actually, I think it's drowning.
Google tries a lot of stuff. It doesn't always work out for them, and sometimes they give up on it. I don't think that's a complete loss for the rest of us. We get to see what didn't work in the real world, and if it's really valuable, someone else can pick up the idea and build on that knowledge. I say this as someone who misses Google Reader.
I think the negative impact lots of humans regularly experience from Google's approach to business is far more than the benefit of the very, very specific learning that "this idea doesn't make enough money for Google the way Google did it".
Yes thats fine, but i would never trust to use one of their productions in production so to speak.
Yeah I wouldn’t build anything on Google’s products that might not exist in x years
"Gosh, that Italian family at the next table sure is quiet"
Do you see what you wrote?
I for one read it as intentional sarcasm.
guilty as charged
ditto
Tell us more about your grandma.
I think my dog has spent all weekend refactoring its own framework to integrate Jev.
First time?
This sort of works.
The problem is that you'll end up wanting to run 2 or 3 (or 20, 100, 10,000) agents at once and that gets very hard with a single VM.
There's also an argument that you should be using a separate sandbox for each code operation a LLM performs (or at least each set of related operations). That's even harder to do with conventional VMs.
I just use git worktrees in a single container. Albeit I don’t have more than 3 agents going at a time generally due to my own context switching limitations.
It's a different level of isolation, worktrees help agent work on different code repository in parallel but things get wonky once you consider processes and environments variables
The way I do it is each worktree gets its own .env, with dev credentials to whatever it needs access to, and my coworkers have theirs set up to have an isolated Postgres database per worktree, but I didn’t bother with that in my setup— maybe I will, though. It’s a simple script to create a worktree and properly prepare it.
I'm working on something in the "cloud VMs for agents" space[1], so I have some battle scars and opinions!
IMO, you want the flexibility to create either: (a) permanent devbox VMs, and (b) per-task VMs
Agent sandbox platforms tend to be tuned for the latter, which sometimes involves VMM hackery for fast boot, snapshotting VM filesystem and RAM, etc.
Some workflows are a lot simpler if the multiple agents share a VM. These are workflows where agents must share state. A simple one we have: making related changes in our public OSS repo and our private repo, and then testing the change.
And other times you want to split up the tasks onto isolated VMs so they don't interfere with each other (ie run two dev servers without database or port collisions).
I tweeted a bit about this (https://x.com/dbmikus/status/2099264325231771878) and had a little debate with folks about ephemeral vs persistent VMs for agents
[1]: https://github.com/gofixpoint/amika
Why would you need multiple agents and not one agent with multiple repos?
Tasks are a good scope for zero trust permissions
For my example where I modify both the OSS and private repos, I sometimes have one agent coordinate both changes (maybe with subagents), or if the work is modular and separate, I have disjoint agents do the work separately.
That said, most of the time, I only want an agent to work in one repo. I could give it multiple repos at once and instruct it to work in just one, but that risks it forgetting my instructions and it can load more context into the agent's window.
I think my main point was to have flexibility about the topology of VMs, repos, and agents.
From what I've seen, the vast majority of agent sandboxes with funding aren't for developers to use when coding, but for production applications that want to have LLMs do work. It's just a different model - APIs are better than a great terminal experience for a coding harness.
I've been working on https://lullabot.github.io/sandbar/latest/ which works with Proxmox for VMs (and lima for locals or regular linux hosts over ssh). There's a diagram in https://lullabot.github.io/sandbar/latest/why/#recommended-w... with what we're currently recommending. Though, after some feedback, I'm in the process of integrating a colleague's web-based review tool as it turns out many preferred fully reviewing locally instead of using draft PRs.
It's got some opinions in terms of default tools for our team and industry so it may not fit yours. Forgive some of the AI-isms in the docs, I want to get the UX and feature set to a solid place before doing a full review.
The goal is "ticket in JIRA -> solution in production" without human intervention. Right now agentic frameworks are multiplying to bring us closer to that solution like JavaScript frameworks did ~10y ago. We still haven't uncovered the "React" of this space yet, the one that business decides is good enough to standardize on.
We started building that but it quickly turned out to be too narrow. Often we want agents to do task that have no input ticket and often the output is not a code change (Slack bot, incident investigatior, scheduled daily tasks, ...)
Correct and that is why bash coding agents like pi.dev got things completely backwards with their anti-minimalist bloated core tools.
No, giving the agent access to every single command on the system is not minimalist. It is actively detrimental if you want to do more than just attended coding with the agent.
I don't mind that as long as that system gets regularly wiped. If you don't wipe it, you can't reasonably measure the actual output and it's pets Vs cattle all over again, only with agents.
I love how I don't even know how many years after they were created mainstream languages haven't yet figured out that the only sandboxing that works is default deny, like Tcl or Lua.
Especially with autonomous agents, it's the only way to sanity.
We might need new OS abstractions.
I've worked on that as well, and agree with you. You do need the "build this thing" flow, but that just shifts the bottleneck. You also need a whole infrastructure around it, where the jira-to-production pipeline isn't the interesting part.
What do you mean by the "React" of this space? The overall base layer has more or less converged on Kubernetes + MicroVM, which makes total sense. But if you mean how to interact with it as well as how permissions should work, yeah I agree.
Basically everyone has a sandbox of some sort to run agents inside. Everyone has a registry of some sort for tools. Everyone has a way of running agents inside a sandbox and giving it some tools.
Now the stuff people are coming up with is: how do you do authorization in this model? do you need a full sandbox all the time or can it be a workflow? how do you specify an agent is it a prompt or does it have some kind of control flow structure? How do you coordinate among many running agents?
I would say thats where we are now is there’s loads of people all solving the same problems a bit like when CoreOS, Kube etc. were all competing.
Note you don't need a sandbox if you are not doing code execution. There are a lot of applications where inference only is sufficient e.g. web scraping websites that don't change frequently or OCR on scanned documents.
Code execution (usually TS/JS or Python) is useful most when you are dealing with truly open ended problems. It's the opposite of the use cases of most enterprise SaaS.
You need it if your agent can access the internet and read files you don’t want public. That’s a relatively minimal case.
It is really easy to restrict and contain an AI agent as long as you don't give it access to a terminal. If you only give it tools to read files and access the internet, then it is much easier to just restrict the tools themselves rather than setting up a whole isolated sandbox.
Access to read sensitive files and access to internet could easily lead to data exposures on the internet, no? Without any terminal or shell access.
Yes, but what I'm saying is that it is much easier to put a limit on the read_file tool or the http tool rather than sandboxing your whole environment.
You only really need to sandbox when you provide access to tools that are almost impossible to filter correctly, such as a bash tool or a tool for arbitrary code execution.
Do you actually need it, or do you just fear you need it?
What's the actual realistic threat model for median developer or median user here?
By realistic, I mean that leaking your grandma's recipes or your SSN or your million dollar idea to some pastebin is neither likely nor going to meaningfully make things worse for you, or be useful for any malicious actor. Surely this is not what everyone is worried about?
I spent today doing forensics on ten compromised WordPress sites sharing one hosting account.
I used two agents: One with network access to collect the evidence, and one with everything except the model endpoint cut off, which did the analysis.
The second agent's entire input was attacker-authored. So PHP droppers, obfuscated loaders, database rows, filenames, blah blah.
In this case I'm more worried about hostile input attacking the agent, and I need to contain the damage. My sandboxing solution does that by restricting access to the source data, making it read-only. The work dir can only transfer data via patch and apply (like a git workflow), so even my workspace can't be modified until I approve each change. And then restricted network means that any compromise ain't going noplace.
The second agent couldn't even install PHP or contact any CVE site to check if it was looking at a known attack, and that was by design. All it could do is write up a report about what it observed, not make assumptions about what it is. I could then take its (much smaller) clean output and pass that to a third agent with network access.
This is forensic work, so of course not your median dev's bread & butter. But the attack surface is only just starting to be plumbed. Compromising input can turn your agent into their agent, planting things as easily as planting worms was back in the early internet days when people connected without a firewall.
Security by obscurity is just a bet on weights, a belief that the economic motivations for attacking are insufficient. That worked before, but developments in ML calls to account all the debt we’ve accumulated through that practice.
Nothing at all. You'll know when you've outgrown it.
Graph-based workflows where agents pick up work as it becomes available, structured output, while you manage the work queue and outcomes. Maybe? IDK really, it's all moving quite fast.
Where are the revolutionary software products?
Why would you expect that?y AI output are all similar to what I built before, but much more of it.
Revolutionary products depends on revolutionary ideas, not faster execution.
https://chatgpt.com/ is a good starting point. Enable voice control and ask the robot to draw a pelican riding a bicycle. Much wow, very revolutionary. More to come.
My advice: stop your fomo. Just get useful work done.
There's no convergence, but there sure as hell is a lot of pseudo-scientific voodoo and overengineering going on.
I think at this point there is also lots of wild enthusiasm and not that much proof mass-agent anything + orchestration is actually financially viable or even useful.
I outgrew this when I wanted to bring different sets of skills and templates to different machines, wanted to be able to share a small number of credentials, different agents in different machines, different egress rules etc. I wrote https://github.com/pjlsergeant/byre which gives you a TUI and some machinery for doing this easily on top of Docker or Podman.
I have no idea, I just live in my terminal at this point, on linux. Don't use Visual Studio or Jetbrains products anymore at all. I use visual studio code to view the occasional diff and run sql queries. Other than that, zero desktop apps, just terminal & cli tooling. Its great! I have about 20 terminals open at any given time though. I have no idea how some of my colleagues stay productive as they are messing around with all these workflow tools, desktop apps etc.
Yup. I'm also working more and more in a terminal typing English sentences to my computer. It looks like a very smart adventure game UI from the 80s.
A little git log, show, diff almost always in another terminal.
A customer of mine wants to standardize his developers on a Jetbrains IDE for python but I think that he is late by one year. Furthermore he is using the subsidized plans for Claude, not paying for token, so it makes sense to keep using the Claude TUI.
Doing the same more or less, replacing vscode for vim. Usually three/four panes open, one codex running in a container so it doesn't share my host filesystem, one pane for git diffs/git, one for general shelling and one for neovim basically. One of these per project/task/worktree I'm working on, usually 1-2, maximum 3 at the same time.
Not sure what all the other folks are doing, but the industry/ecosystem tends to over-engineer every single thing instead of just working on the thing, while I just want proper code, proper design/architecture, and proper high-quality results.
It's kinda true. I use the terminal approach too, but I wouldn't trust every type of user with this long of a leash. There's a place for instituting systems that share context, manage authentication, and is permissions-aware.
For my workflow I need both, a permanent-ish VM and emphemeral sandboxes. Both have their place and pros and cons.
VMs are better for personal assistant work, GUI clicktesting, investigating bugs in your personal dogfooding dev instance and anything you haven‘t yet made repeatable and fast to set up.
Sandboxes are better when you need resource isolation or security and have a graph of tasks to work through. My agents often starve each other on one VM, so if they don‘t need any of the above it‘s just easier to isolate them.
Everyone is working in this area, including me [0], but either option really isn‘t that convenient to use yet. It‘s a bit of a „isn‘t Dropbox just FTP on a VM“ moment right now.
[0] https://github.com/madeinorbit/podium
Since this is not the first mention of Dropbox I've seen in HN threads in the last 48 hours:
Let's not forget that Dropbox was at its best when it was "just" a streamlined ftpd over sshfs or whatever - when it was just "a folder that syncs". That didn't last long, the downfall started with them killing their most useful accidental feature[0], which started them on a path of enshittification[1], which they followed swiftly and diligently into complete irrelevancy they enjoy today.
So if the agentic tooling is now enjoying its "Dropbox moment", I implore people working on these tools, don't overdo it.
--
[0] - The "Public" folder initially supported direct linking, meaning you could publish static web sites by simply putting them in Dropbox/Public/, you could update the files there and changes were immediately "live". Notably, this was the heyday of phpBB and similar discussion boards, back between the rise and subsequent fall of free image hosting - so the ability to put images in your Dropbox/Public/ and hotlink them in a discussion was extremely useful and popular way to use the service.
[1] - They didn't just kill direct links, they replaced them with what I consider to be OG enshittification pattern - captive page that asks you to press a button to download. Yes, same one every "synced drive" service offers now, to enable various functionality that's 99% harmful to the user with the link.
Maybe I knew how they explained the change back then but I forgot about it. If I must take a guess now, maybe their legal office had a word with marketing about the risk of becoming a publisher, with responsibility for what their users shared publicly on the internet.
Perhaps. This was also before CDNs were a thing, so they may have decided they can't afford to be one, as more and more people discovered just how versatile the Public/ folder is.
Either way, this was the peak of Dropbox; after shutting down direct Public/ links, it was still useful for its main job as seamless cross-machine, cross-platform "folder that syncs", but gradually lost market share as OneDrive and Google Drive became more broadly useful (and had the advantage of being first-party on their respective platforms), and then Dropbox the company itself lost focus and tried a bunch of failed pivots in the direction towards cloudification, away from "just syncing files".
End result for end users? We now have zero options for bullshit-free, file-first, seamless "folder that syncs" experience for non-tech users (techies that like fiddling with things have Syncthing). Only cloud-first options remain, and they're full of footguns and enshittified to the core (which becomes apparent the moment you want to share a file outside of the vendor's cloud ecosystem).
The general workflow I have seen for non-technical people building software is something like:
- Do multiple tasks in the same context window / session, conflate different changes into the same prompt
- Repo mixed with old markdown files from previous tasks, excel and word docs and 300 playwright screenshots
- 5 tools all calling each other, test and deployment scripts are all markdown skills
Personally I prefer a ticketing system and isolated work trees
Pros and cons. The sloppy approach actually works well for Claude because its memory retrieval is more reliable than AGENTS.md style instructions.
I wrote a little program to spin up isolated rootless workspaces: https://github.com/jgillich/tpd
Works pretty well for me but I haven't put any effort into promoting it
Personally I think microVM are the future but nothing wrong with a solid proxmox setup.
Probably we'll converge on a virtualised IO / Storage layer running microVMs beneath for isolation and security. Keep the network and storage layer separate for compatibility running a variety of stuff and a second security boundary.
From their own example "Setting up a Python 3 environment" is best it can do.
An even simpler "sandbox" is to create a separate user and home dir for your LLM.
I've been using https://github.com/mastra-ai/mastra which is pretty similar but has workflow visibility and a number of templates.
For a generic swarm, workflows aren't too useful which does away with the visibility, so I may give this a try instead.
People can afford to run billions of concurrent agents?
Not sure about billions, but companies doing evals or RL or training will create really big bursty agent workloads. I think they are the best fit for AX, as opposed to individual dev teams building software, etc.
An example of this, Moonshot (kimi) open sourced this: https://kvcache-ai.github.io/AgentENV/latest/getting-started...
billions? who is running BILLIONS of agents?
tens, hundreds, maybe a couple thousand at a time? absolutely.
I'm planning to buy a whole linux mini-PC to run my agents/code servers for more isolation. Codex/Claude Code let you run prompts on code over ssh (same with most IDEs) even on the desktop apps.
I wonder if that's going to be the new standard practice. You get a work laptop and an isolated agent box.
Running access control and network whitelists is always a maintenance challenge and it's easy to make mistakes.
I think it will be, but I don't think you need a standalone machine! If you run things inside a VM, you can get safety and control over access and networks
A standalone machine is nice if you need more compute resources or if you want an always-on machine you can connect to from your laptop, phone, etc.
It doesn't look like Google's AX is quite the plug-and-play fit for running agents on a computer you own, since it requires setting up a K8S cluster, etc.
I think what's needed is something like a zero-setup combo of Tailscale and Firecracker
I'm trying to work towards that with my startup (https://github.com/gofixpoint/amika) but the bring-your-own-computer part doesn't work quite yet.
I have 6 and ended up needing to use my gaming PC for a build server.
I think you could get by with 1 computer, but it’ll have to have a pretty decent machine.
Between agents running tests, CI, docker image builds, an average $400 mini PC won’t cut it.
Don’t forget also many older mini PCs don’t support KVM. Some newer ones don’t support AVX/ mongodb.
It’s not so easy to buy any old hardware sadly.
You might be right, it likely needs a full proper PC setup with the test suite stuff. I was looking at this vendor, https://www.gmktec.com/collections/all there's this whole AI mini-pc market but they aren't quite a full dev machine replacement
I’m running ~5 agents at a time very comfortably on a $280 mini PC with 8GB. They’re all in Docker containers, a couple have sidecar VMs they can own and run. Not having any issue with load.
Depends a lot on your workload. I build large Rust projects so really only one can build and test at a time.
Interesting, we had developed a very similar framework for our internal agents: https://github.com/apoxy-dev/clrk For us main use-case was intercepting all network I/O including LLM providers, HTTP, and random TCP/UDP calls
Nobody has a use for this, and anybody who can look at this website and work out what it's for is kidding themselves. Even the demo gif playing just has them pausing a task and resuming the task.
it feels like the kind of interface a devops engineer who hates AI would design
yes, has a bad smell of k8s
Fully agreed. Google just can't stop losing.
https://github.com/google/ax/blob/main/docs/concepts.md
Remember kids, a model is not a model.
There are two hard problems in computer science, naming things, cache invalidation and off-by-one errors:
https://github.com/google/ax/issues/356
If there's a criticism here, it's that they don't mention k8s right up front. It's completely opaque what this tool is. "Orchestration" can mean literally anything.
The web page presents ax as a typical developer tool, but it's actually not for developers.
Kubernetes is the last thing I wanted to see recreated for agents. It’s like Multics of cloud, now for agents. Complexity for the sake of it, powered by your favourite YAML slop bowl.
Am I being gaslighted into thinking over engineered systems are not?
Interesting approach. How does it compare to LangGraph for multi-step agent workflows? The orchestration layer always seems to be the hardest part to get right in practice.
The project seems like an open-source initiative born out of the experience of some Googlers but not being used at Google. So, the title appears a bit misleading - people will be misled.
Anyone who uses this promotion packet fodder for anything important is a fool
I looked at the website, and I still don't understand the purpose.
Same. I don't get it.
The best part of all of this is the most if not all people have no idea what the hell is going on when every day a new paradigm/tooling/harness emerges. It's hard to keep up. On the plus side, it's a great equalizer.
Offload the work of an LLM agent to a box in the cloud, so it doesn’t run on your laptop. This has security benefits (no access to your laptop’s files) and you can scale it up (run a lot of agents at the same time).
Then put a “sandbox” around these agents, that word has many meanings. In this case they fence the network traffic, so likely some kind of allowlist for network requests so that the agent doesn’t exfil crap to random websites. They also limit the resource limits of the sandbox, so that is beneficial to the cost of running these agents.
Everything is always better with more YAML, are you perhaps new to this industry?
Next we also need an instruction style guide and CoC. It's important to treat your agents with respect. I almost forgot, the YAML template meta-language to YAML the YAML. Then we will need a foundation employing 12 FTEs to maintain it all and of course to run the certification process. You are certified, right? Statistics show a 10x increased chance of an agent going rogue and hacking competitors if it has been mistreated or been run in an unvalidated sandbox. It goes without saying the sandbox certification process is separate and must be repeated yearly by a trusted third party auditing company.
k8s, but for agents
must look cool for people who want to solve every problem with k8s
it starts with interesting misnomers like "Task" which is not a work item but a sandbox.
"billions of tasks" is a "solution" to problem nobody has (maybe some RL labs? but they solve it other way and with orders of magnitude better optimizations).
freezes design too early – unless they'll actually focus on developing it and make tons of breaking changes it looks shit.
shared state in the same workspace, identity, authority, etc – stuff like that needs to be solved
AX, not to be confused with Ax the machine learning tool from Meta for optimizing experiments.
https://ax.dev
No to confuse with Ax, the DSPy inspired agent framework
https://axllm.dev/
Unnecessary complexity packaged as product!
The real difficult in arranging agents is not to run them, but to identify the state change - to judge whether an agent stops to wait for you, or is stuck, or finished, and whether the two should be handled automatically or someone should be found. You have made this judgment for 12 agents, and you need to know how unreliable it is.
It looks great from a security standpoint, but it also feels like overkill.
How is this different than k8s jobs?
K8s jobs don’t run in a secure runtime. K8s jobs don’t give you dynamic zero trust permissions scopes. Restoring a harness in 500ms is really fast, much faster than naively creating a new job downloading session and ‘codex resume’ etc.
not exactly for agent orchestrator, but I did make something similar in terms of design: https://github.com/yash-srivastava19/cadence
Surprising no mention of Google on the page or domain.
Oh no, Kubernetes for agents. I guess all roads lead to complex YAML.
"2. Deploy the control plane
You need a Kubernetes cluster"
LOL. Bye!
I was expecting that, in the AI era, even Google will start using Rust for everything. But they chose Go for the this project.
I'm curious, what type of problems is this tooling aimed to solve? Isn't it a bit of an overkill regular webdev i.e.?
Didn't you see the example on the website? It can set-up a Python 3 environment. Duh!
Yeah, my grandmother also did something on this https://github.com/nillebco/varda
Essentially there is no out of the box solution about orchestrating agents and increasing LLMs sandboxing. That's why everyone and their grandmother are re-inventing the wheel.
At the same time, it's an incredibly complicated problem, with a variable perimeter (OS support, sandboxing primitives support).
I am quite happy about my own solution (because it supports my use case!) but I hope something with a decent dev UX will appear one day. AX definitely is NOT.
Is the logo a cheerful little parasitic skin mite?
Axolotl
ah...! I never would have guessed in a hundred years, but after googling a picture I can see it now
I believe it's a stylized lo-fi rendition of an axolotl.
It's most likely an axolotl.
Kubernetes but for agent ?
I am wondering how this is related to https://agent-sandbox.sigs.k8s.io/ ? Since there is also https://docs.cloud.google.com/kubernetes-engine/docs/concept...
IIUC agent-sandbox and agent-substrate (the one ax builds upon) are similar. Agent-sandbox is more k8s-native, while agent-substrate is less so.
Personally I think that this kind of workload is better off not being tied too much into kubernetes. I've worked with crossplane and other controller who put a lot of load on the k8s-apiserver and etcd and can easily slow the whole machinery down / grind them to a halt.
btw, agent-substrate is in the process of being moved to CNCF: https://github.com/cncf/sandbox/issues/523
This is different from langchain etc because lanchain works at the app layer but this one works at the infra layer with tool calls etc?
It's crazy how far behind Google has fallen in this space in just a single year
We have built similar abstractions directly on top of Kubernetes [1]
I was looking at this project a couple of months ago, and I did not understand why not use Kubernetes instead of rebuilding the abstractions. The reason is that Kubernetes already provides other abstractions to run services and batch job, gang scheduling, gpu and other accelerators enabled workflow.
[1]: https://polyaxon.com/docs/sandboxes/overview/
So on one hand the page says
On the the other hand, the readme quickstart section says
Call me old-fashioned but I don't find this "easier". Maybe it's easier in the same way that Kubernetes itself is easier than managing VMs and container deployments at massive scale without such a tool. But there's a vast chasm between what this tool is being sold as and what it actually is.
It is easier in that if you have infrastructure already, it's trivial to add this on top. The primitives look also very familiar.
lmao found the dev who's only ever worked on the dev side of things.
Easy as in "Google cloud console interface" easy
As with everything in software engineering, any average dev can fire up a few containers with Docker, but running anything in a real environment takes a whole team of people who actually understand the platforms involved in order to deliver a fully functional service, hopefully via properly designed code (TF) for ease of reusability. Most devs simply don't care to think about that part, and then throw it over the fence for "someone else" to deal with. Just as long as they can say "DONE!" (I created a thing!), that's all that matters.
Obligatory https://youtu.be/3t6L-FlfeaI
Google operates at such a scale with a wide surface area of serious production considerations that even “ergonomic” solutions internally feel extremely heavyweight externally.
Source: I’m an Xoogler
Easier from enterprise perspective.
Classic AI solution, you do the hardwork so that you can chat to an agent to do the easy part.
If you are approaching anything looking like an enterprise environment, you are likely to have Kubernetes nearby. ko is a weird requirement. ax requires agent substrate, so it makes sense that it's needed. Agent Substrate also has experimental support in kagent, so it makes sense that it is growing.
I don't really like the oversubscription of agent pods though, as you can no longer trust the k8s pod identity as being from a singular workload. Haven't seen a solution to this for ax yet and it is a barrier to adoption for us.
One thing I’d say, is that I find it progressively more interesting/fun to rollout your “everything”. I mean… hello security, but by the time I’ve read somebody’s documentation I’ve already halfway done making thing exactly how I want it.
Kubernetes but for agents :(
Since hearing the word orchestrator in ai context it was clear that ClanKernetes was coming.
Pick one.