A few months ago, I've been training several simple models as a side hobby. I tried training N models at once on a single GPU, but OOM spikes kept crashing my runs. So I built my own simple way to manage this kind of training. At the time, I also wanted to make some sort of interface that AI agents could use to dynamically allocate GPU resources and train models on their own. It's been quite fun working on this, so I wanted to share it in case any of you are looking for something similar. Claude helped me write most of the code, but I've made sure that the stuff works as described (I've used this exact repo for my own training runs).
A few months ago, I've been training several simple models as a side hobby. I tried training N models at once on a single GPU, but OOM spikes kept crashing my runs. So I built my own simple way to manage this kind of training. At the time, I also wanted to make some sort of interface that AI agents could use to dynamically allocate GPU resources and train models on their own. It's been quite fun working on this, so I wanted to share it in case any of you are looking for something similar. Claude helped me write most of the code, but I've made sure that the stuff works as described (I've used this exact repo for my own training runs).
Hope you like it!