I read you migrated to an S3-based git backend — would love to hear more about that story and how you managed the migration. Did you migrate all repos at once or on demand?
Also chapeau for the quick turnaround, I logged into lovable and saw the incident and you didn't let GitHub slow you down :-)
regardless of scale there will most likely be code paths where performance is not prioritized, and unexpected use-cases can put the stability at risk. In this case that was creating new repos. I believe if we only made a crazy amount of commits there would have been no issue.
They hopped call with AWS Suport folks about their use of S3 and yet this happened:
“ 17:00: While rewriting the migration we find other issues since S3 allows multiple concurrent writes to the same project. This is a race condition that would have been prevented by Github since it handles out of sync commits thanks to it being an actual git repository. On Github, if a pushed head already exists as a commit it won’t overwrite a newer head, but there are no controls like this in S3. The impact mostly seems to be for new projects.”
That is a relatively well known failure pattern with git on s3 and it’s a shame the s3 representatives weren’t aware or there isn’t a knowledge base to tap into. I have found with AWS you have to get beyond the first level of support to have any chance of having a meaningful conversation.
When it comes to user data, you have to be extremely careful, so I was surprised on X how quickly they were moving things off of GitHub in a relatively celebratory manner. They had a ton of attention from this incident so I guess they wanted to recover as fast as possible versus being sure they couldn’t lose user data (and to be fair it seems like it was all recoverable). Still should be careful as their service becomes more popular and important.
To the devs: if I type "https://lovable.dev/projects/zzz" as the URL, then the browser will start continuously DDOSing the web server with retries. Currently I'm at 2k requests on the page.
that's one way to wake up the team after New Year's eve
Founder here:
Not part of the report was that getting intro to people on Github significantly helped accelerate things.
Any questions about Lovable infra for those curious generally – happy to answer.
(also, we're hiring senior platform engineers in Europe to build the first European AI outlier success story:)
I read you migrated to an S3-based git backend — would love to hear more about that story and how you managed the migration. Did you migrate all repos at once or on demand?
Also chapeau for the quick turnaround, I logged into lovable and saw the incident and you didn't let GitHub slow you down :-)
Very happy this was fixed, I was just about to ship an update to my Lovable app when this happened..
heard many that were just about to demo lovable apps when this happened..
Kicking off 2025 with the Lovable Support Team
regardless of scale there will most likely be code paths where performance is not prioritized, and unexpected use-cases can put the stability at risk. In this case that was creating new repos. I believe if we only made a crazy amount of commits there would have been no issue.
Kudos to the dev team for being so quick
315k repositories and creating 10k repositories a day?
Sounds like you need to run your own git server.
They hopped call with AWS Suport folks about their use of S3 and yet this happened:
“ 17:00: While rewriting the migration we find other issues since S3 allows multiple concurrent writes to the same project. This is a race condition that would have been prevented by Github since it handles out of sync commits thanks to it being an actual git repository. On Github, if a pushed head already exists as a commit it won’t overwrite a newer head, but there are no controls like this in S3. The impact mostly seems to be for new projects.”
That is a relatively well known failure pattern with git on s3 and it’s a shame the s3 representatives weren’t aware or there isn’t a knowledge base to tap into. I have found with AWS you have to get beyond the first level of support to have any chance of having a meaningful conversation.
When it comes to user data, you have to be extremely careful, so I was surprised on X how quickly they were moving things off of GitHub in a relatively celebratory manner. They had a ton of attention from this incident so I guess they wanted to recover as fast as possible versus being sure they couldn’t lose user data (and to be fair it seems like it was all recoverable). Still should be careful as their service becomes more popular and important.
To the devs: if I type "https://lovable.dev/projects/zzz" as the URL, then the browser will start continuously DDOSing the web server with retries. Currently I'm at 2k requests on the page.