Apple's web services are shit, so I hope not. When it works it's great/okay, but when it breaks 5-10% of the time there's absolutely no indication what the hell is going on. Apple's allergic to progress indicators and useful error messages.
I'm hearing an awful lot of "How could they do this to us?" and not a lot of "Wow, maybe we should have read some of this 3rd party code we bundled into 'ALLLL' of our apps"
Just an ignorant and naive outsider's take, but to me it paints a pretty damning picture of the state of mobile development.
maybe we should have read some of this 3rd party code we bundled
Not really practical though, if you push the thought to its limit. That 3rd party code is literally everything that's not written by you, from firmware to the launch UI. And if you don't push the thought to the limit then there's always that risk leading to the same "maybe we should've read X" if something happens in X. Trusting that others will be good stewards of all that 3rd party stuff is a hard requirement for making progress.
I agree its not practical, but including any 3rd party libraries in your project puts it at real risk of upstream bugs. There needs to be acceptance of this rather than blame culture.
Because their client SDK might crash the entire app if the server does something funky. We saw it before with Facebook, and now it's Firebase's turn.
A dependency on code you can't read reaching out to servers you don't control is a recipe for disaster. If you can control the server, you can try to add fixes, redirect DNS to a backup cluster, you name it. If you implement the client, you can write the code in a way that doesn't cause full crashes when the remote server does something weird. If you can do neither, you're handing over your business flow and uptime to a third party that doesn't care about you in the slightest.
I remember first learning about Firebase when working on Android push notifications, a loooooong time ago (~2011 i think). Over time it grew into this full-featured app development platform, after an aquisition (don’t remember the name).
These days however it feels a bit neglected, and somehow poorly bolted on to GCP. I still have a few production apps running on it and news like this reinforces my belief that it’s in decline.
What are folks using these days that (ideally) is open source and self-hostable? I don’t want to lock in with another platform. Some of these apps use the offline sync feature of Firestore (apps used in basements and other low-connectivity areas).
I guess Supabase is the obvious contender. It doesn't provide the actual frontend hosting though, which is a bit of a pain because now you have two problems.
I wasn't aware that Supabase is self-hostable - thanks for pointing that out, will have a look. Customer is in Europe, if they decide to move away from Firebase, they don't want to migrate to another managed platform run by a US company (digital sovereignty etc.)
Yes that's the sad part, if you want to integrate Push Notification in your app, and you want to use a 3rd party vendor like OneSignal etc, you still need a fuckin FIREBASE account & FIREBASE dependencies in your app. So every app on the Play Store comes with firebase dependency.
Calling them a 3rd party push notification vendor is being generous. Google is still doing the push notification and I guess OneSignal offers a wrapper around the libs.
This will be the new normal as all human “engineers” from staff to seniors are now down levelled to interns and junior engineers when using AI; unable to understand what they are doing and pushing broken updates like this into production with “agents”.
The coding side of this isn't even the problem - that a broken change like this made it to production is an operational failure.
Why didn't testing catch this? Why was there no canary? How come alarms didn't wake up an on call as soon as the call-volume on the backend dropped? Why was this update rolled out to every customer at once, instead of gradually?
I think verification is still a grey area in the agentic software dev.
QA and ppl testing it often resort to coding agents for the QA work, and we all know how good agents are at convincing themselves.
Wrong.
They will be totally blamed. And the pushback will be that management forced me to use AI and go fast. Then, management will be the one NOT being blamed for this.
And in many companies, if they move at a responsible speed (rather than the speed the AI vendors promise, if you let their tools work in yolo mode), that person will get replaced by someone who will just yolo it
Resolved - https://github.com/firebase/firebase-ios-sdk/issues/16728#is...
Resolved, but can still affect customers for up to 4 hours...
So many problems here, and no signs of an updated SDK that behaves properly on malformed input.
This is not the first time this has happened.
I was wondering why random apps kept crashing today. Good to know.
But this is completely not acceptable from the Google
They’ve been accepting this for years and so have users
Ugh.
That's the dark side of these types of dependencies.
But they provide a great deal of utility, so I can understand the attraction.
I guess it's time Apple sherlocked Firebase.
You can write your own crash reporter now.
Apple's web services are shit, so I hope not. When it works it's great/okay, but when it breaks 5-10% of the time there's absolutely no indication what the hell is going on. Apple's allergic to progress indicators and useful error messages.
I'm hearing an awful lot of "How could they do this to us?" and not a lot of "Wow, maybe we should have read some of this 3rd party code we bundled into 'ALLLL' of our apps"
Just an ignorant and naive outsider's take, but to me it paints a pretty damning picture of the state of mobile development.
Mobile and frontend development, both, now-a-days contain way way way more dependencies than they should.
Yes, but the SaaS adoption is also the reason. Many analytics SDKs, observability tooling etc.
Not really practical though, if you push the thought to its limit. That 3rd party code is literally everything that's not written by you, from firmware to the launch UI. And if you don't push the thought to the limit then there's always that risk leading to the same "maybe we should've read X" if something happens in X. Trusting that others will be good stewards of all that 3rd party stuff is a hard requirement for making progress.
I agree its not practical, but including any 3rd party libraries in your project puts it at real risk of upstream bugs. There needs to be acceptance of this rather than blame culture.
You're already trusting them with their proprietary cloud products, whose code you can't read. Why wouldn't you trust them with their client SDK code.
Because their client SDK might crash the entire app if the server does something funky. We saw it before with Facebook, and now it's Firebase's turn.
A dependency on code you can't read reaching out to servers you don't control is a recipe for disaster. If you can control the server, you can try to add fixes, redirect DNS to a backup cluster, you name it. If you implement the client, you can write the code in a way that doesn't cause full crashes when the remote server does something weird. If you can do neither, you're handing over your business flow and uptime to a third party that doesn't care about you in the slightest.
I remember first learning about Firebase when working on Android push notifications, a loooooong time ago (~2011 i think). Over time it grew into this full-featured app development platform, after an aquisition (don’t remember the name).
These days however it feels a bit neglected, and somehow poorly bolted on to GCP. I still have a few production apps running on it and news like this reinforces my belief that it’s in decline.
What are folks using these days that (ideally) is open source and self-hostable? I don’t want to lock in with another platform. Some of these apps use the offline sync feature of Firestore (apps used in basements and other low-connectivity areas).
I guess Supabase is the obvious contender. It doesn't provide the actual frontend hosting though, which is a bit of a pain because now you have two problems.
I wasn't aware that Supabase is self-hostable - thanks for pointing that out, will have a look. Customer is in Europe, if they decide to move away from Firebase, they don't want to migrate to another managed platform run by a US company (digital sovereignty etc.)
Especially for offline sync Firebase is still much stronger than other all-in-one solutions.
The real alternative there would be to use some SQLite syncing solution and a different solution for auth and cloud compute.
While Firebase has support for queries with realtime updates and offline caching, I wouldn't quite call it offline sync.
Yes that's the sad part, if you want to integrate Push Notification in your app, and you want to use a 3rd party vendor like OneSignal etc, you still need a fuckin FIREBASE account & FIREBASE dependencies in your app. So every app on the Play Store comes with firebase dependency.
Calling them a 3rd party push notification vendor is being generous. Google is still doing the push notification and I guess OneSignal offers a wrapper around the libs.
If only there was any way to avoid this, right..? (I don't mean from Google's side, that as well, but that's not the point)
No, not under your control. It is like "trust me bro" thing from your dependency maintainers.
Keep vibe coding and breaking everything.
This will be the new normal as all human “engineers” from staff to seniors are now down levelled to interns and junior engineers when using AI; unable to understand what they are doing and pushing broken updates like this into production with “agents”.
Better not blame Claude on this outage.
No matter who writes the code, the name in the commit is responsible as well as the person who approved / reviewed the PR.
The coding side of this isn't even the problem - that a broken change like this made it to production is an operational failure.
Why didn't testing catch this? Why was there no canary? How come alarms didn't wake up an on call as soon as the call-volume on the backend dropped? Why was this update rolled out to every customer at once, instead of gradually?
I think verification is still a grey area in the agentic software dev. QA and ppl testing it often resort to coding agents for the QA work, and we all know how good agents are at convincing themselves.
-
The solution is testing it yourself, not more AI
Why pay for and bother with all of that when you can outsource it to your users?
Users who will forget about it, and due to inertia stay on your platform.
No matter how many times this is said, the reality is that it is not true: no one will get blamed if the AI can be blamed.
Wrong. They will be totally blamed. And the pushback will be that management forced me to use AI and go fast. Then, management will be the one NOT being blamed for this.
And in many companies, if they move at a responsible speed (rather than the speed the AI vendors promise, if you let their tools work in yolo mode), that person will get replaced by someone who will just yolo it
At least Google engineers can invert a binary tree
Lol, but they can't build BREW ;)
Yeah, surely atleast 12 of those engineers tested their SDK thoroughly for a minimum of 14 continuous days[1]. /s
1. https://support.google.com/googleplay/android-developer/answ...
Yeah, and pinning versions doesn't save you when the config comes from their server.
Ironic. They could save others from crashes, but not themselves.
A friendly piece of vocabulary assistance: it's "this morning" ("today morning" is not correct English)
Exactly the same thing also happened in 2020 with the Facebook SDK:
https://news.ycombinator.com/item?id=23097459