It lists all instances, you can drill down into each one of them, see which services are affected, and each affected service pops up the incident timeline?
Isn't it actually amazing, and not "the most salesforce thing ever"?
"And in tonight's news, the worldwide CRM solution Salesforce had a global outage affecting one hundred percent of its customer base. We interviewed users of the service to find out the scope of the impact. Everyone agreed that they were impacted, but strangely, nobody could describe _in what way_ they were affected."
And nothing of value was lost. God, I hate everything about Salesforce. Sometimes I have to integrate against their services, and it is always a pain, not to mention what the core project actually is: optimization of marketing and spam.
My gut instinct is that this is about when all of their on prem servers were EOL and their /public cloud solution was required. This must have had something to do with that
You can click on any of the instances and then the service that is down to read the updates. It’s not 100% clear but some sort of issue with a “legacy login service”. The latest updates say a fix is rolling out.
Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it.
We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue.
At least it didn't fix the problem so they can actually start finding the real cause.
We're no longer pursuing restarts as a path to remediation.
This is a good example of why isn't the AI they sell telling them what's wrong? Why do they need to try restarting and "see if that resolves the issue"
This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.
Cause: Legacy Salesforce login service got into a resource-exhaustion cascade.
Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed).
Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
That status page is the most salesforce thing ever.
Scroll down. >_<
At least they're consistent about UX. My only complaint is that it needs more tabs.
I see tabs with a spinner loading infinitely, which is indeed very salesforce like
Wow, it's almost as long as the Every UUID V4 or Every Floating Point Number pages.
Yup. No mention of outage. Even drilling down gets nothing more than "Service Disruption".
It lists all instances, you can drill down into each one of them, see which services are affected, and each affected service pops up the incident timeline?
Isn't it actually amazing, and not "the most salesforce thing ever"?
fucked
This is certainly a unique status page.
At least now we can figure out what Salesforce does.
"And in tonight's news, the worldwide CRM solution Salesforce had a global outage affecting one hundred percent of its customer base. We interviewed users of the service to find out the scope of the impact. Everyone agreed that they were impacted, but strangely, nobody could describe _in what way_ they were affected."
Perfect timing with Dreamforce this week.
Remind me please, what are folks currently paying per seat for this glorified CRUD app?
It's this thing called golf course driven development
Permanent cache for that one.
If I had been just out of Uni I would think this is edgy
Now I'm just glad I'm not responsible for this fire
Was it DNS? Any guesses? :)
Vibe-slop, push to Prod?
They let their agentic AI take over system maintenance at Dreamforce yesterday like they were pushing in the talks
/s (partly)
Could it be people doing Claude/GPT automations and they just can't handle it?
[delayed]
And nothing of value was lost. God, I hate everything about Salesforce. Sometimes I have to integrate against their services, and it is always a pain, not to mention what the core project actually is: optimization of marketing and spam.
not really defending salesforce but OAuth+REST is a pain? Pretty plain vanilla in terms of integration requirements.
You better bet someone started their agents with a prompt "Make a salesforce clone but with 100% uptime"
"Make a salesforce clone but with 100% uptime"
It's dns isn't it
Classic
No, IPv6
It's always DNS.-
More like 70% human-configured DNS, 25% human-configured routing configuration, 5% interesting software bug.
Entirely correct.-
(Nowadays any of those need to fit in an "agent dropped all tables. Apologized" moment.-)
Feel like it has to be for all of this to go down at the same time.
Never before in the history of global compute outages was so little lost by so many down servers, whose purpose was known to so few.
So I guess today everyone gets actual work done
Wow, the intern must have tripped over a very big power cable this time
Sam speaks at Salesforce.
Salesforce goes down.
No causation here...move on.
Wasn't it Dario
"Trust just got personal!"
My gut instinct is that this is about when all of their on prem servers were EOL and their /public cloud solution was required. This must have had something to do with that
My gut instinct is that this is about Dreamforce with the rickshaws and whatnot
Ah yes. Exactly what a status page should look like: an endless list of random ID’s that don’t mean anything and no information whatsoever
At least salesforce is consistent with their design language
If you use salesforce you know what all of that stuff means. Just click on one, it’s not rocket surgery.
Was really just poking fun at them - AWS’s status page isn’t much better
Looks like a region list to me, maybe just with a lot of regions
You can click on any of the instances and then the service that is down to read the updates. It’s not 100% clear but some sort of issue with a “legacy login service”. The latest updates say a fix is rolling out.
Here is link to incident details: https://status.salesforce.com/incidents/20004433
Dreamforce, wake up. You’ve overslept!
It’s akways DNS or login
Haven't they got some kind of new fancy ai interface they can use to fix it?
Have you tried turning it off and then on again?
Oh you have
Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it.
At least it didn't fix the problem so they can actually start finding the real cause.
This is a good example of why isn't the AI they sell telling them what's wrong? Why do they need to try restarting and "see if that resolves the issue"
"Yeah, I'm with Rob. Just let's reboot and see what happens"
Unplanned outage timing is never good but this is really not good.
https://www.salesforce.com/dreamforce/
Sept 15-17
Probably not a coincidence
This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.
Arguably, making customers and partners feel good is the more important part of the business
Cause: Legacy Salesforce login service got into a resource-exhaustion cascade.
Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed).
Details at https://status.salesforce.com/incidents/20004433
Seems it is back up now. Damn.
At this point OpenAI really ought to let us know when they're testing again.
Leetcode developers win again
Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
ClaudeForce in action!