I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?

https://en.wikipedia.org/wiki/Exponential_backoff

https://devblogs.microsoft.com/oldnewthing/20051107-20/?p=33...

This is trading a good developer experience for a bad user experience. There are situations where it makes sense to force manual retry, but there's no reason to apply one universal rule to all possible situations. Lack of considering nuance for your situation is just intellectual laziness.

My point was that just throwing exponential backoffs at the retry problem is not a magic solution

I don't follow how being cautious about avoiding multiplicative layers of backoffs is trading a good developer experience for a bad user experience. The described situation is an awful user experience. Simply adding a retry and calling it a day sounds like the easy developer experience at the expense of the user experience

The described situation is an awful user experience

Sure, the worst case scenario is. 99.9999% of the time, a transient error will actually just work on the first or second auto-retry and save your users the effort of paying attention and manually retrying things. This is especially prudent for background tasks where the failure may not be noticed right away; coming back to something fire-and-forget 30m later to see it never tried to finish is not a good user experience.

My point was that just throwing exponential backoffs at the retry problem is not a magic solution

Nobody said it was. In fact, I suggested the exact opposite - a proper solution takes dev effort. Adhering to an iron rule of "just make them manually retry" is throwing your hands up and not even trying to solve the problem because laziness is convenient.

There's a good amount of literature about this (check the other comments), but you can vastly simplify this into two things you need to do:

1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.

2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.

Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.

2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.

Ooh, I like the idea of propagating "no retries" hints in the responses back upstream. Have you seen it implemented in the wild, or in public discussions about the practice?

I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point