Retry storm: jab ek chhoti si retry ne poora system gira diya
Retry storm tab hota hai jab ek service slow ya down hoti hai aur uske clients turant, baar baar retry karte hain. Har layer ki retries multiply hoti hain: AWS Builders' Library ke example mein 5 layers aur har layer par 3 retries se database par load 243 guna ho jaata hai. Google SRE book isse cascading failure kehti hai, ek failure jo positive feedback se badhta jaata hai. Teen standard fix hain: exponential backoff with jitter (retries ko time mein phaila do), circuit breaker (gire hue service ko call karna kuch der band), aur retry budget (har request ki retries limit, aur retry sirf ek layer par). Neeche wala retry.mjs teeno fix ek file mein deta hai, aur humne 9 Nov 2026 ko node par iske tests chalaye.
Links
Aaj ye karo
- Pehchano: traffic normal hai par requests per second, latency aur error rate saath mein upar ja rahe hain, aur logs mein timeout ke turant baad retrying dikh raha hai. Ye retry storm ke lakshan hain.
- Apne code aur config mein har retry dhundo: while loop jo fail par turant dobara call kare, retry: Infinity, fixed delay wali retry, aur SDK ya HTTP client ki default retries. Dekho kitni layers retry kar rahi hain.
- Fix #1, exponential backoff with full jitter: har attempt ke baad wait doguna karo aur us wait mein se random time chuno: wait = random(0, min(cap, base x 2^attempt)). Isse saare clients ek hi second par wapas nahi aate.
- Fix #2, circuit breaker: lagataar kuch failures ke baad breaker open ho jaata hai aur calls bina service ko chhue turant fail hoti hain. Cooldown ke baad ek trial call jaati hai (half-open); woh chal gayi toh breaker phir close.
- Fix #3, retry budget: ek request ko maximum 2 retries (kul 3 attempts). Retry sirf ek layer par rakho, har layer par nahi. Google SRE book client-wide budget bhi batati hai: retries kul requests ke 10% se kam.
- Code lo: retry.mjs download karo (link neeche). Ye plain JavaScript hai, Node 18+ ya browser mein chalta hai, koi dependency nahi. Apni call ko callWithRetry(() => fetch(...), breaker) mein wrap karo.
- Test humne kiya: hamesha fail hone wali call exactly 3 baar chali (1 try + 2 retries), 2 fail ke baad chalne wali call ne result diya, 5 failures ke baad breaker ne bina call kiye error diya, aur cooldown ke baad ek trial call se breaker wapas close hua.
- Production mein: retries sirf un errors par karo jo dobara try karne se theek ho sakte hain (timeout, 503), aur jo request do baar chalne par nuksaan kare (jaise payment) usme idempotency key zaroor rakho.
const wait = Math.random() * Math.min(capMs, baseMs * 2 ** attempt); // backoff + full jitterif (!breaker.allow()) throw new Error("circuit open"); // circuit breakerif (attempt >= maxRetries) throw err; // retry budget: max 2curl -O https://howitworksio.com/files/retry.mjsSources
AWS Builders' Library (Marc Brooker), AWS Architecture Blog, Google SRE book ch. 21-22, Martin Fowler; retry.mjs tests 9 Nov 2026 par node se chalaye.
howitworks