ClassroomThe classroom
Back to the lesson libraryMECHANISM · 12 MIN
M33.5 CONNECT THE MECHANISM

Build a reliable request path around local or remote inference

On a snowy Monday the model vendor slows to 20-second replies, and the chat widget just spins. Design the timeouts, fallbacks, and limits that keep visitors answered anyway.

LESSON OVERVIEW12 min lesson

Lesson overview

On a snowy Monday the model vendor slows to 20-second replies, and the chat widget just spins. Design the timeouts, fallbacks, and limits that keep visitors answered anyway.

What you’ll explore

  • Design the service around a model call, with a response contract, a timeout budget, a fallback ladder, rate limits, correct cache keys, and idempotent retries, so the application behaves well when the model is slow, busy, or wrong.
GO TO THE SOURCE

Original explanations, connected to the research.

Google SRE Book, "Addressing Cascading Failures"Google SRE Book, "Handling Overload"RFC 6585, which defines HTTP status 429 Too Many RequestsIdempotent requests (Stripe API reference)Hidden Technical Debt in Machine Learning Systems (Sculley et al., NeurIPS 2015)
Suggest a correction

A precise note can make an explanation better.

Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.

The file includes this note, the scene title, and lesson metadata. Your saved progress and quiz responses are excluded. Download before leaving or reloading to keep your note.