Think about what the feature (potentially) buys you:
(1) zero-copy, zero-allocation request processing
(2) up to a 2x latency reduction by intermingling networking and actual work
(3) more cache friendliness
(4) better performance characteristics on composition (multiple stages which all have to batch their requests and responses will balloon perceived latency)
If you have a simple system (only a few layers of networking), low QPS (under a million), small requests (average under 1KB, max under 1MB), and reasonable latency requirements (no human user can tell a microsecond from a millisecond), just batch everything and be done with it. It's not worth the engineering costs to do anything fancy when a mid-tier laptop can run your service with the dumb implementation.
As soon as those features start to matter, streaming starts to make more sense. I normally see it being used for cost reasons in very popular services, when every latency improvement matters for a given application, when you don't have room to buffer the whole request, or to create very complicated networked systems.
(1) zero-copy, zero-allocation request processing
(2) up to a 2x latency reduction by intermingling networking and actual work
(3) more cache friendliness
(4) better performance characteristics on composition (multiple stages which all have to batch their requests and responses will balloon perceived latency)
If you have a simple system (only a few layers of networking), low QPS (under a million), small requests (average under 1KB, max under 1MB), and reasonable latency requirements (no human user can tell a microsecond from a millisecond), just batch everything and be done with it. It's not worth the engineering costs to do anything fancy when a mid-tier laptop can run your service with the dumb implementation.
As soon as those features start to matter, streaming starts to make more sense. I normally see it being used for cost reasons in very popular services, when every latency improvement matters for a given application, when you don't have room to buffer the whole request, or to create very complicated networked systems.