Context

On servant-server, I must serve some data that I query from a store. The persistance interface returns me some Stream of resources, so actual queries to the store are done only when the stream is consumed.

Unsatisfactory solution

Fold the stream into a list and respond with that list in a normal Servant endpoint. This is unsatisfactory because by folding the stream, all resources are read from the persistance layer and accumulated in memory before being sent, which leads me to huge memory usage.

Objective

Pipe the resources directly from the Prelude.Streaming.Stream into a Servant.Types.SourceT, so that the persistance layer is queried only when the resource must be sent in the response stream, and thus avoid to accumulate many resources in memory.

To achieve this, I can generate a SourceIO with a fromAction or fromActionStep, that reads the next element of the stream.

Issue

Since all the actions run by fromAction are actually the same, I cannot consume the stream by accessing the next values.

Solution

Tweak fromActionStep to make it pass a value from an action to the next one:

fromActionStep' :: Functor m => (c -> m (Maybe (a,c))) -> c -> StepT m a
fromActionStep' action = loop where
  loop c = Effect $ step <$> action c
  step Nothing = Stop
  step (Just (x,t)) = Yield x $ loop t

And then call it on Prelude.Streaming.uncons.

Of course, Maybe (a,c) can be generalized to any type v, provided that we pass additionally some stop :: v -> Bool, value :: v -> a and rest :: v -> v functions.

Question

Was there an easier out-of-the-box solution I did not see?

Otherwise, is it worth adding such fromActionStepWithConsumable method to the library?