How to configure Python structlog to work with Dagster?

[jimmyzzxhlh]

I'm using structlog in my code for logging information, e.g.

logger = structlog.get_logger(__name__)

And I'm trying to configure a dagster.yaml in $DAGSTER_HOME as follows (my modules are all under datapipeline, so the above __name__ will print something like datapipeline.<submodule>...

python_logs:
  managed_python_loggers:
    - datapipeline

But I'm only seeing the logs being emitted in stdout/stderr in Dagster, and more importantly, the logs are not streamed in real-time (I can only view the logs after the run is finished)

Appreciate any help!

[OwenKephart]

Hi @jimmyzzxhlh ! The dagster-native log capturing feature relies on hooking into the standard logging module's logging system, which structlog actually does not interact with by default. However, you can configure structlog to do so with:

structlog.configure(logger_factory=structlog.stdlib.LoggerFactory())

This will force structlog to create an actual logging.Logger class (which the structlog logger will then wrap), which enables dagster to detect those logs.

That being said, it's often easier to just rely on stderr/stdout log capture (rather than emitting logs into the dagster event stream) if you expect a high volume of logs.

[jimmyzzxhlh]

Got it, thanks! We do have a bunch logs emitted so you might be right that using stderr/stdout could be the best for now. That being said, the problem is the logs are not streaming live in stdout/stderr.