Python Logging Filters

The python logging package provides a Filter class that can be used for filtering log records. This is a simple way to ensure that a logger or handler will only output desired log messages. Here’s an example filter that only allows INFO messages to be logged:

import logging

class InfoFilter(logging.Filter):
	def filter(self, rec):
		return rec.levelno == logging.INFO

Configuring Python Logging Filters

Filters can be added to a logger instance or a handler instance using the addFilter(filt) method. For a logger, the best time to do this is probably right after calling getLogger, like so:

log = logging.getLogger()
log.addFilter(InfoFilter())

What about adding a filter to a handler? If you’re programmatically configuring handlers with addHandler(hdlr), then you can do the same thing by calling addFilter(filt) on the handler instance. But if you’re using fileConfig to configure handlers and loggers, it’s a little bit harder. Unfortunately, the logging configuration format does not support adding filters. And it’s not always clear which logger the handler instances are attached to in the logger hierarchy. So the simplest way to add a filter to a handler in this case is to subclass the handler:

class InfoHandler(logging.StreamHandler):
	def __init__(self, *args, **kwargs):
		StreamHandler.__init__(self, *args, **kwargs)
		self.addFilter(InfoFilter())

Then in your file config, make sure to set the class value for your custom handler to a complete code path for import:

[handler_infohandler]
class=mypackage.mylogging.InfoHandler
level=INFO

Now your handler will only handle the log records that pass your custom filter. As long your handlers aren’t changing much, the above method is much more reusable than having to call addFilter(filt) everytime a new logger is instantiated.

  • http://www.red-dove.com/ Vinay Sajip

    Your comment about a lack of support for filters in fileConfig is correct, but the new dictConfig API in Python 2.7 and Python 3.2 (see PEP 391) will support Filters in the configuration. Also, it should be clear which loggers handlers are attached to in a configuration: the handlers whose keys are in the handlers: value for a logger are attached to that logger.

  • http://streamhacker.com/ Jacob Perkins

    Thanks, I didn't know about PEP 391 & dictConfig; it looks like a great improvement to fileConfig.

    On the handlers attached to loggers, I was referring to the logger parent hierarchy, where you might have a child logger with no specific handlers itself, but whose parent's have handlers. So then the only programmatic way to find out which logger has the handler you're looking for is thru introspection. But dictConfig should clear that all up.

  • http://www.red-dove.com/ Vinay Sajip

    Jacob Perkins wrote, in response to Vinay Sajip:

  • http://www.red-dove.com/ Vinay Sajip

    Well, it's not always safe to assume there are any specific handlers for a logger at runtime, as the logging package allows for the logging configuration to be changed – sometimes on the fly. Still, when dictConfig lands, that will hopefully take away the need to do any working around missing package functionality.

  • xio

    That was really helpful, thanks! OTL

  • http://blog.tplus1.com Matt Wilson

    I’m working on using filters to send one kind of critical error to one place, and another kind of critical error somewhere else.  So I found your blog article.  Very helpful.  I like the idea of combining a handler with a filter by making a subclassed handler.  That approach keeps things obvious.

    Anyhow, I’m reading your post here, and I see the link to programmatically configuring handlers, and BAM, sure enough, that is my old post about how logging beats print statements.

    This made my day!

    One stupid question that I could answer myself through research: does our beloved logging module automatically set up some internal multi-threaded thing?

    Also, I’m going to do a talk at PyOhio 2012 all about logging. It breaks my heart every time I inherit some project and see a crap ton of print statements (or even worse, files opened to be written to).

  • http://streamhacker.com/ Jacob Perkins

    I’m pretty sure logging is thread-safe, but I’ve never checked to be sure since I avoid threads as much as possible :)
    And thanks for the log vs print article, it saved me the effort of writing it myself.