Tag Archives: filter

python

Python Logging Filters

7 Comments

The python logging package provides a Filter class that can be used for filtering log records. This is a simple way to ensure that a logger or handler will only output desired log messages. Here’s an example filter that only allows INFO messages to be logged:

import logging

class InfoFilter(logging.Filter):
	def filter(self, rec):
		return rec.levelno == logging.INFO

Configuring Python Logging Filters

Filters can be added to a logger instance or a handler instance using the addFilter(filt) method. For a logger, the best time to do this is probably right after calling getLogger, like so:

log = logging.getLogger()
log.addFilter(InfoFilter())

What about adding a filter to a handler? If you’re programmatically configuring handlers with addHandler(hdlr), then you can do the same thing by calling addFilter(filt) on the handler instance. But if you’re using fileConfig to configure handlers and loggers, it’s a little bit harder. Unfortunately, the logging configuration format does not support adding filters. And it’s not always clear which logger the handler instances are attached to in the logger hierarchy. So the simplest way to add a filter to a handler in this case is to subclass the handler:

class InfoHandler(logging.StreamHandler):
	def __init__(self, *args, **kwargs):
		StreamHandler.__init__(self, *args, **kwargs)
		self.addFilter(InfoFilter())

Then in your file config, make sure to set the class value for your custom handler to a complete code path for import:

[handler_infohandler]
class=mypackage.mylogging.InfoHandler
level=INFO

Now your handler will only handle the log records that pass your custom filter. As long your handlers aren’t changing much, the above method is much more reusable than having to call addFilter(filt) everytime a new logger is instantiated.

erlang, programming

mapfilter

6 Comments

Have you ever mapped a list, then filtered it? Or filtered first, then mapped? Why not do it all in one pass with mapfilter?

mapfilter?

mapfilter is a function that combines the traditional map & filter of functional programming by using the following logic:

  1. if your function returns false, then the element is discarded
  2. any other return value is mapped into the list

Why?

Doing a map and then a filter is O(2N), whereas mapfilter is O(N). That’s twice as a fast! If you are dealing with large lists, this can be a huge time saver. And for the case where a large list contains small IDs for looking up a larger data structure, then using mapfilter can result in half the number of database lookups.

Obviously, mapfilter won’t work if you want to produce a list of boolean values, as it would filter out all the false values. But why would you want to map to a list booleans?

Erlang Code

Here’s some erlang code I’ve been using for a while:

mapfilter(F, List) -> lists:reverse(mapfilter(F, List, [])).

mapfilter(_, [], Results) ->
        Results;
mapfilter(F, [Item | Rest], Results) ->
        case F(Item) of
                false -> mapfilter(F, Rest, Results);
                Term -> mapfilter(F, Rest, [Term | Results])
        end.

Has anyone else done this for themselves? Does mapfilter exist in any programming language? If so, please leave a comment. I think mapfilter is a very simple & useful concept that should be a included in the standard library of every (functional) programming language. Erlang already has mapfoldl (map-reduce in one pass), so why not also have mapfilter?