functools – Tools for Manipulating Functions

Purpose:Functions that operate on other functions.
Available In:2.5 and later

The functools module provides tools for working with functions and other callable objects, to adapt or extend them for new purposes without completely rewriting them.

Decorators

The primary tool supplied by the functools module is the class partial, which can be used to “wrap” a callable object with default arguments. The resulting object is itself callable, and can be treated as though it is the original function. It takes all of the same arguments as the original, and can be invoked with extra positional or named arguments as well.

partial

This example shows two simple partial objects for the function myfunc(). Notice that show_details() prints the func, args, and keywords attributes of the partial object.

import functools

def myfunc(a, b=2):
    """Docstring for myfunc()."""
    print '\tcalled myfunc with:', (a, b)
    return

def show_details(name, f, is_partial=False):
    """Show details of a callable object."""
    print '%s:' % name
    print '\tobject:', f
    if not is_partial:
        print '\t__name__:', f.__name__
    print '\t__doc__', repr(f.__doc__)
    if is_partial:
        print '\tfunc:', f.func
        print '\targs:', f.args
        print '\tkeywords:', f.keywords
    return

show_details('myfunc', myfunc)
myfunc('a', 3)
print

p1 = functools.partial(myfunc, b=4)
show_details('partial with named default', p1, True)
p1('default a')
p1('override b', b=5)
print

p2 = functools.partial(myfunc, 'default a', b=99)
show_details('partial with defaults', p2, True)
p2()
p2(b='override b')
print

print 'Insufficient arguments:'
p1()

At the end of the example, the first partial created is invoked without passing a value for a, causing an exception.

$ python functools_partial.py

myfunc:
        object: <function myfunc at 0x100468c08>
        __name__: myfunc
        __doc__ 'Docstring for myfunc().'
        called myfunc with: ('a', 3)

partial with named default:
        object: <functools.partial object at 0x10046b050>
        __doc__ 'partial(func, *args, **keywords) - new function with partial
 application\n    of the given arguments and keywords.\n'
        func: <function myfunc at 0x100468c08>
        args: ()
        keywords: {'b': 4}
        called myfunc with: ('default a', 4)
        called myfunc with: ('override b', 5)

partial with defaults:
        object: <functools.partial object at 0x10046b0a8>
        __doc__ 'partial(func, *args, **keywords) - new function with partial
 application\n    of the given arguments and keywords.\n'
        func: <function myfunc at 0x100468c08>
        args: ('default a',)
        keywords: {'b': 99}
        called myfunc with: ('default a', 99)
        called myfunc with: ('default a', 'override b')

Insufficient arguments:
Traceback (most recent call last):
  File "functools_partial.py", line 49, in <module>
    p1()
TypeError: myfunc() takes at least 1 argument (1 given)

update_wrapper

The partial object does not have __name__ or __doc__ attributes by default, and without those attributes decorated functions are more difficult to debug. Using update_wrapper(), copies or adds attributes from the original function to the partial object.

import functools

def myfunc(a, b=2):
    """Docstring for myfunc()."""
    print '\tcalled myfunc with:', (a, b)
    return

def show_details(name, f):
    """Show details of a callable object."""
    print '%s:' % name
    print '\tobject:', f
    print '\t__name__:', 
    try:
        print f.__name__
    except AttributeError:
        print '(no __name__)'
    print '\t__doc__', repr(f.__doc__)
    print
    return

show_details('myfunc', myfunc)

p1 = functools.partial(myfunc, b=4)
show_details('raw wrapper', p1)

print 'Updating wrapper:'
print '\tassign:', functools.WRAPPER_ASSIGNMENTS
print '\tupdate:', functools.WRAPPER_UPDATES
print

functools.update_wrapper(p1, myfunc)
show_details('updated wrapper', p1)

The attributes added to the wrapper are defined in functools.WRAPPER_ASSIGNMENTS, while functools.WRAPPER_UPDATES lists values to be modified.

$ python functools_update_wrapper.py

myfunc:
        object: <function myfunc at 0x100468c80>
        __name__: myfunc
        __doc__ 'Docstring for myfunc().'

raw wrapper:
        object: <functools.partial object at 0x10046c0a8>
        __name__: (no __name__)
        __doc__ 'partial(func, *args, **keywords) - new function with partial
 application\n    of the given arguments and keywords.\n'

Updating wrapper:
        assign: ('__module__', '__name__', '__doc__')
        update: ('__dict__',)

updated wrapper:
        object: <functools.partial object at 0x10046c0a8>
        __name__: myfunc
        __doc__ 'Docstring for myfunc().'

Other Callables

Partials work with any callable object, not just standalone functions.

import functools

class MyClass(object):
    """Demonstration class for functools"""
    
    def meth1(self, a, b=2):
        """Docstring for meth1()."""
        print '\tcalled meth1 with:', (self, a, b)
        return
    
    def meth2(self, c, d=5):
        """Docstring for meth2"""
        print '\tcalled meth2 with:', (self, c, d)
        return
    wrapped_meth2 = functools.partial(meth2, 'wrapped c')
    functools.update_wrapper(wrapped_meth2, meth2)
    
    def __call__(self, e, f=6):
        """Docstring for MyClass.__call__"""
        print '\tcalled object with:', (self, e, f)
        return

def show_details(name, f):
    """Show details of a callable object."""
    print '%s:' % name
    print '\tobject:', f
    print '\t__name__:', 
    try:
        print f.__name__
    except AttributeError:
        print '(no __name__)'
    print '\t__doc__', repr(f.__doc__)
    return
    
o = MyClass()

show_details('meth1 straight', o.meth1)
o.meth1('no default for a', b=3)
print

p1 = functools.partial(o.meth1, b=4)
functools.update_wrapper(p1, o.meth1)
show_details('meth1 wrapper', p1)
p1('a goes here')
print

show_details('meth2', o.meth2)
o.meth2('no default for c', d=6)
print

show_details('wrapped meth2', o.wrapped_meth2)
o.wrapped_meth2('no default for c', d=6)
print

show_details('instance', o)
o('no default for e')
print

p2 = functools.partial(o, f=7)
show_details('instance wrapper', p2)
p2('e goes here')

This example creates partials from an instance, and methods of an instance.

$ python functools_method.py

meth1 straight:
        object: <bound method MyClass.meth1 of <__main__.MyClass object at
0x10046a3d0>>
        __name__: meth1
        __doc__ 'Docstring for meth1().'
        called meth1 with: (<__main__.MyClass object at 0x10046a3d0>, 'no d
efault for a', 3)

meth1 wrapper:
        object: <functools.partial object at 0x10046c158>
        __name__: meth1
        __doc__ 'Docstring for meth1().'
        called meth1 with: (<__main__.MyClass object at 0x10046a3d0>, 'a go
es here', 4)

meth2:
        object: <bound method MyClass.meth2 of <__main__.MyClass object at
0x10046a3d0>>
        __name__: meth2
        __doc__ 'Docstring for meth2'
        called meth2 with: (<__main__.MyClass object at 0x10046a3d0>, 'no d
efault for c', 6)

wrapped meth2:
        object: <functools.partial object at 0x10046c0a8>
        __name__: meth2
        __doc__ 'Docstring for meth2'
        called meth2 with: ('wrapped c', 'no default for c', 6)

instance:
        object: <__main__.MyClass object at 0x10046a3d0>
        __name__: (no __name__)
        __doc__ 'Demonstration class for functools'
        called object with: (<__main__.MyClass object at 0x10046a3d0>, 'no
default for e', 6)

instance wrapper:
        object: <functools.partial object at 0x10046c1b0>
        __name__: (no __name__)
        __doc__ 'partial(func, *args, **keywords) - new function with parti
al application\n    of the given arguments and keywords.\n'
        called object with: (<__main__.MyClass object at 0x10046a3d0>, 'e g
oes here', 7)

wraps

Updating the properties of a wrapped callable is especially useful when used in a decorator, since the transformed function ends up with properties of the original, “bare”, function.

import functools

def show_details(name, f):
    """Show details of a callable object."""
    print '%s:' % name
    print '\tobject:', f
    print '\t__name__:', 
    try:
        print f.__name__
    except AttributeError:
        print '(no __name__)'
    print '\t__doc__', repr(f.__doc__)
    print
    return

def simple_decorator(f):
    @functools.wraps(f)
    def decorated(a='decorated defaults', b=1):
        print '\tdecorated:', (a, b)
        print '\t',
        f(a, b=b)
        return
    return decorated

def myfunc(a, b=2):
    print '\tmyfunc:', (a,b)
    return

show_details('myfunc', myfunc)
myfunc('unwrapped, default b')
myfunc('unwrapped, passing b', 3)
print

wrapped_myfunc = simple_decorator(myfunc)
show_details('wrapped_myfunc', wrapped_myfunc)
wrapped_myfunc()
wrapped_myfunc('args to decorated', 4)

functools provides a decorator, wraps(), which applies update_wrapper() to the decorated function.

$ python functools_wraps.py

myfunc:
        object: <function myfunc at 0x10046c050>
        __name__: myfunc
        __doc__ None

        myfunc: ('unwrapped, default b', 2)
        myfunc: ('unwrapped, passing b', 3)

wrapped_myfunc:
        object: <function myfunc at 0x10046c0c8>
        __name__: myfunc
        __doc__ None

        decorated: ('decorated defaults', 1)
                myfunc: ('decorated defaults', 1)
        decorated: ('args to decorated', 4)
                myfunc: ('args to decorated', 4)

Comparison

Under Python 2, classes can define a __cmp__() method that returns -1, 0, or 1 based on whether the object is less than, equal to, or greater than the item being compared. Python 2.1 introduces the rich comparison methods API, __lt__(), __le__(), __eq__(), __ne__(), __gt__(), and __ge__(), which perform a single comparison operation and return a boolean value. Python 3 deprecated __cmp__() in favor of these new methods, so functools provides tools to make it easier to write Python 2 classes that comply with the new comparison requirements in Python 3.

Rich Comparison

The rich comparison API is designed to allow classes with complex comparisons to implement each test in the most efficient way possible. However, for classes where comparison is relatively simple, there is no point in manually creating each of the rich comparison methods. The total_ordering() class decorator takes a class that provides some of the methods, and adds the rest of them.

import functools
import inspect
from pprint import pprint

@functools.total_ordering
class MyObject(object):
    def __init__(self, val):
        self.val = val
    def __eq__(self, other):
        print '  testing __eq__(%s, %s)' % (self.val, other.val)
        return self.val == other.val
    def __gt__(self, other):
        print '  testing __gt__(%s, %s)' % (self.val, other.val)
        return self.val > other.val

print 'Methods:\n'
pprint(inspect.getmembers(MyObject, inspect.ismethod))

a = MyObject(1)
b = MyObject(2)

print '\nComparisons:'
for expr in [ 'a < b', 'a <= b', 'a == b', 'a >= b', 'a > b' ]:
    print '\n%-6s:' % expr
    result = eval(expr)
    print '  result of %s: %s' % (expr, result)

The class must provide an implmentation of __eq__() and any one of the other rich comparison methods. The decorator adds implementations of the other methods that work by using the comparisons provided.

$ python functools_total_ordering.py

Methods:

[('__eq__', <unbound method MyObject.__eq__>),
 ('__ge__', <unbound method MyObject.__ge__>),
 ('__gt__', <unbound method MyObject.__gt__>),
 ('__init__', <unbound method MyObject.__init__>),
 ('__le__', <unbound method MyObject.__le__>),
 ('__lt__', <unbound method MyObject.__lt__>)]

Comparisons:

a < b :
  testing __gt__(1, 2)
  testing __eq__(1, 2)
  result of a < b: True

a <= b:
  testing __gt__(1, 2)
  result of a <= b: True

a == b:
  testing __eq__(1, 2)
  result of a == b: False

a >= b:
  testing __gt__(1, 2)
  testing __eq__(1, 2)
  result of a >= b: False

a > b :
  testing __gt__(1, 2)
  result of a > b: False

Collation Order

Since old-style comparison functions are deprecated in Python 3, the cmp argument to functions like sort() are also no longer supported. Python 2 programs that use comparison functions can use cmp_to_key() to convert them to a function that returns a collation key, which is used to determine the position in the final sequence.

import functools

class MyObject(object):
    def __init__(self, val):
        self.val = val
    def __str__(self):
        return 'MyObject(%s)' % self.val

def compare_obj(a, b):
    """Old-style comparison function.
    """
    print 'comparing %s and %s' % (a, b)
    return cmp(a.val, b.val)

# Make a key function using cmp_to_key()
get_key = functools.cmp_to_key(compare_obj)

def get_key_wrapper(o):
    """Wrapper function for get_key to allow for print statements.
    """
    new_key = get_key(o)
    print 'key_wrapper(%s) -> %s' % (o, new_key)
    return new_key
        
objs = [ MyObject(x) for x in xrange(5, 0, -1) ]

for o in sorted(objs, key=get_key_wrapper):
    print o
    

Note

Normally cmp_to_key() would be used directly, but in this example an extra wrapper function is introduced to print out more information as the key function is being called.

The output shows that sorted() starts by calling get_key_wrapper() for each item in the sequence to produce a key. The keys returned by cmp_to_key() are instances of a class defined in functools that implements the rich comparison API based on the return value of the provided old-style comparison function. After all of the keys are created, the sequence is sorted by comparing the keys.

$ python functools_cmp_to_key.py

key_wrapper(MyObject(5)) -> <functools.K object at 0x100466558>
key_wrapper(MyObject(4)) -> <functools.K object at 0x100466590>
key_wrapper(MyObject(3)) -> <functools.K object at 0x1004665c8>
key_wrapper(MyObject(2)) -> <functools.K object at 0x100466600>
key_wrapper(MyObject(1)) -> <functools.K object at 0x100466638>
comparing MyObject(4) and MyObject(5)
comparing MyObject(3) and MyObject(4)
comparing MyObject(2) and MyObject(3)
comparing MyObject(1) and MyObject(2)
MyObject(1)
MyObject(2)
MyObject(3)
MyObject(4)
MyObject(5)

See also

functools
The standard library documentation for this module.
Rich comparison methods
Description of the rich comparison methods from the Python Reference Guide.