Python's unittest module ain't that bad

Collin Winter was kind enough to speak to BayPiggies last night about his unittest module replacement, test_harness. The basic premise of the talk is that unittest does not support extensions very well, hence he wrote his own testing framework that did. The same argument is presented in Collin's blog posting titled "Python's unittest module sucks".

However, I have an issue with several of Collin's claimed deficiencies in unittest: they simply aren't there. For example, he claimed that extensions cannot be composed (i.e. multiple extensions cannot be applied to a single test case) easily. I raised the point in the meeting that python's decorators are trivially composable and the TODO annotation described in his blog is trivially implementable as a decorator, usable via unittest, nose, or just about any testing framework. In his presentation, he claimed TODO annotations required over a hundred lines of code across 5 classes to implement using unittest. This simply isn't true: I implemented it in 8 lines while he spoke; adding some polish it's up to 11 plus doc-string, but nowhere near 100 and there isn't a class in sight:

def TODO(func):
    """unittest test method decorator that ignores
       exceptions raised by test
   
    Used to annotate test methods for code that may
    not be written yet.  Ignores failures in the
    annotated test method; fails if the text
    unexpectedly succeeds.
    """
    def wrapper(*args, **kw):
        try:
            func(*args, **kw)
            succeeded = True
        except:
            succeeded = False
        assert succeeded is False, \
               "%s marked TODO but passed" % func.__name__
    wrapper.__name__ = func.__name__
    wrapper.__doc__ = func.__doc__
    return wrapper

Collin demonstrated a platform-specific test annotation in his framework. He claimed this would require almost 200 lines of code to implement in unittest, but that too is an overstatement. I had it implemented before he could finish the slide:

def PlatformSpecific(platformList):
    """unittest test method decorator that only
       runs test method if os.name is in the
       given list of platforms
    """
    def decorator(func):
        import os
        def wrapper(*args, **kw):
            if os.name in platformList:
                return func(*args, **kw)
        wrapper.__name__ = func.__name__
        wrapper.__doc__ = func.__doc__
        return wrapper
    return decorator

The point is that python decorators are a language feature that allow you to trivially wrap any callable with another callable; the latter of which can perform any pre- or post- processing or even avoid calling the decorated function at all. You get transparent composition for free:

class ExampleTestCase(unittest.TestCase):
    @TODO
    def testToDo(self):
        MyModule.DoesNotExistYet('boo')

    @PlatformSpecific(('mac', ))
    def testMacOnly(self):
        MyModule.SomeMacSpecificFunction()

    @TODO
    @PlatformSpecific(('nt', 'ce'))
    def testComposition(self):
        MyModule.PukePukePuke()

(If you aren't familar with decorators in python, IBM has a pretty thorough article on the subject)

For the record, I also implemented a proof-of-concept of Collin's reference counting example in a similarly-succinct decorator. In the example presented at BayPiggies, Collin ran the test case 5 times, checking a reference count after each run. I missed how he was getting references counts (len(gc.get_referrers(...)) maybe?) so you need to fill in how to get your object reference counts:

def CheckReferences(func):
    def wrapper(*args, **kw):
        refCounts = []
        for i in range(5):
            func(*args, **kw)
            refCounts.append(XXXGetRefCount())
        assert min(refCounts) != max(refCounts), \
               "reference counts changed"
    wrapper.__name__ = func.__name__
    wrapper.__doc__ = func.__doc__
    return wrapper

Adding the repetition count as a parameter would be trivial (see the PlatformSpecific decorator above for an example how). I hard-coded 5 repetitions since that is what Collin used in his presentation.

In all, it took me about 5 minutes to write all three decorators and test them (admittedly, I already had unittest test cases to try my decorators on). I didn't tinker a bit nor was there any poking or prodding of unittest; I didn't subclass anything. You can implement these extensions using standard python syntax and the standard pytingn unittest module. To claim otherwise is simply disingenuous.

Of course, Collin's testframework module uses decorators too, so Collin was clearly aware of their existence. Which prompted me to question Collin's claims of 100+ lines to implement these features using unittest when simple decorators are sufficient. His response was that his numbers were the number of lines of code that would be necessary to implement the TODO and platform-specific annotations using the unittest module without decorators. Which seemed inconsistent with his examples, involving decorators, of how easy it is to use these annotations with test_harness. I wanted to ask him about this contradiction face-to-face after the presentation, but unfortunately he had to catch the Google shuttle home immediately after his talk.

One point that Collin did repeatedly come back to was that logging extensions cannot be implemented using decorators. For example, you cannot have the unittest module log the test run results to a database by wrapping test methods in decorators. In theory you just need to implement your own TestRunner and TestResult subclasses and pass the TestRunner subclass to unittest.TestProgram(). However, if Sebastian Rittau's XML logger TestRunner class for unittest is any indication, changing loggers is non-trivial.

Collin said in his presentation, and I would have to agree, extending unittest logging is painful; composing multiple loggers is prohibitively painful. Of course, if more TestRunner implementations were included in the python standard library, half of this argument would be moot as there would be less need to extend. Right now, only a text logger TestRunner is included.

But to be honest, I don't expect most people really need to replace the logging mechanism (which may be why the standard library doesn't include more loggers). Marking tests as TODO or platform-specific or whatever is pretty universal; recording test run results to a database for analysis is probably far outside the realm of what most people (or even companies outside the Fortune 500 for that matter) need from their test framework. Which may be more of a comment on the sad state of testing than anything, but I digress. In any event, Collin's re-implementation adds value by facilitating logger composition, but to say that not facilitating logger composition makes unittest "suck" seems like a gross overstatement to me.

In all, I left BayPiggies last night having thought a lot more about unittest than I ever have before. And I can't help but think that, for the vast majority of us python hackers down in the trenchs, python's unittest ain't that bad.

Update 2007/06/15 10:05am:

I found the lines-of-code numbers quoted in Collin's presentation in his blog also. My memory was pretty close on the supposed ~100 lines to implement TODO annotations. But it looks like I may have confused his ~200 lines quote for implementing composition of TODO and reference counting with the supposed number of lines to implement platform-specific test annotations. To be clear, though, composition using decorators as I described above requires 0 core classes and 2 lines of code (see my testComposition example above).

Keyword arguments in XML-RPC

This isn't the least bit novel, for example I know I've been using this trick for years, but nonetheless here is a way to simulate named arguments in XML-RPC. XML-RPC only natively supports positional parameters, but by passing a single positional argument that is itself an XML-RPC struct (which is actually a mapping), you can simulate named and/or optional arguments. Rather than reproduce a sample XML-RPC document demonstrating this usage, I'll refer you to one of my earlier posts that utilized this technique; you'll see that the method is called with two named parameters: path and args.

If you are familiar with perl, you may also be aware of the trick perl 5, which also only natively supports positional arguments, uses to simulate named parameters. In perl 5, it is common to pass a hash of name/value pairs as arguments. However, what perl actually does under the scenes, and which is different from this XML-RPC trick, is to serialize the hash into an array of alternating names and values; it then passes this array as the positional argument list for the subroutine being called. The called subroutine then de-serializes the name and value pairs from the argument array, reconstructing the original hash. This flattening of a hash has to be a documented protocol between the subroutine and its callers.

Of course, you could do exactly the same thing using XML-RPC: serialize the argument dictionary into an array of alternating names and values and populate the method's param list with this array's elements. The XML-RPC server method could then reconstruct the original dictionary from the param list.

But XML-RPC also supports passing dictionaries and dictionaries: using the struct data type. Hence my original suggestion. Since to support named (or generic optional arguments) we have to document a protocol between the caller and the method, we might as well make the protocol as straightforward as possible. Rather than serialize and deserialize a dictionary of named arguments, just pass the dictionary as-is, as the one and only positional argument.

C: Converting struct tm times with timezone to time_t

Both the BSD and GNU standard C library have extended the struct tm to include a tm_gmtoff member that holds the offset from UTC of the time represented by the structure. Which might lead you to believe that mktime(3) would honor the time offset indicated by tm_gmtoff when converting to a time_t representation.

Nope.

mktime(3) always assumes the "current timezone" defined by the executing environment. Since ISO C and POSIX define the semantics for mktime(3) but neither defines a tm_gmtoff member for the tm structure, not surprisingly mktime(3) does not honor it.

So, lets say you have a struct tm, complete with correctly-populated tm_gmtoff field: how do you convert it to a time_t representation?

Many modern C libraries (including glibc and FreeBSD's libc) include a timegm(3) function. No, this function doesn't honor tm_gmtoff either. Instead, gmtime(3) converts the struct tm to a time_t just like mktime(3), but ignores the timezone of the executing environment and always assumes GMT as the timezone.

However, if your libc implements both tm_gmtoff and timegm(3) you are in luck. You just need to use timegm(3) to get the time_t representing the time in GMT and then subtract the offset stored in tm_gmtoff. The tricky part is that calling timegm(3) will modify the struct tm, clearing the tm_gmtoff field to zero (at least it does on the FreeBSD 4.10 machine I'm testing with). Combined with C's lack of guaranteed left-to-right evaluation, you need to save the tm_gmtoff so it doesn't get clobbered before you can use it. Something like:

time_t
tm2time(const struct tm *src)
{
       struct tm tmp;

       tmp = *src;
       return timegm(&tmp) - src->tm_gmtoff;
}

Note that I copy the entire struct tm into a temporary variable. This prevents timegm(3) from clobbering the tm_gmtoff so that we can use it to accurately compute the seconds since the epoch. The copy in tmp gets clobbered, but the copy in src is left intact. Also, by copying the src struct tm into a temporary, we never modify the argument passed in -- which is just a generally friendly thing to do.

All that said, the truly pedantic will point out that neither ISO C nor POSIX specs dictate that time_t must represents seconds. However, since we are already depending on two non-standard extensions, it seems reasonable to also depend on the fact that systems implementing timegm(3) and the tm_gmtoff field all implement time_t values in seconds.

NTTMCL is hiring

NTTMCL is looking for software engineers (but then, who isn't right now?). We are a small research-and-development subsidiary of NTT Communications of Japan. I won't rehash our company profile because it is all on our web site.

We use and develop a lot of different technologies, so while the current positions are focused on wireless, encryption, and VoIP, one of the great things about working at NTTMCL is that you have the chance to work on many different projects beyond those you were originally hired for. I will add that we are far more focused on development than research, with the proportion of full-time engineers to full-time researchers somewhere around 5:1.

The positions ask for C, C++, or Java experience, but we also have at least as much Python and Perl code in use on various projects. Which is to say that being flexible is quite an asset here at NTTMCL. Just knowing/using languages isn't the point: the point is having a large tool chest to pull from and being able to identify which tool is the best tool for a job. Being a R&D company, management is relatively open to trying new tools/languages/etc. if you can justify the choice by explaining how it gets the job done better than the alternatives.

Speaking of which, I've been meaning to write up a post about how much I like my job, but it turns out my co-worker Zach has already beaten me to it. Hopefully, I'll still get around to writing up my own thoughts based on my 5+ years at NTTMCL sometime soon.

XML-RPC patented

Ever being the astute one, I just now discovered that webMethods was awarded a patent for XML-RPC back in April of last year. They don't seem to be cracking down yet, but could XML-RPC be the next GIF/LZW controversy?

It would be hard to compose an unencumbered alternative to XML-RPC when the first claim of the patent reads:

A method of communicating between first and second machines, said method comprising the steps of: generating a message at a first machine including at least one argument and a type label for said argument; and transmitting said message from said first machine.

Since S.O.A.P was developed by Microsoft it goes without saying but that is patented too. I guess I need to start converting my XML-RPC clients and servers to JSON-RPC to be on the safe side.

Python: islambda()

The Python inspect module provides functions that determine whether objects are methods, functions, classes, modules, etc. However, there is no method that tells you whether something is a lambda expression. The isfunction() is probably close enough for most applications, but believe it or not, I recently encountered a case where it made sense to issue a warning if an argument was a lambda expression.

Here is a python function that determines whether or not its argument is a lambda expression:

import inspect
def islambda(f):
    return inspect.isfunction(f) and \
           f.__name__ == (lambda: True).__name__

Currently, the __name__ of anonymous functions created by lambda is "<lambda>", so I could have just hard-coded that string into the comparison. But I chose to use the (lambda: True).__name__ expression instead just in case python uses a different name for lambda expressions in the future.

This function works as expected in all common cases:

>>> islambda(lambda: 1)
True
>>> islambda(islambda)
False
>>> islambda(globals)
False
>>> islambda(str)
False
>>> islambda(str.join)
False
>>> islambda("".join)
False

The only case that I am aware of where it will not work is if you carefully craft a function with the name "<lambda>":

import new
>>> x = new.function(
            compile("print 'Hello World!'", "<string>", "exec"),
            {}, '<lambda>')
>>> islambda(x)
True

Consider yourself warned. :)

In case you are curious, the application I was working on had a method that took a callable as an argument and held a weak reference to it. I would have loved to have been able to issue a warning anytime a callable was passed that would "immediately" be garbage collected before anything useful was done, but that is a non-trivial condition to detect (hint: it involves reading the programmer's mind). But there is a common subset of that error case that is relatively easy to detect: callers passing their only reference to a lambda expression. That case can be trivially detected using the islambda() function described above along with the sys.getrefcount() function like so:

import sys
from warnings import warn
...
def myfunc(f):
    if sys.getrefcount(f) == 3 and islambda(f):
        warn('f is too short-lived to be useful', stacklevel=2)
    ...

Since it is not obvious, I should point out that (in this example) a reference count of 3 indicates that myfunc()'s caller holds no references to the callable f. The reason is that sys.getrefcount() will hold one reference, the name f is bound to one reference, and there is a temporary reference held by the python interpreter across the call to myfunc(), so if sys.getrefcount() returns 3, we know those are the only three references.

Incidentally, the fact that islambda() erroniously identifies a function with the same "<lambda>" as a lambda expression is inconsequential for my stated purpose: if the crafted function has no other references, I want to issue a warning just the same as if it had truly been a lambda expression.

Which brings me back to isfunction(). It turns out, not surprisingly, that isfunction() is sufficient for my needs since a function with only 3 references has, by definition, no external references. In the end, I didn't actually use my islambda() function and went with isfunction() for my application instead:

import inspect
import sys
from warnings import warn
...
def myfunc(f):
    if sys.getrefcount(f) == 3 and inspect.isfunction(f):
        warn('f is too short-lived to be useful', stacklevel=2)
    ...

This handles both lambda expressions and functions dynamically created using the new module.

Python Pitfall: Not all objects are created equal

I've been entertaining the idea of writing a series of posts about Python warts for a couple of weeks now. Overall, python is a remarkably consistent programming language, but there are a few edge cases that people should be aware of. My hope is that, by pointing out their existence, others can save themselves a rude surprise. I've decided to call the series "python pitfalls".

So here goes my inaugural post: not all objects are created equal in python. New-style classes (which are only "new" in the sense they were introduced in python 2.2 which is quite old now) all inherit from the base object class. For example, consider the following simple class:

>>> class MyObject(object):
...     pass

This is about as simple a class as you can make in Python; MyObject inherits all of its behaviour from the object base class. Now, lets set an attribute on an instance of our new class:

>>> b = MyObject()
>>> b.myattr = 42
>>> print b.myattr
42

Nothing fancy here. But recall that our MyObject class adds nothing to the base object class, implying that the ability to set arbitrary attributes on an instance must originate with the object class's implementation. Let's give it a try:

>>> a = object()
>>> a.myattr = 42
Traceback (most recent call last):
File "", line 1, in 
AttributeError: 'object' object has no attribute 'myattr'
>>> setattr(a, 'myattr', 42)
Traceback (most recent call last):
File "", line 1, in 
AttributeError: 'object' object has no attribute 'myattr'

What is going on here? We were able to set attributes on instances of MyObject, but not on instances of object itself? That's odd: MyObject inherits all of its behaviour from object, so they should be exactly the same!

My first guess was that the object class had a __slots__ attribute restricting which attributes could be set on it (see this article for an explanation of __slots__). One of the properties of __slots__ is that, unlike most other class attributes, it is not inherited by subclasses. Which would explain why we can set arbitrary attributes on instances of MyObject, which is a subclass of object, but not on instances of object itself. However, to my surprise, object does not define a __slots__ attribute:

>>> '__slots__' in dir(object)
False
>>> dir(object)
['__class__', '__delattr__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__']

Look: no __slots__! So that isn't it.

As far as I can tell, the fact instances of object do not allow attributes to be set on them is simply an implementation artifact. They should, but they don't. Go figure. Luckily, there is seldom need to create instances of object directly; the class really just exists as a base class for deriving new-style classes. But I do still find it odd somehow subclassing object adds functionality not present in the base class.