Debugging Python Windows Services

Mark Hammond has a simple example of writing Windows Services in Python using the win32serviceutil.ServiceFramework class in his book Python Programming on Win32.

The problem is, that if you use a breakpoint in your service code and try to debug it, you'll find that it never stops at your breakpoint. The reason is that the win32serviceutil module will actually run PythonService.exe and have that program run your python script. Since PythonService.exe is its own process that has no knowledge of your IDE's breakpoints, it just runs without breaking.

Here is a simple trick for the small intersection of people who:

1. Write Windows Services
   
2. in Python
   
3. using an IDE (I use pyDev)
   
4. and need to step through their code to debug.

It turns out that win32serviceutil has a reimplementation of the logic in PythonService.exe that it uses to emulate PythonService.exe's debug mode when you convert your python script you an executable (e.g. via py2exe). The implementation is in win32serviceutil's DebugService() function. The trick to being able to set breakpoints and debug your python service is to convince win32serviceutil into calling DebugService rather than spawning PythonService.exe.

Luckily, this is trivially easy: just add the line
sys.frozen = 'windows_exe' # Fake py2exe so we can debug
before you call
win32serviceutil.HandleCommandLine(...).

Then, you just need to pass the '-debug' command-line argument when you run your service to force it into debugging mode. Your debugger should then control the process so you can debug it.

RFC868 UDP Time Protocol Client

Here is an RFC 868 UDP Time Protocol Client implementation in less than 30 lines of python.
I know...just what you always wanted. It is a long story, but I needed a time protocol client (that would run on Windows) for testing a service we're developing at work. There are tons of implementations of the TCP version of the protocol, but I couldn't find a UDP implementation for Windows to save my life. Python to the rescue.

from socket import *
from struct import unpack
from time import ctime, sleep
from sys import argv

argv = argv[1:]
if len(argv) == 0:
   argv = [ 'time-nw.nist.gov' ]

s = socket(AF_INET, SOCK_DGRAM)
s.settimeout(5.0)

for server in argv:
   print server, ":",
   try:
       s.sendto('', 0, (server, 37))
       t = long(unpack('!L', s.recv(16)[:4])[0])
       # Convert from 1900/01/01 epoch to 1970/01/01 epoch
       t -= 2208988800
       print ctime(t)
   except timeout:
       print "TIMEOUT"
   except:
       print "ERROR"

s.close()
sleep(2)

In case you are curious, the sleep(2) at the end is there so our testers can simply click the icon on their desktop and have time to actually see the results before Windows closes the console window.

By default, it queries the time-nw.nist.gov server, but you can specify any number of servers to query on the command-line.

Useless geolocation information?

I just ran across the Windows' geographical location information API hidden amongst their national language support functions. One thing that caught my eye was that you can get latitude and longitude coordinates via the API. For example, in python:

>>> from ctypes import *
>>> nation = windll.kernel32.GetUserGeoID(16)
>>> buf = create_string_buffer(20)
>>> GEO_LATITUDE=0x0002
>>> GEO_LONGITUDE=0x0003
>>> windll.kernel32.GetGeoInfoA(nation, GEO_LATITUDE,
...                             byref(buf), sizeof(buf), 0)
6
>>> buf.value
'39.45'
>>> windll.kernel32.GetGeoInfoA(nation, GEO_LONGITUDE,
...                             byref(buf), sizeof(buf), 0)
8
>>> buf.value
'-98.908'

At first glance, this looks pretty cool. You can get geospacial coordinates on any Windows XP or Vista box. Except that the API only returns the latitude and longitude of the country selected in the "Regional and Language Options" control panel. That is: the coordinates returned are for the user's selected location (i.e. country).

Sure enough, plugging the returned latitude and longitude into Yahoo Maps, I find that the coordinates fall squarely in the middle of the country. A good thousand miles from where I sit in Palo Alto, California. Of course, being that I don't have a GPS receiver attached to my laptop, I didn't really expect the coordinates to be accurate.

For the life of me, I can't think of a single application of this information. But there it is: someone at Microsoft had to populate the database mapping countries to geospacial coordinates and some engineer had to code the API to retrieve them. For what purpose? What can you do with the latitude and longitude of a country (whatever that is supposed to mean)? Can anyone think of a use for this data?

PowerPoint centimeters

Me, I'm 5'10" tall. That would be approximately 178 centimeters. Or 168 PowerPoint centimeters.

Unfortunately, Google Calculator doesn't recognize PowerPoint centimeters as a unit yet so you'll need to whip out a calculator and do the conversion yourself:
1 inch = 2.4 PowerPoint centimeters
1 centimeter = 1.0583 PowerPoint centimeters

How did this unit come about? From Microsoft's Knowledge Base article 189826:

To improve its usability, PowerPoint slightly misdefines the size of a centimeter to make the invisible gridlines fall at convenient points on the ruler. With this conversion, there are 5 picas per centimeter and the gridlines fall at very convenient points on the ruler. So convenient, in fact, that working in the metric system is really easier than working in the English system.

Apparently, the gridlines in PowerPoint 97 were on pica boundaries, which aligned nicely to the inch ruler since a pica is defined as 1/6 of an inch. But if you were to use PowerPoint 97 in one of those metric-using nations, the pica gridlines didn't fall on convenient points on the centimeter ruler. Solution: redefine centimeters to be an even multiple of picas, which are defined to be an integral fraction of an inch. In other words, Microsoft redefined the meaning of centimeters in PowerPoint such that a centimeter was defined in terms of an inch.

Microsoft had the arrogance to redefine an international standard measurement unit to fit their software's user interface. What makes the arrogance all the more palpable is that their Knowledge Base article acknowledging the existence of PowerPoint centimeters doesn't sound the least bit apologetic for the discrepancy.

Personally, I kind of prefer PowerPoint centimeters to metric centimeters: 168 PowerPoint centimeters tall is a lot easier to remember to an American like me than 177.8 metric centimeters tall is. <wink>

Update 2007/06/15 03:45pm:

My wife pointed out that apparently Microsoft redefined the size of a pica too: their Knowledge Base claims 12 picas to an inch, whereas there are supposed to be only 6 to an inch. I would like to suggest the name "PowerPicas" for these new pint-sized picas. That would make me 840 PowerPicas tall.

Security is a process

Let's say, hypothetically, you had a library of C code which implemented some function, frobulate(const struct foo *fooarray, size_t numfoos). For the sake of discussion, it really isn't important what frobulate does, lets just say that it performs an action on an array of structures. Now, it turns out that it is pretty common to call frobulate() with just a single structure, so we provide a helper function frobulateOne(const struct foo *foop).
I realize this is silly, but bear with me.

Now, lets say a developer using your library finds that passing a malformed structure to frobulateOne causes it to crash. You identify that there is buffer overflow in your library code that is caused by not properly validating the contents of the structure passed as an argument. How do you fix it?

You might add some additional validation to frobulateOne(), but certainly you would fix frobulate() to validate each any every structure. After all, frobulateOne() just turns around and calls frobulate() to do the real work. Obviously, it is necessary to validate the input to frobulate().

In fact, it is so obvious you are probably wondering why I'm wasting my time mentioning it.

Because somehow Microsoft, who made a high-profile effort to fix security bugs in their flagship product, screwed it up. When it was found in November 2004 that malformed animated cursors could lead to a buffer overflow, Microsoft issued a fix (MS05-002) 4 months later. The fix issued added input validation to the LoadCursorIconFromFileMap() function. Being highly-security conscious, you would assume that in that four months developing a fix, someone would have noticed that LoadCursorIconFromFileMap() is a wrapper that loads the icon data from the disk and calls LoadAniIcon() to actually process the data. Or that, after discovering such a bug in one piece of code, they would have their crack team of security experts scour through related code looking for similar bugs.

But, sadly, they did not. Earlier this week, Microsoft had to scramble to issue new patches applying the exact same input validation they added to two years ago to LoadCursorIconFromFileMap() to LoadAniIcon(). Independent security researchers had found Microsoft's obvious error before they did and hackers were exploiting the vulnerability to take control of Windows users' machines.

Of course, the IT press is encouraging everyone to apply Microsoft's patches ASAP due to the severity of the bug. Microsoft's detractors will hold it up as another example of how Microsoft's Windows operating system is inherently more insecure than their favorite OS.

But what strikes me is not the severity of the bug itself, but what it says about the internal processes at Microsoft. This latest security flaw is not the result of some previously-undiscovered bug -- it is the direct manifestation of Microsoft's failure to properly address the exact same flaw two years ago. And the correct solution, as I hope my little example at the start of this post demonstrated, should have been obvious from the start.

I'll avoid the debate of whether Microsoft's development processes are fundamentally flawed; I've never worked there so I wouldn't know. However, this bug should make it abundantly clear that Microsoft's processes for addressing known security issues are not just flawed, but practically non-existent. No organization employing professional software engineers could have missed the proper fix two years ago, much less an organization that professes to take security issues seriously. Microsoft claims to "have dedicated staff whose job it is to perform root-cause analysis of defects". Somehow I don't believe it. Or, if they do, I should expect they'll be hiring a new group soon.

Update 2007/04/06: Apparently I'm not the only one who is baffled how Microsoft could fail to have proactively fixed this recent bug and still profess to be focused on making Windows more secure.