Calculating MD5 of binaries without debug symbols

If you compile a binary with gcc with debugging information enabled (-g), the MD5 of the resulting binary will change depending on the name of the directory you compile it in. Which means that if two developers compile the same source code with the same options on the same machine, only they do it in their own home directories, the MD5 of the resulting binaries may differ.

However, as soon as you strip the binaries, their MD5s will be the same.

Which leads me to this little tool I whipped up to compare two binaries without stripping them:

#!/bin/sh
# Display the MD5 of a file, ignoring any debugging symbols in
# binaries.

# The strip(1)/objdump(1) commands for removing debugging
# symbols do not support writing to stdout so we need to
# allocate a temp file to write the stripped binary too.
tempfoo=`basename $0`
TMPFILE=`mktemp -q /tmp/${tempfoo}.XXXXXX`
if [ $? -ne 0 ]; then
    echo "$0: Can't create temp file, exiting..."
    exit 1
fi

while [ "$1" != "" ]; do
    # The following line is a cheezy way to accurately
    # reproduce the same error messages as md5(1) when a
    # specified file is unreadable.
    md5 "$1" > /dev/null
    if [ $? -eq 0 ]; then

        # Try to strip symbols from the file on the
        # assumption it is a binary and, if successful,
        # compute the md5 of the stripped file.  Note that
        # objcopy is the same as the strip(1) command.  If
        # objcopy failed to parse the file (i.e. because it
        # is not in ELF format), simply compute the md5 of
        # the whole file since there are no debugging symbols
        # to strip.
        m=`(objcopy -g "$1" $TMPFILE >/dev/null 2>&1 && \
            md5 -q $TMPFILE;) || md5 -q "$1"`

        # Output the result in a md5(1)-compatible format.
        echo "MD5($1) = $m"
    fi
    shift
done

rm $TMPFILE

su Forces Terminal Echo to On

I just discovered an interesting quirk of the su command, at least on FreeBSD: it forces terminal echo to on and you cannot turn it back off. Interestingly, sudo does not have the same behavior. Here is a sample terminal session showing the behavior (input that was not echoed is shown in red):

[14:54:35] myhost:~ $ stty
speed 38400 baud;
lflags: echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
[14:54:38] myhost:~ $ stty -echo
[14:54:57] myhost:~ $ stty[enter] speed 38400 baud;
lflags: -echo echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
[14:55:00] myhost:~ $ sudo -s[enter] Password: mypassword[enter]
[14:55:15] myhost:~ # stty[enter] speed 38400 baud;
lflags: -echo echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
[14:55:20] myhost:~ # exit
[14:55:25] myhost:~ $ su[enter] Password: rootpassword[enter]
myhost# stty
speed 38400 baud;
lflags: echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
myhost# stty -echo
myhost# stty
speed 38400 baud;
lflags: echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
myhost#

As you can see, the -echo flag was preserved by the sudo session, but not be the su session. Furthermore, it was not possible to re-disable terminal echo within the su session -- it simply ignored the directive.

I suspect the cause is somewhere in the PAM libraries that su uses to authenticate the user. However, I would have expected sudo to use the same PAM libraries, so I cannot explain the difference in behavior.

Not-so-Cool Biz

It is still the rainy season here in Japan but the temperature is slowly creeping up as we head towards summer. Today's high is forecast to be 30 degrees (86 degrees Fahrenheit).

Which wouldn't be too much different than the weather this time of year in my home town in Virginia except for two things:

1. In Virginia we commute by car and cars have air-conditioning; in Japan most people commute by walking or riding a bike to the train station and taking a crowded train.
2. Cool Biz

Today I'm going to focus on #2. If you aren't familiar with "Cool Biz", there is a nice article over on Slate about this Japanese campaign to reign in energy usage by making everyone uncomfortable.

The gist of Cool Biz is to crank up the temperature in offices to a balmy 28 degrees Celcius (82.4 degrees Fahrenheit) and allow people to "dress down" to compensate. By "dress down", we're talking not having to wear a tie.

I've heard the figure 4 degrees bantered around as the difference not wearing a tie can make. But when I tried researching where that number came from, I cannot find any research measuring the relative apparent temperature with and without a tie. The number (which I presume is in degrees Celsius) seems to appear only in articles about Cool Biz and only as a number the Japanese government cites.

Not surprisingly, the large company I'm working at here in Tokyo is following the government's recommendation and has the thermostats set for 28 degrees. And I can tell you first-hand that the heat is incapacitating.

I was doing some research on how to deal with the heat and ran across some tips for living without air conditioning. For a wussy American such as myself who has been spoiled by comfortable working conditions for my whole life, 82.4 degrees might as well be no air conditioning. The tips:

• Turn on fans to keep the air circulating.

We have 2 fans in a room with over 100 people in it.

• Set ceiling fans to turn clockwise (counter-clockwise looking toward it).

Not really applicable in an office environment.

• Keep curtains or blinds closed during the day.

Blinds are kept open all day.

• Use light colored curtains to deflect heat.

The blinds are white. So if we were allowed to close them, it might help.

• Keep windows and doors closed tightly to keep the house cool.

Check.

• If there is a breeze or if outside air feels cooler than inside, open the window bringing in the cooler.
• Keep sunny windows closed if there is no breeze.
• In late afternoon, as soon as outside temperatures feel lower than inside temperatures, open all windows and doors.
• In the evening, when it’s cool outdoors but still hot indoors, place fans in front of open windows to draw in the cool air.

I work in a fairly old office building that still has windows you can actually open. However, we are forbidden from opening them. Rumor has it that a couple years back someone snapped and try to jump out a window. Since then, we've been strictly forbidden from opening the windows. I suspect it was the heat that drove him to it.

• Keep as many windows as possible open during night to take advantage of the cool night air to lower inside temperatures.
• In the morning, close windows as soon as the outside air begins feeling warmer than inside air.

Besides not being feasible in an office environment, see the notes regarding the previous tips.

• Limit strenuous physical activity until evening.

Check.

• Drink lots of cold water.

And check.

• When hot, use a water spritzer to spray yourself. The evaporating water will make you feel cool. (For fun, you can also spritz the water upwards and feel it fall down on you. The coldness can be quite shocking.)

I work in an open office floor plan with people sitting 2 feet from me in each direction and no walls. This might work if they are amicable to the idea too.

• Tie a wet bandanna around your neck (this is something I do for my dog to prevent overheating and heatstroke during the summer).
  

I think we have a winner. I think I may have to try this one when the summer heat really sets in. Maybe I can start a fashion trend of white buttoned-up dress shirt, dress slacks, and a wet bandana. If I combining it with spritzing water, I can call it "wet biz".

Whats and Whys

I just realized something about myself today: when someone is talking to me, I'm not only listening to what they are saying, but also trying to deduce why they are saying it. Not consciously, of course, or else I would have realized that I'm doing it long before now. :)

In hindsight, I think this has been mostly a good thing. As a software engineer, when a request comes in for feature "X", rather than just understanding that the customer wants feature "X" and immediately starting to work on solving that problem, somewhere in the back of my brain I'm thinking about why the user asked or "X". What are they probably trying to accomplish? Is there perhaps a simpler way to achieve the same result that we can present as a counter-proposal? Is this something other customers might need too? If they are asking for feature "X" now, then they'll probably need feature "Y" at some point in the future too.

Which is to say that I think this has made me a better software engineer. However, sometimes it takes me a night or two of "sleeping on it" before I start being aware of the implications of the request. I've gotten into the habit of splitting my project to-do list into "things needed to do what the customer specifically asked for" and "things needed to do what the customer probably wants" categories. Obviously, the items in the list needed to satisfy the customer's explicit requirements always take priority. But I'll freely add items to the "customer will probably want" list as they occur to me, to be implemented in down time between projects or assigned as introductory tasks to junior engineers to get them familiar with the code base. From experience, I've found that this process results in more generic solutions to problems that make the product more resilient to changing requirements and adaptable to new customers.

On the flip side, I've gotten myself in trouble a few times by reading to much into my wife's words. I don't think I'm unique in that regards either. :)

Today I realized that I am sometimes perplexed by requests/statements from my Japanese co-workers. Of course there is the language barrier which means that it takes a lot more effort on my part to understand what is being said. But sometimes I know I comprehend what they are saying, but I still have an uneasy, discontent feeling. That is when I realized that my brain is subconsciously trying discern the "why" behind the statement and failing, causing the uncomfortable feeling.

As for why it is failing, I'm still unsure. But now that I think I know what is going on, I can work to address it. I suspect it may have nothing to do with the fact that the statements are in Japanese, but rather be a symptom of information asymmetry inherent in working in a large company. In which case, it would explain why I always seem to flourish in small company environments where information is more evenly distributed, hence making it easier to deduce the "why" behind new tasks.

A Score I'm Not Proud Of

I was doing a little research to write a post comparing incentives in the Japanese and U.S. workplace. In the process, I ran across the Zung Self-Rating Depression Scale. I know I'm not happy here in Japan, but I thought I would take the online test just to see how bad it was.

I got a 75 out of 100. That would be "severely depressed"...which is quite an eye opener for me.

The funny thing is that, compared to the people I see on my commute, I think I seem pretty well-adjusted. I mean, Japan has some of the highest incidence rates of suicide and alcoholism amongst industrial countries. I guess I can see why. At least I have something to look forward to: I can go home and get back to a normal life. But I feel for the poor Japanese salarymen I see slumped over on the train platform nursing a beer and staring blankly off into the distance...this is as good as it gets; there is nowhere else to go.

Anyway, I'll try to get back to writing my observations on incentive systems for a future post. For now, I'll leave with the same comment I give everyone who asks about Japan: it is a nice place to visit, but I wouldn't want to live here.

See Me at Interop Tokyo 2009

In an effort to put a face behind its technology, NTT Communications will be showcasing portraits of some of their engineers at their booth at Interop Tokyo this year. And unless I had spinach in my teeth, there is a good chance my portrait will be there this year....even if I'm not. Six members of the team that runs the servers behind the HOTSPOT service, myself included, were gathered after lunch today to have our pictures taken. It wasn't anything fancy: first, they each of us pose for individual pictures, then did a few pictures of us in groups of three, and finally one big group photo of us at a conference table.

So if you find yourself at Interop Tokyo 2009 this upcoming June, swing by the NTT Communications booth. Above the booth you should see a number of portaits of engineers. I'll be the stupid-looking one.

Racism in Japan

I generally try to keep myself upbeat and try to maintain a positive outlook, but the racism in Japan really wears me down. The "帰れ(go home)", "死ね(die)", and host of other comments muttered in passing or said behind my back have really dampened my excitement about living in Japan. While less blatant, the rampant tendency for strangers to clear their throats or spit when they become aware of my presence suggests the racism runs pretty deep.

In hindsight, learning Japanese may have been a mistake; if you can't understand the language, you don't know when people are making rude comments.

Even as a white male, I do my best to not draw attention to myself. I've seen the looks of distrust and fear my fellow commuters give to foreigners of middle-eastern or African decent on the train. I can only imagine the death by the thousand cuts that everyday life in Japan must be for them.

My poor 5'2" wife isn't immune either. She's fed up with the way little old ladies stare at her and awkwardly avoid her like she was some sort of circus freak.

My coworkers often ask me why my wife and I don't go out and see more of Japan on the weekend. Frankly, at the end of a grueling work week, I don't have any urge to stand on the train some more. But more important, the last thing either of us wants is our weekend spoiled by bigots. Just going to the store can be emotionally draining. Trying to rent a room in the country can be outright impossible (70% x 37.8% = ~26.5% of hotels surveyed illegally refuse to serve foreigners; they cite "language barriers" -- apparently confusing nationality with language ability).

The almost-daily little hints, most of which seems trivially minor in isolation, add up to a roar of "you're not welcome here".

Before we came to Japan, we were familiar with the complaints many foreigners living in Japan have. The comment sections of many popular news sites aimed for expatiates living in Japan are filled with disillusionment. I always figured it was because the people hadn't taken the effort to learn the language, or had unreasonable expectations, or were being obnoxious in public...it had to be something they were doing wrong. Maybe it was; maybe I've made the same mistakes.

Since coming to Japan, I have come to appreciate Dave Aldwinckle's complaints and the hard work he has been doing to try and bring the injustices in Japan into the forefront. Whenever I get worked up enough about something that I want to bitch about it on my own blog, I just need to go hit his debito.org to commiserate.

When I was sitting in the comfort of the U.S.A., Debito's stories seem farfetched and, frankly, unbelievable. More than once I thought he was making a mountain out of a molehill. However, I now realize that he doesn't have to go digging to find examples of racism, discrimination, injustice, and hypocracy...it turns out there is just a lot of material to pull from here in Japan.

Unfortunately, while brave individuals like Debito are trying to recitify the situation, apologists still abound. Guides for foreigners coming to Japan are filled with such insightful advice as:

NOTE: At certain points through this and other articles we note that in some cases foreigners may be refused entry to particular hotels or rental of accommodation. This is not intended to imply any form of prejudice; it is merely a statement of the facts. Almost exclusively this is due to the very low number of foreigners in Japan, and general ignorance among the Japanese regarding foreigners. Smile, persevere, and try to be a good ambassador not just for your own country but for all non-Japanese in general.

(taken from http://educationjapan.org/jguide/accommodation.html)

That's right, denying you access to shops just because you are a foreigner isn't any form for prejudice. Nope. People may be ignorant and racist, but if there is one they are not: they are not prejudiced. So just suck it up and deal.

The implicit message: "no one wants you here and if you don't like it, well go home."

Anyway, I don't want to give the wrong impression: I've met a lot of nice people in Japan (and Japanese living overseas). If it weren't for the great folks I have (and have had) the pleasure to work with, I probably wouldn't still be here. That said, I am looking forward to going home. Honestly, many people I have met seem to have no comprehension of how intolerant their countrymen are or, as the quote above demonstrates, don't even recognize actions as being discriminatory in the first place.

So until I can go home, I guess I'll have to continue to find solice on Debito's blog.