The Other Kelly Yancey

Yes We Could've

2012-01-10T18:40:00.002-08:00

It has been a sad ten years for the American people and it is shaping up to be a sad ten more.

Tomorrow is the 10th anniversary of the arrival of the first prisoners detained at Guantanamo Bay. As despicable as it is that the United States government ever operated the facility, the fact that it is still open with 171 prisoners held without trial is a sad testament to the morality (or impotence) of our leaders, our politicians, and our people.

This isn't a simple matter of inheriting a bad legacy from the previous administration. By passing the National Defense Authorization Act for Fiscal Year 2012, authorizing the arrest and imprisonment without charge or trial of terrorism suspects, Congress recently demonstrated a renewed commitment to the policies that led to the creation of the Guantanamo Bay prison in the first place.

Bon Nouvel An

2012-01-01T02:35:00.005-08:00

12 years ago, almost to the day, my wife and I started learning Japanese. It began when my wife gave me Pimsleur's Japanese 1 for Christmas 2000. We studied together for the following 8 years, driving to San Francisco almost every Saturday to take lessons at Soko Gakuen, spending every Saturday and Sunday evening watching Japanese programming on KTSF, and most other evenings practicing Kanji and vocabulary flashcards. It was a lot of work, but I really enjoyed it. For the first time in my life, I felt like I was capable of learning another language (as a point of reference: I studied Spanish for 5 years in High School & College and can't say I ever felt competent in Spanish).

Finally, we moved to Japan in October 2007 where we lived for the following 2 and a half years. That also marked the end of our studies. During our time in Japan, I spent pretty much all day every day conversing and corresponding in Japanese, so I feel that my reading, speaking, and understanding skills continued to improve, but I stopped learning new vocabulary and stopped hand writing Kanji so I feel like I left Japan in some ways worse off than I had arrived.

Since returning to the U.S., I have been on the fence about resuming my Japanese studies. While I really enjoyed learning the language and met a number of kind and interesting people through my studies, it just isn't practical for me to continue. I have no intention of going back to Japan to work, my current work doesn't require Japanese skills, I've never been very keen on anime, and while being able to play Japanese video games in their native language is a neat trick, it really isn't worth spending years of your life for.

So it is with some sadness that I've finally decided that I won't be resuming my Japanese studies for the foreseeable future.

As a sort of final trip down memory lane (and possible reminder list should I decide later to study Japanese again), here is are some of the reference books that I found invaluable in learning Japanese:

Japanese for Busy People Volume 1, 2, and 3. Kana versions, of course. If you aren't willing to learn the Kana, then you might as well stop pretending you are going to learn Japanese. These are a great place to start learning grammar, vocabulary, and kanji. Once you finish all 3 volumes, you are probably ready to take level 3 of the Japanese Language Proficiency Test.
After you pass the level 3 test, it is time to start studying for the level 2 and level 1 tests. Really, the JLPT tests can be thought of as two sets of tests: level 4 and level 3 are for beginners; level 2 and level 1 are intermediate/advanced. The grammar covered in the level 4 & 3 tests is very similar, but level 3 is harder due to including more vocabulary and Kanji. Likewise, level 2 and level 1 cover almost the same grammer, but level 1 requires almost twice as much kanji and level 2. Anyway, when you get to studying for the level 2 and level 1 tests, you have to pick up a copy of どんな時どう使う日本語表現文型５００ from ALC Press. I found this book to be absolutely invaluable; my copy is well-worn with lots of notes in the margins.
Now, to actually prepare for the level 2 and level 1 tests, I recommend the following study guides:

Finally, if you are thinking of living and working in Japan, I highly recommend getting a copy of 敬語これだけBOOK. I picked this up on a whim while waiting in line at a bookstore and simply cannot recommend it enough. People will tell you that you don't need to know 敬語(keigo)...they are lying. Unless you have a significant other who can take care of your housing arrangements you will need keigo. Unless you work for a foreign company (where you can speak English) or you have no intention of working or researching in a professional office environment, you will need keigo. This book is extremely easy to understand; it consists of a series of situations where keigo would be appropriate and tells you 3 acceptable phrases -- ranked "good", "better", and "best" -- for each scenario along with a brief explanation why one phrase might be better than another. This books was written for young people in Japan, who also have trouble with honorific speech, so the opposing page consists of 3 common mistakes with explanations of why the phrases are wrong. As a non-native speaker, you can reasonably skip reading the mistake page -- if nothing else, skipping it will make sure that you don't have the wrong words floating around in your head ready to slip out when it matters most. Seriously, get this book.

Other than that, I highly recommend the full 3-course Pimsleur Japanese language program to anyone getting started with learning Japanese. I'd go so far as to suggest that you complete the entire first course -- 30 lessons -- before you even start with the Japanese for Busy People books. Buying the series new is pricey, so check with your local library, E-bay, or Craigslist for a used copy.

Finally, if you are in the San Francisco Bay Area, you have a great local resource in Soko Gakuen. It is a non-profit language school associated with the Buddhist church in San Francisco's Japantown. The classes are reasonably-priced, have small class sizes, and are taught by great teachers. My wife and I took classes there every quarter for over 6 years.

So, what now?

Next up, we are trying our hands at learning French (hence the title of this post). We started the same way we had success with learning Japanese: with the Pimsleur Language Program. We're still on the first course, but so far, so good. Here's hoping that our studies will, again, lead to something rewarding.

Happy New Year!

That's a lot of dough

2011-12-12T10:57:00.000-08:00

As we prepare to celebrate another happy end of the fiscal fourth quarter, the good folks over at xkcd.com have prepared an awesome infographic reminding us what a dollar is worth. Click to see an intelligibly-sized image:

Wither posi.net

2011-11-30T22:15:00.003-08:00

I first registered the domain name posi.net back in 1995 during the great domain name gold rush. OK, so I was a little late to the party and all the good names were taken, but I was pretty happy to get a short domain name. I also kept the e-mail address kbyanc@posi.net until last year, when the volume of spam overwhelmed the little 233Mhz Soekris box I've been using as my firewall, mail, web, and home file server.

Earlier this year, I stopped renewing my Dynamic DNS service, effectively taking posi.net offline. And this month, after 16 years, I decided not to renew my domain registration.

I never really did much with the posi.net domain -- I mostly just used it to host my open source code repositories and a few toy web sites. I've been slowly working on converting the toy web sites to run on Google App Engine and am thinking of uploading the open source repositories to github. I'm not sure where I'm going to host my resume, but I'm not in any hurry to figure that one out.

We the people

2011-11-20T22:26:00.007-08:00

...apparently flunked our high school government classes.

As you probably know, the Obama administration has set up a "We the people" section on the White House web site so as to solicit petitions from the American people. While there are a few well thought-out petitions, the majority just serve to demonstrate that many Americans have no idea how our federal government is designed to work nor what the function of the Executive branch is. For example:

Amend the Constitution, making the Internet an unalienable right. Newsflash: the president does not have the power to add amendments to the constitution. Not even close. Petition for it all you want, he can't make this happen.
Void the voting results that took place in Nicaragua this past Sunday 06 of November due to fraud by the FSLN. Seriously? He's the President of the United States. When did Nicaragua become a state?
Legalize Mixed Martial Arts in New York. The President doesn't pass laws, much less state laws. You too are barking up the wrong tree.
Re-establish and maintain the separation between investment banks and commercial banks. I'm 100% for the restoration of the Glass-Steagall Act...but the President doesn't pass laws. You need to petition your Congress-critters to get this done, not the Obama administration.
Bar courts and lawmakers from a "second-class religion" status for minority religions like Wicca and NeoPaganism. He's the President, not Dictator; he can neither tell courts nor lawmakers what to do. Have you not heard of the separation of powers?
Actually take these petitions seriously instead of just using them as an excuse to pretend you are listening. See above. I imagine it is hard to respond to petitions demanding the President to do things he has no legal power to do. Well, respond with something other than "/facepalm", that is.

I'm sure some of the authors of the above petitions would try and defend their misdirected efforts at government participation by pointing out that neither the Senate nor the House of Representatives offers a similar venue for soliciting petitions. That is true, and it is unfortunate. But that doesn't change the fact that the White House cannot act on many of the petitions. And, when some staffer writes a lengthy explanation of what the Executive branch has been doing to try and support your cause, it is doubly unfortunate that you don't seem to comprehend that it is all that he can do to support your cause.

If you want to change the laws, you need to write your Congressman. It isn't the President's job to get your Congressman to vote the way you want.

Ken Burns: The National Parks

2011-11-05T17:24:00.004-07:00

We finished watching six of the most inspirational movies I've ever seen: the six part series that is Ken Burns' The National Parks: America's Best Idea. Each part runs at a little over 2 hours long, full of American history set to the most beautiful scenery in the country.

Starting with the founding of the first national parks, with Yosemite and Yellowstone, and continuing through to the present, the series chronicles the American values that led to the creation of the national park system. And, rather than just painting the picture in broad stokes, the series follows the individuals who fought to establish the parks for the common good as well as those who wanted to exploit the land for private gain.

I can't recommend the series enough. Of course, PBS airs them from time to time, but you can also watch them on Netflix (which is what we did). The visuals are breathtaking, the history illuminating, and the issues as topical today as they were 150 years ago.

Ingenious fraud

2011-09-07T14:30:00.006-07:00

I just got a SMS from 503-929-3160 with the text

WELLS FARGO ALERT: Your ATM CARD has been DEACTIVATED. Please contact us at: 650-550-9255.

There were 2 tip-offs that this was a fraud: first, the stereotypical use of unnecessary capitalization and, second, the fact that I don't have an account with Wells Fargo.

Out of curiosity, I called the number to be greeted with an automated voice claiming to be the Wells Fargo card activation line and asking me to enter my 16 digit card number. I entered a bogus card number (sixteens ones) and was promptly cut off. Rumor has it that if you enter something with a valid check digit that the automated service will then prompt for your PIN number and then proceed to drain your account empty.

The ingenious part of this scam is that it relies on the fact that there is no way to authenticate who sent a SMS. With online phishing attacks you can look at the URL to confirm that you are dealing with the entity that you expect. In addition, banks and other high-profile web sites get Extended Verification Certificates for their websites to help make it more clear when you are interacting with the real thing. But there is no such thing for text messages: you just see a phone number. How many people know the phone number of their bank and/or have entered it into their phone's address book?

With services like Twilio making it trivial for ne'er-do-wells to extend their phishing attacks out of cyberspace into telephony, I suspect we'll be seeing more of these types of fraud attempts in the future. Of course, savvy people will never trust random text messages, but that still leaves a huge potential target for increasingly sophisticated fraud. God knows I hope my mother doesn't get one of these texts.

Introducing Jamie

2011-08-18T20:21:00.004-07:00

Born 4:30am this morning at 20 inches long and 9 pounds. Both he and Mom are doing fine.

The "Keep Houses Unaffordable" Initiative

2011-07-04T17:26:00.004-07:00

In a misguided attempt to prop up the "values" of homes, the Federal Government is giving free money to people who bought houses they couldn't afford by extending low- or zero-interest loans with an option to not repay the loan. There are two separate programs: the HUD Emergency Homeowners' Loan Program and the Treasury's Hardest Hit Fund.

As a responsible family that did not overextend ourselves to by a house that we couldn't afford, this is just a slap in the face. I don't begrudge people who bought homes and have now fallen on hard times...although the possibility of job-loss is supposed to factor into the calculation of how much house you can afford. No, I'm irritated because these programs are feel-good attempts to prop up the still-overpriced housing market so that banks don't have to recognize the true value of their mortgage-backed assets. You see, as a lender, banks should care about the borrowers ability to repay. So long as the government is willing to step in and pay when borrowers can't, it just reinforces bad lending practices.

Meanwhile, keeping home prices high is in no way good for us little people. It doesn't help people to have the burden of a home they can't afford hanging like an albatross around their necks. It doesn't help young families (such as my own) buy their first home...in fact, it actively obstructs that. Unless income rises to the point that homes are no longer ridiculously overpriced, which -- with real wages falling for 40 years -- doesn't seem likely, it does nothing to make homes more affordable. And affordability is the real problem with the housing market.

The Robber Barons of the 21st Century

2011-06-20T20:19:00.006-07:00

I guess this is old news for subscribers of Rolling Stone, but I just ran across this excellent article they published in 2009 regarding the role the Goldman Sachs plays in the U.S. economy. In contrast to the random sound bites that fill so much of our media, it is well-researched article that it well worth the long read. Here is a link directly to the print version so you don't have to click through the 8 pages.

If, after reading that article, you are left with any doubt regarding Goldman Sachs' rigging of the U.S. economy in favor of themselves (not even their investors...just the bankers!), here is a current article from the Wall Street Journal chronicling their manipulation of the aluminum commodities market.

Folks, deregulation has gone too far. Over the last 25 years, Goldman Sachs has spearheaded the dismantling of the protections put in place in the wake of the Great Depression and look where it has gotten us.

Update 2011/07/04:: The Wall Street Journal has replaced their article with a partial snippet. Here are a few more articles on Goldman Sachs' manipulation of the aluminum commodities market.

I can haz free house?

2011-06-16T20:16:00.004-07:00

In case anyone was wondering, I'd like a free house. Shoot, I'm not greedy, I'd settle for just not paying rent for a couple of years. Oh, and some free money too. Thanks!

Personal Exemption

2011-04-05T22:17:00.005-07:00

As part of my research while implementing my mortgage estimator I discovered that, in addition to the standard deduction, the federal income tax code includes a personal exemption. Together, these two are intended to prevent subsistence-level income from being taxed. In other words, Congress intended to shelter the lowest rungs of society from taxation.

Now, obviously, the richer you get the less you need the personal exemption...when you make $200,000/year there is no risk of not being able to afford to eat. Accordingly, the IRS phased-out the amount of the personal exemption one could claim on their federal taxes as their income increased. For example, in 2009, the personal exemption was $3,650 but, for couples filing jointly whose adjusted gross income exceeded $372,700 the personal exemption was reduced to $2,433.

It is only a $1,217 difference, which at the maximum marginal tax rate would only amount to a little over $400 per person. Hardly a drop in the bucket for a couple making $372,700/year.

But the personal exemption phaseout was eliminated under President Bush's 2001 tax reforms...effective 2010. Besides ignoring the intent of the personal exemption, were the richest 1% of Americans really hurting for $400?

Fortunately, this is one tax cut for the rich that won't be sticking around: the budget proposals for 2011 restore the personal exemption phaseout. I hope they spent their one-time $400 windfall wisely.

It is hard being rich

2011-04-01T09:49:00.005-07:00

As Republican Representative Sean Duffy reminds us, it is hard to live on almost 200 grand a year:

I guess that is why the ultra-rich need to pay a lower tax rate than the rest of us...it must be tough at the top.

Yet Another Mortgage Estimator

2011-03-16T22:48:00.005-07:00

The New York Times has a really slick rent-versus-buy calculator; that site has everything you need to compare renting versus taking on a mortgage. I can't recommend it it highly enough.

One thing that the New York Times' calculator confirms is that renting is always better than buying in Silicon Valley. You would need to stay in the same house for about 40 years before it became cheaper than renting for the same period of time.

But "wait!", you say, " you are just throwing your money away when you rent." That is true. But it turns out that housing prices are so high in Silicon Valley, that you actually throw away more money in interest when you take on a mortgage to buy a house (or condo even).

It is at this point someone helpfully adds, "but you can deduct interest from your taxes".

So the questions are:

How much money do you "throw away" paying mortgage?
And, how much of that do you "get back" from Uncle Sam via the federal mortgage interest deduction tax break?

The first is pretty easy to answer; any loan amortization table will tell you how much interest you pay given a loan amount, interest rate, and number of years you'll be borrowing the money. The second is a little more complicated. Let me explain:

The federal mortgage interest deduction is just that: it allows you to deduct the amount of money you paid in mortgage interest from your income for that year. In other words, it is as if you never made that money. So, if you earn $60,000 in gross income and pay $12,000 in mortgage interest, Uncle Sam will pretend you only earned $48,000 for the sake of calculating your federal income tax.

That wouldn't mean that you get $12,000 off your federal income tax, though; it just reduces the amount of income you have to pay tax on. Meanwhile, you only get to deduct the mortgage interest if you itemize your deductions. This means more work for you in preparing your taxes, but more importantly, it means that you have to give up the "standard deduction". The standard deduction is the amount that everyone is entitled to deduct from their gross income; but you can't take both the standard deduction and an itemized deduction at the same time. So you only want to take an itemized deduction if it would be greater than your standard deduction. For a married couple filing jointly the standard deduction in 2011 will be $11,600.

Going back to the previous example, if you paid $12,000 in mortgage interest you could itemize your deductions and reduce your taxable income by $12,000 or you could not itemize and take the standard deduction of $11,600. With a hypothetical $60,000 of gross income, itemizing would bring your taxable income down to just $48,000 versus $48,400 without itemizing. Obviously, in this case, the itemized deduction leaves less of your money subject to federal tax, so you would want to take that. But it should also be clear that the benefit of the federal mortgage interest deduction is not really $12,000 since, if you didn't have a mortgage, you could still deduct $11,600. So the tax break is really worth the difference between an itemized deduction and the standard deduction...or just $400. But it isn't worth $400 either because it just means that $400 less of your money is taxed; at $60,000 your marginal tax rate would be 15% so the difference in tax is just $400 x 15%, or $60.

That's right, the federal income tax break on $12,000 of mortgage interest is a whopping $60.

The value of the deduction depends on your taxable income and the amount of mortgage interest you pay each year. The amount of mortgage interest you pay goes down as you pay back principle so the value of the tax break also goes down each year.

Anyway, I think you get the idea...calculating the value of the federal mortgage interest deduction isn't trivial. Basically, you have to do your taxes twice, once itemizing to take the mortgage interest deduction and again with the standard deduction, each year, in order to calculate the total value of the tax break.

So I made my own mortgage estimator that approximates the value of the mortgage interest deduction and, factoring that number in, tells you how much money you actually "throw away" on mortgage interest. It also amortizes the after-tax-break interest over the number of months you plan to live in the house, which yields a number comparable to the amount you would "throw away" on rent.

My calculator is here: http://mortgage-estimator.appspot.com/.

The Loan Amount, Interest Rate, and Term fields are used to calculate the amortization table for the mortgage and hopefully are self-explanatory. You can also input the loan amount by entering the Home Price and Down Payment; in which case, how much equity you are putting down is displayed next to the Down Payment as a percentage.

The Gross Income, Filing Status, and Children (actually, dependents) fields are used to estimate your federal tax obligations. The tax estimator is pretty simple; it doesn't know anything about the more obscure deductions nor does it handle investment income, but I believe it is sufficient for its purpose in comparing your hypothetical tax obligation with and without the mortgage deduction.

Finally, the Expected Residency field is an acknowledgement that none of us are likely to live in the same house for the rest of our lives; here you can enter the number of years you expect to live in the home. This is important in the "how much money am I throwing away in interest" calculation because mortgage interest -- like all loan interest -- is front-end loaded. You pay more interest at the beginning of your loan and more principle towards the end. So the fewer years you live in the house before you sell it, the higher your paid interest-to-principle ratio will be.

And now for the minutia:

All numbers are rounded when they are displayed. So $31.586 will be displayed as $31.59 and $16.134 will be displayed as $16.13; their difference is $15.452 so will be displayed as $15.45 not $15.46. Not that you should expect this estimator to be accurate to the penny anyway.

The estimator does not currently take into account private mortgage insurance nor FHA loan assistance programs.

The estimator does not currently take into account state taxes and any possible state-specific home ownership incentives.

More TSA lunacy

2011-03-14T14:26:00.002-07:00

As if the TSA's gratuitous strip-searches at the airport weren't enough, they've branched out into train travel too. You know, in case someone gets the bright idea of highjacking a train and running it into a skyscraper. But seriously, groping people as they get off the train? What could the logic possibly be? Certainly not safety: the trip had already concluded incident-free.

We are paying perverts to do nothing of value.

So this is what it has come to? The American taxpayer shells out 8.1 billion dollars a year to get felt up by strangers while our lawmakers fight to kill Big Bird to save 451 million dollars. I'd like to actually meet the people who think paying perverts to feel up law-abiding people after they demonstratively proved they are no threat is a better use of our tax dollars than, well, anything.

Dear Amtrak

2011-02-03T12:15:00.006-08:00

Dear Amtrak,

A bus is not a train. If I wanted to take a bus, I'd go to greyhound.com and plan my trip. But I didn't; I went to amtrak.com because I wanted to plan a train trip. To that end, I want:

A list of actual train stations, not bus terminals
A map of train routes and their transfer points
An option on your trip planner to not include bus routes

Look, I'm not stupid; I realize you can't stop in every city. But I don't expect you to. If I'm planning a cruise, I can go to carnival.com, and they provide me a list of departure ports to choose from. They don't claim to service Omaha, Nebraska and, when I choose Omaha, offer a trip that includes a bus to New Orleans. But that is exactly what your retarded web site does when I try to plan a train trip.

So, please Amtrak, don't make it so damn hard to use your service. With the TSA intent on ogling and fondling our wives and children, you've got a great chance to win the business of concerned middle-class citizens. President Obama even threw you a bone in his State of the Union address. But here's a newsflash: while train travel can compensate for being slow by being nostalgic, bus travel is slow, uncomfortable, and generally un-cool.

So quit hiding behind the transport of last resort, the bus, and take a page from the cruise lines: some of us are as interested in the trip as we are in the destination.

Personal income taxes and job creation

2010-12-02T19:11:00.008-08:00

With elections over, the issue of whether to extend the Bush-era personal income tax cuts to families earning over $250,000 a year is back in the news. Currently, there is a 33% tax bracket that affects income in excess of $209,250 and a 35% tax bracket for income in excess of $372,950 (for joint filers). So, in effect, under the Democratic proposal, a new tax bracket will be created between the two existing brackets like so:

33%	$209,250-$250,000
36%	$250,000-$373,650
39.6%	$373,650+

The exact numbers may still change as the details are worked out, but bear with me.

The Republicans are currently pushing to extend the Bush-era personal income tax cuts to families earning over $1,000,000 a year or extend them to everyone. If this were to come to pass, again an extra tax bracket would be created, but it would look something like:

33%	$209,250-$373,650
35%	$373,650-$1,000,000
39.6%	$1,000,000+

Again, the exact numbers aren't all that important. For the sake of example, though, let's look at a scenario of a family making a cool half-a-million dollars a year in personal income. Currently, this family would be paying approximately $141,563 in federal income tax (assuming the standard deduction of $10,700 for 2010).

Bracket	Range	Dollars Taxed in this Bracket	Tax Amount
10%	$0 – $16,750	$16,750	$1,675.00
15%	$16,751 – $68,000	$51,250	$7,687.50
25%	$68,001 – $137,300	$69,300	$17,325.00
28%	$137,301 – $209,250	$71,950	$20,146.00
33%	$209,251 – $373,650	$164,400	$54,252.00
35%	$373,651 +	$115,650	$40,477.50
Total:		$489,300	$141,563.00

Wow, that's a big number. It is funny to think we're having all this argument over people earning so much money that their taxes are triple what the average American grosses in a year.

Anyway, our poor put-out example family is paying $141,563 out of their $500,000 annual income in federal income tax. That is an effective tax rate of 28.3%.

Under the Republicans' proposed plan, there would be no change in the amount of federal income tax paid by our hypothetical family since they earn less than 1 million dollars a year.

Under the Democrats' proposed plan, our hypothetical family would have to pay $150,592 in tax, which is 30.1% of their income.

Bracket	Range	Dollars Taxed in this Bracket	Tax Amount
10%	$0 – $16,750	$16,750	$1,675.00
15%	$16,751 – $68,000	$51,250	$7,687.50
25%	$68,001 – $137,300	$69,300	$17,325.00
28%	$137,301 – $209,250	$71,950	$20,146.00
33%	$209,251 – $250,000	$40,750	$13,447.50
36%	$250,001 – $373,650	$123,650	$44,514.00
39.6%	$373,651 +	$115,650	$45,797.40
Total:		$489,300	$150,592.40

So the difference between the two proposals amounts to a 1.8% tax increase in this example. With a little hand-waving, let's just say the argument is over a 2% tax increase affecting families earning between $250,000 and $1,000,000 a year.

The Republicans claim that this tax will curb job creation. In response to the vote in the House of Representatives approving of the Democrat's proposal, Republican representative Gary Miller of California issued a statement saying

During these difficult economic times, raising taxes on any American family or small business will not help our economy recover nor foster the private-sector job growth needed to achieve economic recovery. The only thing that Democrats have accomplished by today's vote is yet more uncertainty for our nation's job creators.

Certainly, no one will argue that the U.S. could use more jobs. But is a 2% personal income tax increase going to materially affect job creation? Returning to the example above, a small business owner making $500,000 a year would see a difference of about $9000 in their take-home pay. That isn't enough to create even one job.

In any event, this ignores the elephant in the room: the issue being debated is a tax rate on personal income tax, not corporate tax. Why would a business owner pay out income from their business to themselves, incurring personal income tax, only to reinvest that money into their business? Wouldn't it make more sense to create those jobs using *before* tax dollars? And that is what any business owner can do -- and is doing -- right now, under the current tax law. And what they'll be able to continue doing no matter what happens with regards to personal income tax.

So it is patently silly to think that a decrease of any kind in the personal income tax is going to affect job creation. The money that creates jobs isn't taxed. You don't get lower than a zero percent tax rate. Businesses are not directly affected by the personal income tax rate.

No, the Republicans' cherished 2% tax cut on families making more than $250,000 a year only helps wealthy people put more money in their pockets. At best, businesses may benefit indirectly by virtue of the fact that wealthy people have more disposable income.

Where are the protesters?

2010-11-21T00:00:00.005-08:00

I'm waiting with baited breath to see whether there will be any protests over the airport nudey scanners and TSA groping this Thanksgiving holiday. I mean, we all remember the riots over universal health care, right?

If the government putting their nose in our personal business rankled people, surely the government putting their hands in our privates will unleash true rage.

Surely people will be up-in-arms over the TSA taking naked pictures of them and keeping the good ones for fun. Surely people will be angry that their wives and daughters are being felt up by convicted sex offenders. Surely people will be rabid over the loss of their 4th amendment rights.

Given the public shows of anger and disgust -- the public backlash -- in town hall meetings regarding something as mundane as health care, I expect the protests against the government's full assault of our privacy and dignity to be something to see. I can't wait.

Update 2010/11/24:: Well, the busiest travel day of the year has passed without any notable protests. I'm not quite sure what to make of that.

New Chapter

2010-11-14T23:03:00.004-08:00

Well, tomorrow is a big day for me: I'll be starting a new job. I'll be working for a super-secretive company in Cupertino which means that I won't be writing any more tech-related posts. I'm sure the next few weeks will be hectic, so I doubt I'll be up to writing much anyway. But I've got some ideas of what I'd like to write about once things settle down.

I'd say "stay tuned" but I'm pretty sure no one is tuned in to my blog as it is. :)

Python: Enumerating IP Addresses on FreeBSD

2010-11-12T18:45:00.004-08:00

As promised in my earlier post on enumerating local interfaces and their IP addresses on MacOS X, this time I'll cover how to do the same on FreeBSD and other operating systems that implement the getifaddrs API. Basically, this is just a python wrapper around the getifaddrs interface using ctypes.

The code is a bit longer than I typically like to include in a blog post, but here it goes:

"""
Wrapper for getifaddrs(3).
"""

import socket
import sys

from collections import namedtuple
from ctypes import *

class sockaddr_in(Structure):
    _fields_ = [
        ('sin_len',     c_uint8),
        ('sin_family',  c_uint8),
        ('sin_port',    c_uint16),
        ('sin_addr',    c_uint8 * 4),
        ('sin_zero',    c_uint8 * 8)
    ]

    def __str__(self):
        assert self.sin_len >= sizeof(sockaddr_in)
        data = ''.join(map(chr, self.sin_addr))
        return socket.inet_ntop(socket.AF_INET, data)

class sockaddr_in6(Structure):
    _fields_ = [
        ('sin6_len',        c_uint8),
        ('sin6_family',     c_uint8),
        ('sin6_port',       c_uint16),
        ('sin6_flowinfo',   c_uint32),
        ('sin6_addr',       c_uint8 * 16),
        ('sin6_scope_id',   c_uint32)
    ]

    def __str__(self):
        assert self.sin6_len >= sizeof(sockaddr_in6)
        data = ''.join(map(chr, self.sin6_addr))
        return socket.inet_ntop(socket.AF_INET6, data)
        
class sockaddr_dl(Structure):
    _fields_ = [
        ('sdl_len',         c_uint8),
        ('sdl_family',      c_uint8),
        ('sdl_index',       c_short),
        ('sdl_type',        c_uint8),
        ('sdl_nlen',        c_uint8),
        ('sdl_alen',        c_uint8),
        ('sdl_slen',        c_uint8),
        ('sdl_data',        c_uint8 * 12)
    ]
    
    def __str__(self):
        assert self.sdl_len >= sizeof(sockaddr_dl)
        addrdata = self.sdl_data[self.sdl_nlen:self.sdl_nlen+self.sdl_alen]
        return ':'.join('%02x' % x for x in addrdata)

class sockaddr_storage(Structure):
    _fields_ = [
        ('sa_len',      c_uint8),
        ('sa_family',   c_uint8),
        ('sa_data',     c_uint8 * 254)
    ]

class sockaddr(Union):
    _anonymous_ = ('sa_storage', )
    _fields_ = [
        ('sa_storage', sockaddr_storage),
        ('sa_sin', sockaddr_in),
        ('sa_sin6', sockaddr_in6),
        ('sa_sdl', sockaddr_dl),
    ]

    def family(self):
        return self.sa_storage.sa_family

    def __str__(self):
        family = self.family()
        if family == socket.AF_INET:
            return str(self.sa_sin)
        elif family == socket.AF_INET6:
            return str(self.sa_sin6)
        elif family == 18:  # AF_LINK
            return str(self.sa_sdl)
        else:
            print family
            raise NotImplementedError, "address family %d not supported" % family


class ifaddrs(Structure):
    pass
ifaddrs._fields_ = [
    ('ifa_next',        POINTER(ifaddrs)),
    ('ifa_name',        c_char_p),
    ('ifa_flags',       c_uint),
    ('ifa_addr',        POINTER(sockaddr)),
    ('ifa_netmask',     POINTER(sockaddr)),
    ('ifa_dstaddr',     POINTER(sockaddr)),
    ('ifa_data',        c_void_p)
]

# Define constants for the most useful interface flags (from if.h).
IFF_UP            = 0x0001
IFF_BROADCAST     = 0x0002
IFF_LOOPBACK      = 0x0008
IFF_POINTTOPOINT  = 0x0010
IFF_RUNNING       = 0x0040
if sys.platform == 'darwin' or 'bsd' in sys.platform:
    IFF_MULTICAST = 0x8000
elif sys.platform == 'linux':
    IFF_MULTICAST = 0x1000

# Load library implementing getifaddrs and freeifaddrs.
if sys.platform == 'darwin':
    libc = cdll.LoadLibrary('libc.dylib')
else:
    libc = cdll.LoadLibrary('libc.so')

# Tell ctypes the argument and return types for the getifaddrs and
# freeifaddrs functions so it can do marshalling for us.
libc.getifaddrs.argtypes = [POINTER(POINTER(ifaddrs))]
libc.getifaddrs.restype = c_int
libc.freeifaddrs.argtypes = [POINTER(ifaddrs)]


def getifaddrs():
    """
    Get local interface addresses.

    Returns generator of tuples consisting of interface name, interface flags,
    address family (e.g. socket.AF_INET, socket.AF_INET6), address, and netmask.
    The tuple members can also be accessed via the names 'name', 'flags',
    'family', 'address', and 'netmask', respectively.
    """
    # Get address information for each interface.
    addrlist = POINTER(ifaddrs)()
    if libc.getifaddrs(pointer(addrlist)) < 0:
        raise OSError

    X = namedtuple('ifaddrs', 'name flags family address netmask')

    # Iterate through the address information.
    ifaddr = addrlist
    while ifaddr and ifaddr.contents:
        # The following is a hack to workaround a bug in FreeBSD
        # (PR kern/152036) and MacOSX wherein the netmask's sockaddr may be
        # truncated.  Specifically, AF_INET netmasks may have their sin_addr
        # member truncated to the minimum number of bytes necessary to
        # represent the netmask.  For example, a sockaddr_in with the netmask
        # 255.255.254.0 may be truncated to 7 bytes (rather than the normal
        # 16) such that the sin_addr field only contains 0xff, 0xff, 0xfe.
        # All bytes beyond sa_len bytes are assumed to be zero.  Here we work
        # around this truncation by copying the netmask's sockaddr into a
        # zero-filled buffer.
        if ifaddr.contents.ifa_netmask:
            netmask = sockaddr()
            memmove(byref(netmask), ifaddr.contents.ifa_netmask,
                    ifaddr.contents.ifa_netmask.contents.sa_len)
            if netmask.sa_family == socket.AF_INET and netmask.sa_len < sizeof(sockaddr_in):
                netmask.sa_len = sizeof(sockaddr_in)
        else:
            netmask = None
            
        try:
            yield X(ifaddr.contents.ifa_name,
                    ifaddr.contents.ifa_flags,
                    ifaddr.contents.ifa_addr.contents.family(),
                    str(ifaddr.contents.ifa_addr.contents),
                    str(netmask) if netmask else None)
        except NotImplementedError:
            # Unsupported address family.
            yield X(ifaddr.contents.ifa_name,
                    ifaddr.contents.ifa_flags,
                    None,
                    None,
                    None)
        ifaddr = ifaddr.contents.ifa_next

    # When we are done with the address list, ask libc to free whatever memory
    # it allocated for the list.
    libc.freeifaddrs(addrlist)

__all__ = ['getifaddrs'] + [n for n in dir() if n.startswith('IFF_')]

As always, this code is released under a BSD-style license.

Euklas

2010-10-22T11:36:00.002-07:00

Carnegie Mellon University has produced a tool called Euklas (Eclipse Users' Keystrokes Lessened by Attaching from Samples). From the description:

Euklas enhances Eclipse's JavaScript editor to help users to more successfully employ copy-and-paste strategies for reuse.

I think I'm going to cry.

Python: Enumerating IP Addresses on MacOS X

2010-10-20T22:24:00.003-07:00

How do you enumerate the host's local IP addresses from python? This turns out to be a surprisingly common question. Unfortunately, there is no pretty answer; it depends on the host operating system. On Windows, you can wrap the IP Helper GetIpAddrTable using ctypes. On modern Linux, *BSD, or MacOS X systems, you can wrap getifaddrs(). Neither is trivial, though, so I'll save those for a future post.

Luckily, MacOS X provides a simpler way to get the local IP addresses: the system configuration dynamic store. Using pyObjC, which comes pre-installed on every Mac, we can write a straight port of Apple's example in Technical Note TN1145 for retrieving a list of all IPv4 addresses assigned to local interfaces:


from SystemConfiguration import *    # from pyObjC
import socket

def GetIPv4Addresses():
    """
    Get all IPv4 addresses assigned to local interfaces.
    Returns a generator object that produces information
    about each IPv4 address present at the time that the
    function was called.

    For each IPv4 address, the returned generator yields
    a tuple consisting of the interface name, address
    family (always socket.AF_INET), the IP address, and
    the netmask.  The tuple elements may also be accessed
    by the names: "ifname", "family", "address", and
    "netmask".
    """
    ds = SCDynamicStoreCreate(None, 'GetIPv4Addresses', None, None)
    # Get all keys matching pattern State:/Network/Service/[^/]+/IPv4
    pattern = SCDynamicStoreKeyCreateNetworkServiceEntity(None,
                                                          kSCDynamicStoreDomainState,
                                                          kSCCompAnyRegex,
                                                          kSCEntNetIPv4)
    patterns = CFArrayCreate(None, (pattern, ), 1, kCFTypeArrayCallBacks)
    valueDict = SCDynamicStoreCopyMultiple(ds, None, patterns)

    ipv4info = namedtuple('ipv4info', 'ifname family address netmask')

    for serviceDict in valueDict.values():
        ifname = serviceDict[u'InterfaceName']
        for address, netmask in zip(serviceDict[u'Addresses'], serviceDict[u'SubnetMasks']):
            yield ipv4info(ifname, socket.AF_INET, address, netmask)

One interesting point regarding this code is that it doesn't actually inspect interface information in the system configuration dynamic store. The interface-related keys are stored under State:/Network/Interface/, but this code (and Apple's example on which it is based) inspect keys under State:/Network/Service/ instead. However, if you want to get IPv6 addresses then you do have to inspect the system configuration's interface information:


from SystemConfiguration import *    # from pyObjC
import socket
import re
ifnameExtractor = re.compile(r'/Interface/([^/]+)/')

def GetIPv6Addresses():
    """
    Get all IPv6 addresses assigned to local interfaces.
    Returns a generator object that produces information
    about each IPv6 address present at the time that the
    function was called.
    
    For each IPv6 address, the returned generator yields
    a tuple consisting of the interface name, address
    family (always socket.AF_INET6), the IP address, and
    the prefix length.  The tuple elements may also be
    accessed by the names: "ifname", "family", "address",
    and "prefixlen".
    """
    ds = SCDynamicStoreCreate(None, 'GetIPv6Addresses', None, None)
    # Get all keys matching pattern State:/Network/Interface/[^/]+/IPv6
    pattern = SCDynamicStoreKeyCreateNetworkInterfaceEntity(None,
                                                          kSCDynamicStoreDomainState,
                                                          kSCCompAnyRegex,
                                                          kSCEntNetIPv6)
    patterns = CFArrayCreate(None, (pattern, ), 1, kCFTypeArrayCallBacks)
    valueDict = SCDynamicStoreCopyMultiple(ds, None, patterns)

    ipv6info = namedtuple('ipv6info', 'ifname family address prefixlen')

    for key, ifDict in valueDict.items():
        ifname = ifnameExtractor.search(key).group(1)
        for address, prefixlen in zip(ifDict[u'Addresses'], ifDict[u'PrefixLength']):
            yield ipv6info(ifname, socket.AF_INET6, address, prefixlen)

In fact, you could easily adapt the above function to be able to fetch IPv4 addresses from the interface configuration.

Caffeine Deficiency

2010-10-01T14:03:00.009-07:00

My wife has been complaining that she always feels nauseous after taking her daily vitamin. On the theory that it was perhaps a result of taking them first thing in the morning on an empty stomach, she tried taking them at night with dinner. I saw with my own eyes what she was talking about: she almost vomited.

So, I told her to start taking mine instead and I'd finish her vitamins off. Well, I'm not doing that again. I don't know what it is about One A Day Women's Active Metabolism vitamins, but they made me sick too. While we hate wasting the money, we just chucked the rest of the bottle because there was no way either of us were ever taking those again. Before we tossed the bottle in the trash, though, we were examining the label for differences between her One A Day Women's Active Metabolism and my One A Day Men's vitamins that might account for the nausea. In particular, we were looking for ingredients that were in higher concentration in the women's vitamins.

There are a few: vitamins D, K, B1, B2, B6, Calcium, and Iron to be exact. But I couldn't find any hard evidence that any of these vitamins could cause the sort of nausea we experienced (at least not in quantities we are likely to be exposed to). But that is when we noticed what makes the women's vitamins earn the "active metabolism" moniker:

Caffeine, and lots of it. Guarana seed is just another source of caffeine, so it is possible that its 50mg is included in the total of 120mg of caffeine in each "vitamin" pill. But if not, that means that 1 One A Day Women's Active Metabolism pill has more caffeine (~170mg) than a cup of coffee (100-150mg) and approaching that of a single Vivarin pill (200mg).

While neither of us drinks much coffee and we try to avoid caffeinated drinks, I doubt that 170mg of coffee in the morning would be enough to induce nausea. After all, millions of people have a cup of coffee in the morning and that is approximately the same amount of caffeine.

But, we never expected to have caffeine added to vitamin pills. I guess that is what we get for not having looked at the label before buying them. We certainly won't be making that mistake again.

All that background out of the way, this experience got me thinking: just how much caffeine do people consume everyday now? Here my wife was getting a base 170mg a day without even knowing it. The marketing for caffeinated products tends to emphasize that it helps you stay "mentally aware"; how long before being constantly-caffeinated is considered the norm and the symptoms of caffeine deficiency are "mentally sluggish"?

At what point does caffeine go from "booster" to baseline? Is there a point in our future wherein caffeine belongs in vitamins and even has a FDA-recommended daily allowance?

Getting the APNS device token

2010-09-27T19:20:00.004-07:00

As alluded to in my previous post, this time I'm covering how to get the APNS device token for a given iOS client. Actually, it is pretty straightforward. First, call registerForRemoteNotificationTypes from your application's didFinishLaunchingWithOptions UIApplicationDelegate callback. You need to specify which type of notifications your application will accept. Here is an example:

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{    
    ...

    // Register with the Apple Push Notification Service
    // so we can receive notifications from our server
    // application.  Upon successful registration, our 
    // didRegisterForRemoteNotificationsWithDeviceToken:
    // delegate callback will be invoked with our unique
    // device token.
    UIRemoteNotificationType allowedNotifications = UIRemoteNotificationTypeAlert
                                                  | UIRemoteNotificationTypeSound
                                                  | UIRemoteNotificationTypeBadge;
    [[UIApplication sharedApplication] registerForRemoteNotificationTypes:allowedNotifications];

    ...
    
    return YES;
}

As mentioned in the code comments, the application will then talk to Apple's Push Notification Service in the background and, when the device's unique token has been issued, your application delegate's didRegisterForRemoteNotificationsWithDeviceToken callback is invoked. This is where you actually get the device token.

- (void)application:(UIApplication *)application
        didRegisterForRemoteNotificationsWithDeviceToken:(NSData *)deviceToken
{
    NSLog(@"didRegisterForRemoteNotificationsWithDeviceToken: %@", deviceToken);
    // Stash the deviceToken data somewhere to send it to
    // your application server.
    ...
}

Once your iOS client application knows its own device token, it needs to send it to your application server so that the application server can push notifications back to the client later. How you do this depends on the architecture of your client-server communication.

At this point, I should add the device tokens can change. So, I recommend repeating the above logic every time your iOS client application starts so your application server always gets the latest device token for the user's terminal.

I would be remiss to not mention error handling. It is possible that the registerForRemoteNotificationTypes call will fail. The most obvious way it could fail is if the user does not have access to the 3G or WiFi networks and, as a result, cannot communicate with Apple's Push Notification Service; for example, when the device is in Airplane Mode with the wireless signals turned off.

In this case, the didFailToRegisterForRemoteNotificationsWithError delegate callback is invoked instead of didRegisterForRemoteNotificationsWithDeviceToken. In which case, you probably want to retry the registration later when network connectivity is restored.

Using pyapns with django

2010-09-20T14:37:00.008-07:00

There is a handy daemon for sending push notifications to iOS-based mobile clients via Apple's Push Notification Service; it is called pyapns. It is implemented in python but, since it runs as a standalone XML-RPC server process, that fact is largely irrelevant. The important facts are that:

It properly and fully implements the client interface to APNs, including the requirement for maintaining a persistent connection with Apple's servers rather than repeatedly setting-up and tearing-down SSL connections.
It includes client libraries for communicating with the pyapns daemon from python and ruby, although any language that can speak XML-RPC (including C) will work too.

The way it works is: you first start the pyapns daemon process. This process acts as a XML-RPC server handling requests from your application(s), packing them into Apple's binary APNS protocol, and sending them to Apple to deliver to the iPhone, iPad, or iPod.

In order for your applications to send a push notification request, though, they must first tell pyapns which client certificate it should use to authenticate with the APNS servers. Here is a decent guide on how to obtain a client certificate. Once you have a certificate, you have to use the pyapns client library's configure and provision APIs to tell the pyapns daemon process to use your certificate.

If you are implementing your application in django, you can accomplish the configuration and provisioning directly from your django settings.py file like so:

# Configuration for connecting to the local pyapns daemon,
# including our certificate for pushing notifications to
# mobile terminals via APNS.
PYAPNS_CONFIG = {
    'HOST': 'http://localhost:7077/',
    'TIMEOUT': 15,
    'INITIAL': [
        ('MyAppName', 'path/to/cert/apns_sandbox.pem', 'sandbox'),
    ]
}

The pyapns python client library will automatically configure and provision itself from these settings. So, assuming you know the APNS device token of the mobile device you want to send a notification to, all you need to do to send a push notification is to call the pyapns.client.notify() function.

If only it were so easy. One complication arises in that the pyapns provisioning and configuration state is split between the client library and the pyapns daemon process. As a result, there are two scenarios to be wary of:

The django application is restarted. In this case, the client library, which is part of your django application, loses its state and tries to re-configure and re-provision itself from your django settings. Luckily, since the client library will re-read the configuration and provisioning settings from settings.py and seamlessly resume communication with the pyapns daemon.

However, as noted in the pyapns documentation, "attempts to provision the same application id multiple times are ignored." As a result, if you change pyapns configuration in the settings.py file and restart django, you need to restart the pyapns daemon too for the new settings to take effect. Otherwise, if the settings are unchanged, the client library will seamlessly resume communication with the pyapns daemon.
The pyapns daemon is restarted. In this case, the client library thinks it has already configured and provisioned the daemon, but the daemon has lost this configuration due to restart. As a result, any attempt to send a push notification will fail as the daemon does not know how to establish the connection with Apple's Push Notification service.

As I mentioned above, the first scenario isn't a big deal. If you have to restart your web application or the web server for some reason, the connection between the pyapns client library and the daemon process will automatically resume right where it left off. In the rare case that you changed the pyapns settings in your django settings.py file, you need to restart both the django application and the pyapns daemon process for the new settings to take effect.

The latter scenario, though, is a bigger problem because it is impossible to detect until it is too late: that is, it doesn't manifest itself until you try to send a push notification and fail. Luckily, however, we can catch the failure condition and resolve the problem automatically. Specifically, if the pyapns client library fails to send a push notification to the daemon process due to the daemon process not being configured or provisioned, we can force the client library to re-configure and re-provision and retry.

So here you go, a wrapper around the pyapns client library to automatically recover when the backend pyapns daemon has been restarted:

"""
Wrappers for the pyapns client to simplify sending APNS
notifications, including support for re-configuring the
pyapns daemon after a restart.
"""

import pyapns.client
import time
import logging

log = logging.getLogger('APNS')

def notify(apns_token, message, badge=None, sound=None):
    """Push notification to device with the given message

    @param apns_token - The device's APNS-issued unique token
    @param message - The message to display in the
                     notification window
    """
    notification = {'aps': {'alert': message}}
    if badge is not None:
        notification['aps']['badge'] = int(badge)
    if sound is not None:
        notification['aps']['sound'] = str(sound)
    for attempt in range(4):
        try:
            pyapns.client.notify('MyAppId', apns_token,
                                 notification)
            break
        except (pyapns.client.UnknownAppID,
                pyapns.client.APNSNotConfigured):
            # This can happen if the pyapns server has been
            # restarted since django started running.  In
            # that case, we need to clear the client's
            # configured flag so we can reconfigure it from
            # our settings.py PYAPNS_CONFIG settings.
            if attempt == 3:
                log.exception()
            pyapns.client.OPTIONS['CONFIGURED'] = False
            pyapns.client.configure({})
            time.sleep(0.5)

Since I glossed over it in this post, I'll cover how to get the APNS device token for a mobile device in my next post. The device token acts as an address, telling Apple's Push Notification service which mobile device it should deliver your notification message to.

Calculating degree deltas for distances on the surface of the Earth

2010-09-07T19:34:00.010-07:00

Here is the scenario: you've got your GPS coordinates (latitude & longitude in degrees) of your current position and you want to find the number of degrees north/south and east/west needed to contain an area of some size around you. I know, this sounds contrived, but it comes up if you use the iPhone's MapKit framework and want to zoom a MKMapView to a level where only a certain distance around a location is displayed. In that case, you need a MKCoordinateRegion to pass to the setRegion:animated: method.

One might think that if you know the rate of conversion between meters (or if you prefer, miles) and degrees, this would be a straightforward conversion. The problem is that it isn't so simple. Since the earth is a sphere, the number of meters in one degree of longitude depends on your latitude. For example, at the equator, there are 111.32 kilometers per degree of longitude; at the poles, however, there are 0 meters per degree.

A MKCoordinateRegion is comprised of two components: the coordinates of the center of the region and a span of latitudinal and longitudinal deltas. Let's assume the center is known; for example, it could be your user's current location. To calculate the span, here is a simple function that takes into account the curvature of the earth:

/*!
 * Calculate longitudinal and latitudinal deltas in
 * degrees for the given linear horizontal and vertical
 * distances in kilometers.  Longitudinal degrees per
 * kilometer vary with latitude, so a coordinate is
 * needed as a frame of reference.
 *
 * @param coord - point of reference.
 * @param xDistance - east-west distance in kilometers.
 * @param yDistance - north-south distance in kilometers.
 * @return MKCoordinateSpan representing the distances
 *         in degrees at the given coordinate.
 */
static
MKCoordinateSpan
spanForDistancesAtCoordinate(CLLocationCoordinate2D coord,
                             double xDistance,
                             double yDistance)
{
    const double kilometersPerDegree = 111.0;

    MKCoordinateSpan span;
    
    // Calculate the latitude and longitude deltas that
    // correspond to the distance (in kilometers) at
    // the given coordinate.  Note that the longitude
    // degrees calculation is complicated by virtue of
    // the fact that the number of meters per degree
    // varies depending on the coordinate's latitude.
    span.latitudeDelta = xDistance / kilometersPerDegree;
    span.longitudeDelta = yDistance / (kilometersPerDegree * cos(coord.latitude * M_PI / 180.0));
    return span;
}

The user's current location, which will become the center of the MKCoordinateRegion, should be passed as the point-of-reference coord argument. This is used to calculate the number of meters per degree at the user's current latitude.

The xDistance and yDistance parameters are the number of kilometers east-west and north-south, respectively, that define the region.

Note that the constant kilometersPerDegree represents the number of kilometers per degree of latitude or the number of kilometers per degree of longitude at the equator; it is only an estimate. Since the Earth isn't a perfect sphere, the actual number varies, but for the sake of most iPhone apps, the estimate of 111.0 kilometers/degree should be sufficient.

RFC 3339-compliant Unicode date format pattern

2010-08-17T16:32:00.005-07:00

Here is a quick note just to say that if you need to generate RFC 3339 timestamps or ISO 8601-compliant combined date/time representations, here is the Unicode date format pattern to do so:

yyyy-MM-dd'T'HH:mm:ss.SSSSZ

This could come in handy if, for example, you are using Apple's NSDateFormatter class. NSDateFormatter has no predefined style corresponding to RFC 3339 / ISO 8601 format so you'll need to use a format specifier string instead; NSDateFormatter format strings comply with the Unicode date format patterns. So you can use the format pattern above to parse or output strings that can be exchanged with other RFC 3339 / ISO 8601 compliant systems.

For example, the following Objective-C code will print out the current timestamp as an RFC 3339 / ISO 8601 compliant combined date/time string:

NSDateFormatter *rfc3339 = [[NSDateFormatter alloc] init];
[rfc3339 setDateFormat:@"yyyy-MM-dd'T'HH:mm:ss.SSSSZ"];

NSDate *now = [NSDate date];
NSLog(@"%@", [rfc3339 stringFromDate:now]);
[rfc3339 release];

At the time of this writing, the output looked like "2010-08-17T16:38:21.9640-0700".

HTML encoding of form inputs

2010-07-16T16:32:00.008-07:00

I suppose this is common knowledge amongst professional web developers but I just discovered myself that if a user enters characters into a HTML form input that is not representable in the character set of the page the form is in, browsers will HTML-encode the non-representable characters when the form is submitted. I just spent over an hour assisting a coworker to track down a bug in one of our web applications that was due to this poorly-documented -- but reasonable -- behavior.

I say "reasonable" because, as obscure as it is, this is really the best thing I think a browser can do given the situation.

To recap, here is the scenario:

You have a web page with a form in it that is served using some locale-specific encoding. In our case it was Shift-JIS, but the default ISO8859-1 encoding leads to the same problem.
The user enters text into a form input field that is not representable in the displayed page's character set or encoding. For example, entering Cyrillic characters into a form displayed on an ISO8859-1 page.
When the user submits the form, the browser tries to convert the inputs to the encoding of the page. Any character not representable in that page's character set or encoding has its Unicode character code point encoded as an HTML numeric character reference (e.g. Ǆ).
The web application or CGI receiving this input needs to a) know the character encoding of the page that was used to submit the form data so it knows how to interpret the data as characters and b) be prepared to convert any embedded HTML numeric character references back to their corresponding characters.

I like that last part where web applications (or CGIs) have to know the encoding of an HTML page served to the client in order to be able to properly parse input from that client. This fact shatters any remaining fantasies I had of HTTP being stateless.

Anyway, the real surprise is that a web application or CGI needs to be prepared to unencode HTML entities in form input. I quick check of perl's CGI.pm and python's cgi module indicates that neither of them do entity decoding of inputs automatically. And considering that information on the web regarding this behavior is sparse , I suspect that most web developers are unaware of it. At the time of writing, I can only find two references [1][2] that document HTML character reference encoding in the scenario described above.

Luckily, there is a really simple solution: always serve pages in UTF8 encoding and always expect form input to be in UTF8 encoding. One of the many great things about UTF8 encoding is that all characters are representable, so you never have to worry about the browser resorting to HTML character reference encoding.

Sharp pointy sticks

2010-07-12T18:10:00.003-07:00

This past Saturday my wife and I bought recurve bows with the intent to make a hobby out of archery. It turns out there are a number of parks with free-to-the-public archery ranges in the Bay Area. A couple of the ranges we have found are:

Huddart County Park in San Mateo county near Woodside.

Stevens Creek Park near Cupertino

We actually got started a couple of weeks ago when we went for a free lesson from the Kings Mountain Archery Club. That was a lot of fun and our instructors were very friendly, helpful, and patient. They give free two-hour introductory lessons about once a month; information about how to signup is available on their web site.

We picked up our bows and arrows at a shop intimidatingly called Predator's Archery down in Gilroy. Much like our experience with Kings Mountain Archery, the staff at Predator's Archery were pretty friendly and helpful and they offer a great "starter package" which includes everything you need as well as 5 lessons.

In other news: Subversion still sucks

2010-07-02T18:08:00.004-07:00

OK, I've had 4 months to get used to Subversion now. And it is growing on me. Or perhaps it is Stockholm Syndrome. But there are still a lot of annoyances.

After being bashed as a "troll" by one of Subversion's authors after daring to suggest it wasn't all ponies and rainbows, I thought I would check to see if others where sharing my pain transitioning from CVS to Subversion.

Not surprisingly, I did. I found a wonderful summary of all the frustrations I've been experiencing, thoughtfully compiled by no less than David O'Brien of the FreeBSD community.

I would add to his list, as my friend John pointed out in comments to my previous post, that it is really annoying to have to depend on external tools (ironically, CVS) to see commits across branches.

Serving file downloads with non-ASCII filenames

2010-07-01T11:31:00.008-07:00

Recently, while helping out one of my coworkers, it came to my attention that there is no universally-agreed on way to download a file to a web browser while suggesting a filename that contains non-ASCII characters.

The common way to tell a browser to download a file (rather than try to display it in-browser) is to include a Content-Disposition header in the HTTP response; the header's value should be "attachment". Additionally, the server can include a filename parameter in the Content-Disposition header as a suggestion to the browser for what filename to save the file as.

As a bit of history, the Content-Disposition header was originally defined in RFC 1806 which was obsoleted and replaced by RFC 2183. However, the Content-Disposition header was originally defined for use in MIME messages and, while RFC 2616 (HTTP 1.1) makes reference to the Content-Disposition header, it does so only to note that:

Content-Disposition is not part of the HTTP standard, but since it is widely implemented, we are documenting its use and risks for implementors.

Luckily, while not officially standardized for use in HTTP, the Content-Disposition header is "widely implemented" indeed; it seems that all modern browsers implement the header. If the web server responds to a request with application/octet-stream data and a Content-Disposition header of "attachment", your browser will display the familiar "Save As..." dialog. If the server included a filename parameter in that Content-Disposition header, your browser will likely pre-fill the filename input field of the "Save As..." dialog with the specified filename.

But here is where things start getting murky.

RFC 2183, skirts the issue of international filenames by disclaiming responsibility:

Current [RFC 2045] grammar restricts parameter values (and hence Content-Disposition filenames) to US-ASCII. We recognize the great desirability of allowing arbitrary character sets in filenames, but it is beyond the scope of this document to define the necessary mechanisms.

So, as long as the downloaded files' names are always representable in the ASCII charactet set, any browser should properly display the filename (although I've seen rumors that some browsers, such as IE, do enforce a limit on the length of the filename). However, I work at a Japanese company, making products largely for the Japanese market, so we don't have the privilege of assuming the whole world is ASCII.

By the way, in case you are curious, even iso-8869-1 (latin1) isn't consistently supported across browsers so Europeans are left high-and-dry too.

You are probably thinking, like I was, that surely this is a solved problem. And actually, it is. Kind of. The Content-Disposition header originates with the MIME protocol which, since the publication of RFC 2231 in 1997, now supports non-ASCII character encodings for header values. So, for example, the filename "foo-ä.html" can be represented in the Content-Disposition header like so:

Content-Disposition: attachment; filename*=UTF-8''foo-%c3%a4.html

The problem is that few browsers actually implement this RFC 2231 syntax. For example, Firefox 3.6 and Opera 10 appear to support the RFC 2231 syntax. On the other hand, for Internet Explorer, Microsoft's developers choose to simply perform URL-style percent-decoding and then interpret the result as bytes of UTF8-encoded characters. So a server would need to send the Content-Disposition header as

Content-Disposition: attachment; filename="foo-%c3%a4.html"

for an MSIE user to see "foo-ä.html" in the "Save As..." dialog.

Despite requests for IETF working group members to fix it, Google's Chrome browser also does not comply with RFC 2231, preferring to follow Microsoft's lead and use simple URL-style percent decoding.

As a result, there is no consistent cross-browser way to suggest a non-ASCII filename for a file download. I'm sure it doesn't help that the Content-Disposition header has never formally been part of the HTTP specification, but yet it is used by all major browsers to implement file download functionality.

Julian Reschke has compiled a test suite and publishes a nifty page illustrating all of incompatibilities between browsers regarding handling of the Content-Disposition header. In addition, as part of the IETF Network Working Group, he is working on an RFC to formally define the interpretation of the Content-Disposition header in the HTTP context.

Unfortunately, because the ambiguity has been left unresolved for so long, some web servers have adopted the MSIE/Chrome encoding technique for their non-ASCII filenames. Actually, my gut feeling is that probably most have, although I don't have any hard numbers to back up that claim. The good news is that since the MSIE/Chrome encoding is only used for parameters in the form filename="..." while the RFC 2231-style encoding used by Firefox, Opera, and Julian's proposal uses filename*=... it is possible for the two to coexist in the same Content-Disposition header (note the presence of the * in the RFC 2231 format to differentiate it).

In fact, probably the most important section of Julian's proposal is section 4.2 where defines the HTTP client's behavior when the server responds with both filename=... and filename*=..., allowing for an easy upgrade path for MSIE and Chrome.

For now, however, Julian's test results show that when presented both traditional and extended formats, only Firefox and Opera will select the extended filename*=... format.

This opens an opportunity for those of us that need to serve file downloads containing non-ASCII filenames: we can include the filename in non-standard encoding supported by MSIE and Chrome first in the Content-Disposition header, followed by the filename in extended RFC 2231 encoding. According to Julian's tests, MSIE and Chrome will always take the first parameter while Firefox and Opera will properly selecte the extended-syntax parameter, no matter what order it appears in.

For example, if the server includes the header:

Content-Disposition: attachment; filename="foo-%c3%a4.html" filename*=UTF-8''foo-%c3%a4.html

all four major browsers should properly display the filename "foo-ä.html" in the "Save As..." dialog. Unfortunately, WebKit-based browsers, like Apple's Safari browser, would display the raw percent-encoded value "foo-%c3%a4.html" as the filename. At least for now, though, I'm afraid this is the best we can do.

Sharon Jones and the Dap Kings

2010-06-28T09:37:00.006-07:00

This past Friday we had the chance to see Sharon Jones and the Dap Kings at the Warfield in San Francisco. The music was fun and Sharon was an amazing performer and entertainer. In addition to her own high-energy dancing around the stage, she invited a few of the audience onto the stage to dance with her. This was sort-of a crap-shoot as at least one person's dance skills ranked somewhere between Elaine from Seinfeld and epileptic seizure victim. However, one guy who referred to himself as merely "The Gator" got up on stage in full 70's era leisure suit and that boy could dance. Sharon even turned the spotlight over to him and let him go to town.

I'll be damned if someone didn't manage to get a video of it and has already uploaded it to Youtube:

The opening act was The Heavy, whose is perhaps best known for their song "How Do You Like Me Now?" that Kia uses in car ads featuring bad-ass stuffed animals. Except for the horns, I wouldn't have thought that they would have been a good match for Sharon Jones and the Dap Kings. But they were another high-energy act that really got the crowd riled up for the main act.

Overall it was a great show. We had some punk kids causing trouble in front of us, but eventually security took care of them and we were able to enjoy the rest of the show.

Experiences with Clipper

2010-06-22T11:16:00.004-07:00

While I was away in Japan, Caltrain and the other Bay-Area transit agencies teamed up to introduce the ~~Translink~~ Clipper payment system. I was excited at the prospect of an easy-to-use IC-card payment system like the Suica system I enjoyed using in Tokyo, and I was tired of having to buy new 8-ride tickets every few days, so I got a Clipper card and tried it out.

The card is easy enough to use, but the experience refilling my balance on the card has been less than stellar. I'm not about to tie my Clipper card to my credit card account, effectively turning it into a backdoor for a thief to clean me out (the "autoload" feature they push on their website). I was hoping to be able to refill it manually using an automated machine, just like I always did with my Suica card.

It turns out that there are a whopping 7 "Add Value" machines in the entire Bay Area: two in San Rafael, one in Sausalito, one in Oakland, two in downtown San Francisco, and one on the freaking Golden Gate Bridge (I guess for the toll-booth workers?). Fortunately, for those of us whose commute doesn't include one of those 7 locations, you can also add to your Clipper card balance at most Walgreens locations.

And here is where my experience with Clipper takes a nose-dive. First, not all Walgreens locations are equipped to add money to your Clipper card balance. There is, however, a map of the locations on the Clipper website. Unfortunately for me, the Walgreens one block from my office in San Mateo is not on the list and, when asked in person, I received the same look one would expect had I asked them if sold live Jabberwockies.

So, each time I need to refill the balance on my card, I hop in the car and drive two miles to the Walgreens nearest my home. Two miles isn't particularly far, but it does feel a little odd to have to drive somewhere so I can pay for public transportation. Anyway, I only have to do it once every couple of weeks, so it isn't a big deal; just a minor annoyance.

What is more annoying about the experience is that apparently only managers have the specialized training necessary to work the Clipper card "add value" device they keep behind the counter. So the poor checkout clerk has to page for the manager who, after finishing his dooby, mosies up to the front counter and fumbles with the machine until (hopefully) it actually has the balance on it I paid for. This past time, my wife and I had to repeatedly tell other customers that they might want to get in the other line because we were going to be a while. Ten minutes in fact.

The good news is that, with the name change from Translink to Clipper, it looks like they have added an option to refill your card balance online. I'm planning on trying that the next time I need to update my balance. I might miss the biweekly visit to the bloodshot-eyed Walgreens manager, though.

Crappy Code Hopscotch

2010-05-03T18:30:00.004-07:00

I'm officially coining the term "crappy code hopscotch" to refer to the stupid games you have to play to workaround crappy code. I guess it could equally well refer to that feeling of being surprised by the effects of crappy code in any otherwise simple task, which might not be altogether unlike the feeling of unpleasant surprise you would get if someone were to throw a pile of dog poo in your hopscotch square.

The term popped into my head today while doing some MySQL wrangling; I was testing a stored function that called LOWER() on the results of CONCAT_WS(). Sounds simple enough: lower-case the result of concatenating strings with a separator. Check this output from MySQL 5.1:


mysql> SELECT LOWER(CONCAT_WS(' ', 'MySQL', 'scores', 'a', 0));
+--------------------------------------------------+
| LOWER(CONCAT_WS(' ', 'MySQL', 'scores', 'a', 0)) |
+--------------------------------------------------+
| MySQL scores a 0                                 |
+--------------------------------------------------+

Silent lower-case fail.

The problem, it seems, is that CONCAT_WS() doesn't convert the numeric argument to a string, but rather decides to convert *all* of the parameters to BINARY types and, as a result, returns a BINARY value. To it's credit, at least LOWER() is documented as being a no-op on BINARY values, hence the useless output shown above. What amazed me is that not just the undocumented, unintuitive behavior of CONCAT_WS() but that MySQL did not emit a single warning when LOWER() returned a value without, you know, lower-casing it.

So out of nowhere, I find myself playing crappy code hopscotch. I can explicitly either cast the numeric argument to CONCAT_WS() to a string or else let CONCAT_WS() return a BINARY value and explicitly convert that back to a string before passing it to LOWER().

Two crappy boxes to pick from and I got to put my foot in one of them.

Invent with Python

2010-04-22T21:13:00.005-07:00

Al Sweigart gave a presentation of his book, Invent Your Own Computer Games with Python, to Baypiggies tonight. His book is aimed at kids that are interested in learning how to program. He said he didn't have a particular age range in mind, but I would say from experience it would probably be fine for anyone age 8 to 15 with an interest in computers.

I was impressed with the overall tone and layout of his book. And his choice of teaching programming via writing simple computer games is right on the money. He mentioned that what got him hooked on programming was tinkering around creating simple games when he was kid; I'd venture that was what got a great many of the best engineers I've met started. In addition, he choose to teach programming concepts via Python, which I also agree is a great language for learning because it is expressive, easy to understand, and yet powerful to build professional applications.

In his presentation, Al accurately pointed out that there is a trend to try and simplify programming for kids until it resembles building with duplo blocks and that really isn't helpful for kids nor interesting for them. I concur enthusiastically. Projects like Scratch are neat, but seem patronizing to me. I learned on BASIC and Pascal and I don't doubt kids today are just as capable. That said, BASIC and Pascal are dated now; python is just as easy to learn yet more powerful and modern so it is a great choice.

Another difference between Al's book and many intro books is that his programs are short, fun, and mostly text. Of course, every kid dreams of writing graphical games like the video games they play but that is, frankly, not realistic. Again, Al doesn't lie to his audience; he presents fun text-based games that kids can tinker with. He starts with a simple guessing game, then hangman, tic-tac-toe, and othello. Towards the end of the book, he does introduce pygame and shows how to use it to make simple graphical games, but the vast majority of the book focuses on teaching fundamental concepts via text-based games.

I actually taught introductory computer programming to high school students, age 14 and 15, a number of years ago. Back then we used C, but I was surprised to find that Al presents software concepts in the same order I did and even uses the same games to drive those concepts home. If I could teach the same class today in python, Al Sweigart's Invent Your Own Computer Games with Python is the book I'd want to use.

Overall I was quite impressed with the great job Al did with the book and appreciate him taking the time to talk about it, and the process of writing it, to us at Baypiggies tonight.

P.S. I should mention that he has published the book under the Creative Commons license so it is free to read; you can even download the latest edition off his website. Amazon sells it in dead-tree format too, but you should hold off because the second edition will be going to print soon.

Good Day

2010-04-19T12:29:00.002-07:00

All of our sea mail from Japan arrived and was just delivered. In addition, my new MacBook Pro I ordered last week (right after the revision bump) arrived today too. Looks like I'm going to be busy this week. I just hope I can find time to make it to BayPiggies this Thursday.

Subversion sucks

2010-04-16T16:14:00.003-07:00

While I was away in Tokyo, NTTMCL switched from CVS to Subversion for their version control system. Perhaps it is just that I'm too accustomed to CVS's eccentricities, but so far I have to say the Subversion sucks. While I'm sitting here waiting for the checkout of one of our heavily-branched repositories to complete (45 minutes and counting!), I took the opportunity to read a little about how much more wonderful Subversion is than CVS.

So far, the best I've come up with is that Subversion is newer, therefore it is better. Yay. With subversion, I just get the delight of knowing I'm playing with a fresh(er) turd.

Sure, CVS sucks too. What bothers me about Subversion is that it sucks at least as much as CVS without giving anything in return. At least with CVS, I can tag a release or create a branch without having to make a whole other copy of the repository (on each developer's machine, no less!). At least with CVS, I can diff and merge files between branches or tags without developing a Repetitive Strain Injury. At least with CVS, the repo files are text so I can recover when it screws up. At least with CVS, I don't have to run a friggin' web server just to do revision control.

So, yeah, maybe I'm just an old fogey. Or maybe Subversion sucks so much, it actually makes me long for CVS. Wouldn't that be sad.

Help me, I'm in RPM Hell

2010-04-15T16:41:00.008-07:00

Dear Internet,

Surely there must be a better way to install packages on linux. For better or worse I've been locked in the FreeBSD ivory tower for the better part of 10 years. But now I am assigned to a project that is using linux and I find myself yearning for BSD-style package management. Specifically, I am looking for:

A way to find a package for my OS (in this case, Red Hat Enterprise Linux 5, 64-bit).
Download that package and the packages for all of its dependencies.
Install a package and all of its dependencies onto a host that is not connected to the Internet.
Is command-line based.

In FreeBSD, you can use the FreeBSD packages database to locate the desired package and download it to your system in a format that you can install offline. Getting all of the dependencies it a little trickier, but I wrote a simple perl script some years ago that does that using the FreeBSD ports collection.

Obviously linux has the rpm format for its packages but I'm finding that searching for rpms and identifying all of their dependencies is just as manual of a process as it was 10 years ago. For example, the yum command can install a package and its dependencies, but does not seem to support downloading the rpms for those packages to reproduce the process on an offline machine. Surely there must be a way.

If anyone could point me to a tool for linux that satisfies the 4 goals above, I would be grateful.

P.S. In a case of "it's a small world after all", I went to elementary school through high school with the author of yum, Seth Vidal.

Soil and "Pimp" Sessions

2010-03-29T17:07:00.004-07:00

Soil and "Pimp" Sessions is a Japanese jazz band that I've taken quite a liking to. I first learned about them when they released a single with Shina Ringo called "カリソメ乙女" (Karisome Otome) back in 2006. They have had a number of videos on YouTube which I enjoyed, but I just couldn't get my hands on one of their albums. It doesn't look like they have a distributor in the U.S. and none of the record stores I visited in Japan stocked them either.

So you can imagine my surprise and delight when iTunes started carrying their albums around November of last year. The U.S. iTunes store even. I've since stocked up bought 20 or so songs of theirs and have yet to be disappointed. The best I can tell, the U.S. iTunes store is selling all of their albums.

To top it all off, Shina Ringo's latest album, 三文ゴシップ (sanmon gossip) not only includes the 2006 Karisome Otome single, but includes another awesome collaboration　with Soil and Pimp Sessions: マヤカシ優男 (Mayakashi Yasaotoko). There isn't a video for it, but you can preview the song on YouTube. Don't worry, the lyrics for this song are all in English.

Even though they are a Japanese band, since it is jazz, most of the songs have no lyrics. That said, What few words there are do tend to be English. Which makes them a very approachable band to international audiences. In fact, right now they are in the middle of a European Tour. They have a live show schedule on their web site; I know I'll be watching to see when they come to San Francisco.

Back in California

2010-03-29T17:00:00.003-07:00

Well, we're back in California and getting settled in. I had a month off to get moved out of our place in Tokyo and moved into our new place in Mountain View. It has been a hectic month but the last few days have been relaxing. That said, I'm ready to get back to work.

Anyway, I've got a whole backlog of things I've been meaning to blog about so hopefully I'll be a little more frequent with the updates for a while.

No place like home

2010-03-04T05:12:00.003-08:00

Well, I've wrapped up my time here in Tokyo and preparing to move back to California. Barring a few small obstacles -- like just noticing that our passports had expired and frantically visiting the U.S. Embassy to get new ones -- we should be back in the U.S. by this time next week.

My coworkers gave me a nice farewell party which was quite fun. I'm not sure why, but I got pretty choked up on my last day at the office. I'm sure part of it was the relief of finally going home, but a lot of it was that I really liked my coworkers. They were the best part of working in Japan.

This past week I've had more time and finally gotten a chance to see more of the neighborhood where we have been living. It is actually a pretty nice area. It is a shame that it took over 2 years before I could actually enjoy it. Nonetheless I'm really looking forward to going home.

Yattsuke Shigoto

2010-02-09T02:30:00.007-08:00

My friend Matt introduced me to a Japanese singer/songwriter named Shiina Ringo a few years back. I've taken quite a liking to her music, but one of her songs has really grown on me since I've been in Tokyo. It is a song called 「やっつけ仕事」(Yattsuke Shigoto), which means a job done half-assed. For me, the lyrics reflect the general feeling of apathy and soullessness that Tokyo seems to emanate. I guess it may be different for natives, but this song speaks to me. It speaks to me every time I get on the train and stair blankly off into the distance or pretend to sleep like everyone around me.

Feeling down today, I thought I would check to see if there were any English translations of the lyrics. All I could find was one terrible almost-literal translation that killed the voice of the song. So I whipped up a translation of my own (below).

I'm pretty confident that this translation captures the spirit of the song, even if I did stray from the literal translation by a wide margin on a couple of lines. The only line I'm not happy with is the one about the high-speed traffic jam. In Japan, if there is a traffic jam on a highway (which is literally called a "high-speed road"), they just say high-speed (road) traffic-jam, omitting the word "road". Hence producing the contradictory, "high-speed" traffic jam.

The only other part I'm not satisfied with at the pronouns. Japanese doesn't have pronouns and it isn't clear who is saying what in the song. I just assumed that everything was being said from a first-person perspective, since that is normally the case in most songs.

In case you are interested, the original Japanese lyrics are here. There are a couple versions of the song on iTunes also.


Yattsuke Shigoto - Shiina Ringo

Every day I'm assaulted by the ring of phones
I just want some peace and quiet

You call it a "high-speed" traffic jam, but isn't it slow?
I'm indifferent to reasoning contrary to reality

I can't think of anything good
But I'm not indignant about anything either
What day was it today?
I guess it doesn't much matter
Ah, all I want is something memorable

This consistency wears down my individuality
Maybe I'll just get an arranged marriage

Please control me
I hate boredom
When's the last Ginza-line train?
I guess it's not a big deal
Ah, I wish I could be a machine

Hey, what was "love" again?
I can't remember
I can't remember

I can't think of anything good
But I'm not indignant about anything either
What day was it today?
I guess it doesn't much matter
Ah, all I want is something memorable

Please control me
I hate boredom
When's the last Ginza-line train?
I guess it's not a big deal

I can't think of anything good
I can't think of anything good
I can't think of anything good

Hey, what was "love" again?

When Standards Collide

2010-02-07T17:46:00.004-08:00

I can't help but shake my head in disappointment every time I run across some specification that is blatantly in violation of the standards it is built on. When I encounter a specification published by an ostensibly-reputable professional organization that contradicts my understanding of the underlying protocols, at first I'm confused. Is my understanding faulty? My memory bad (hint: it is)? Is there a new RFC that updated the protocol when I wasn't looking?

As a recent example, I finally got my hands on the WRIX Interconnect 1.03 specification from the Wireless Broadband Alliance. In which, they specify that partner ISPs must support "passwords up to 253 characters which contain a mixture of alphabetic, numeric, and special characters"; this is written in the comments for the the User-Password attribute in a RADIUS Access-Request packet.

There are two problems with this text that I'm amazed industry professionals did not correct (assuming they noticed):

RFC 2865 section 5.2 clearly states that the maximum length of the User-Password attribute is 130 octets, including the 2-byte attribute header. In other words, the encrypted password text cannot be longer than 128 octets. If this were the only issue, I'd be inclined to believe that the 128 octet limit has been relaxed in a later RFC.

Which leads to the second issue: RFC 2865 says the *encrypted* password text cannot be longer than 128 octets. Section 5.2 also lays out the encryption algorithm, which is a 16-byte block cipher. Being a block cipher, the password plaintext is padded out to a multiple of 16 bytes before the encryption is applied. Which means that a 7 octet password will encrypt to 16 octets of encrypted text. The 253 character password specified by WRIX would encrypt to 256 octets of encrypted text. RADIUS allocates 8 bits per attribute to represent the length of that attribute in octets, including the attribute's 2-byte header. So, if you have 256 octets of encrypted text, the total length of the attribute would be 258 octets.

So, in order to comply with the WRIX Interconnect 1.03 specification, it would appear you are required to violate both RFC 2865 and physics (to get 258 values represented with just 8 bits).

What baffles me is that I would have expected the difficulty of implementing the specification as written would have become obvious in compatibility testing. Surely someone has written unit tests based on the spec to test their implementation. At least once. Right? Right?

Anyway, I imagine the intent of the specification's authors was only to violate RFC 2865 and leave the violation of physics to the hardware guys. It looks like they wanted partner ISPs to allow the longest password representable by the User-Password attribute, ignoring the 128-octet stated limit. Since the encryption algorithm always produces encrypted text that is a multiple of 16-bytes in length and the longest encrypted text that can be stored in the User-Password attribute is 255 - 2 = 253 octets, the question is what is the largest multiple of 16 less than or equal to 253?

240. 240 octets.

Any plaintext 241 octets long or longer will encrypt to 256 bytes of encrypted data, which is not representable by a RADIUS User-Password attribute. So, a more reasonable specification would require implementers to support "passwords up to 240 characters which contain a mixture of alphabetic, numeric, and special characters."

Maybe in revision 1.04.

The Value of a Comfortable Office

2010-01-21T22:52:00.006-08:00

As I sat at my desk sweating in the dead of winter, I got to thinking about how people's comfortable working temperature must be cultural. Offices are hot in Japan. In the summer, their CoolBiz campaign has businesses setting the thermostats to 28 degrees Celsius (82.4 degrees Fahrenheit). I just checked, it is winter and the thermostat in my office is reading 30 degrees (86 degrees Fahrenheit). In contrast, offices in the U.S. have traditionally been regulated to 72 degrees Fahrenheit.

I know my productivity suffers when I'm uncomfortable. It suffers doubly when I can only type with one hand because I'm fanning myself with the other. But is this just because of differences in cultural sensitivity to heat?

Out of curiosity, I did a quick search and found this interesting opinion piece written by Professor Shin-ichi Tanabe of Waseda University. Some of the interesting points in his opinion piece are:

A guidebook recently published by the Federation of European Heating, Ventilating and Air Conditioning Associations (REHVA) reports that 21.8°C is the optimal room temperature to foster intellectual productivity.

21.8°C is a little over 71 degrees Fahrenheit...almost exactly what offices in the U.S. set their temperatures too. Perhaps we have a hint why U.S. and European workers are the most productive in the world?

...28°C seems a little too high for a room temperature setting in summer. The most comfortable temperature when sleeping naked is 29°C. People burn more calories in the workplace than at home where they are more relaxed, and however casually they may dress, they are still not naked in the workplace.

While there are studies showing some variance in comfortable working temperatures depending on culture and gender, it would seem that 28°C can't be comfortable for anyone.

Raising the cooling temperature of a standard building in Tokyo from 25°C to 28°C could increase energy efficiency by 15%, which is equivalent to saving ¥72 per square meter of office space during the COOLBIZ campaign. On the other hand, the resulting decrease in working efficiency could cause a loss of 13,000 yen per square meter of office space.

What kind of company loses 13,000 yen to save 72 yen? A Japanese company, apparently.

1.1.1.1

2010-01-21T20:12:00.003-08:00

A number of captive portal implementations, including products from Cisco and Nomadix, use 1.1.1.1 as a virtual IP address, HTTP requests to which are redirected to the access control server's logout page. A quick google search turns up numerous network service providers, mostly wireless ISPs, that use 1.1.1.1 to access their logout pages.

This trick has worked because the 1.1.1.1 IP address resided in an IP block that was reserved by the IANA, so there could be no server that actually used that IP address.

However, this month the IANA assigned the 1.0.0.0/8 IP block to the Asia-Pacific NIC. As its name implies, APNIC is responsible for the allocation of IP addresses in Asia and the Pacific, meaning that there may come a day when a company in China, Australia, or elsewhere is allocated a subnet containing the 1.1.1.1 IP address.

In short, the 1.1.1.1 IP address no longer resides in reserved IP space. Network access servers should stop using it.

The Public Option

2010-01-07T21:47:00.011-08:00

Every time I have to take off my shoes at the airport, I'm grateful that Richard Reid stuck explosives in his shoes rather than his ass. Waiting in line while people take off their loafers and flip-flops is silly, but waiting in line while people get full cavity searches might make me think twice about flying.

Anyway, as everyone has surely heard by now, an idiot on a flight to Detroit lit his nuts on fire for Christmas. So now the TSA will be installing new full-body scanners to see if we have anything stuffed in our trousers. I haven't heard the details, but I suppose you are privy to a more intimate inspection if something in your pants draws attention.

And then today I received an e-mail from my friend Matt alerting me that, sure enough, suicide bombers are now stuffing explosives in their rectums. Unless these new airport scanners can see clear into my bowels and distinguish between a Taco Bell lunch and IED, I wonder how far off we are from getting free colon exams whenever we fly.

Actually, I think I may have stumbled upon health care reform that everyone can get behind. The TSA can be our delivery method for socialized health care. If the TSA is going to be probing every orifice looking for explosives anyway, with a modicum of medical training, they could alert us to any potential medical conditions we might not have been aware of while they are in there. Everybody wins: planes are safer, people are healthier, and we save money on colon exams, prostate checks, and gynecologist visits.

It might cost a little money to implement, but who would object? Only terrorists could be against making our skies safer.

Debugging Python Windows Services

2009-12-14T19:56:00.004-08:00

Mark Hammond has a simple example of writing Windows Services in Python using the win32serviceutil.ServiceFramework class in his book Python Programming on Win32.

The problem is, that if you use a breakpoint in your service code and try to debug it, you'll find that it never stops at your breakpoint. The reason is that the win32serviceutil module will actually run PythonService.exe and have that program run your python script. Since PythonService.exe is its own process that has no knowledge of your IDE's breakpoints, it just runs without breaking.

Here is a simple trick for the small intersection of people who:

Write Windows Services
in Python
using an IDE (I use pyDev)
and need to step through their code to debug.

It turns out that win32serviceutil has a reimplementation of the logic in PythonService.exe that it uses to emulate PythonService.exe's debug mode when you convert your python script you an executable (e.g. via py2exe). The implementation is in win32serviceutil's DebugService() function. The trick to being able to set breakpoints and debug your python service is to convince win32serviceutil into calling DebugService rather than spawning PythonService.exe.

Luckily, this is trivially easy: just add the line
sys.frozen = 'windows_exe' # Fake py2exe so we can debug
before you call
win32serviceutil.HandleCommandLine(...).

Then, you just need to pass the '-debug' command-line argument when you run your service to force it into debugging mode. Your debugger should then control the process so you can debug it.

Government isn't all bad

2009-11-09T01:21:00.003-08:00

The big news today is that the U.S. House of Representatives passed a health care reform bill. I'm not particularly impressed with the details of the bill passed nor am I particularly concerned with them...they will most certainly change when reconciling differences with whatever the Senate passes (assuming they pass something).

But, I'll use this is setup to post a link to an interesting counter to nay-sayers scared of government intervention in the health insurance industry.

Update 2009/12/15:: It seems the blog I linked to is no more so I'll go ahead and post the entirety of the message here...

This morning I was awoken by my alarm clock powered by electricity generated by the public power monopoly regulated by the U.S. Department of Energy.

I then took a shower in the clean water provided by a municipal water utility.

After that, I turned on the TV to one of the FCC-regulated channels to see what the National Weather Service of the National Oceanographic and Atmospheric Administration determined the weather was going to be like, using satellites designed, built, and launched by the National Aeronautics and Space Administration.

I watched this while eating my breakfast of U.S. Department of Agriculture-inspected food and taking the drugs which have been determined as safe by the U.S. Food and Drug Administration.

At the appropriate time, as regulated by the U.S. Congress and kept accurate by the National Institute of Standards and Technology and the U.S. Naval Observatory, I get into my National Highway Traffic Safety Administration-approved automobile and set out to work on the roads build by the local, state, and federal Departments of Transportation, possibly stopping to purchase additional fuel of a quality level determined by the Environmental Protection Agency, using legal tender issued by the Federal Reserve Bank.

On the way out the door I deposit any mail I have to be sent out via the U.S. Postal Service and drop the kids off at the public school.

After spending another day not being maimed or killed at work thanks to the workplace regulations imposed by the Department of Labor and the Occupational Safety and Health administration, enjoying another two meals which again do not kill me because of the USDA, I drive my NHTSA car back home on the DOT roads, to my house which has not burned down in my absence because of the state and local building codes and Fire Marshal's inspection, and which has not been plundered of all its valuables thanks to the local police department.

And then I log on to the internet -- which was developed by the Defense Advanced Research Projects Administration -- and post on Freerepublic.com and Fox News forums about how SOCIALISM in medicine is BAD because the government can't do anything right.

Two Years of Tokyo

2009-10-18T20:23:00.005-07:00

Well, we've now officially been in Tokyo two years now.
The end of this month will represent two years since I started work here. Originally, I was only scheduled to work here for two years with an option to renew for an additional year.

To be honest, these two years have probably been the two most difficult years of my adult life. I'm just glad my wife is here with me; she should probably be sainted for going along with this crazy idea.

Japan is a perfectly fine place to visit. The idiosyncrasies are cute, even fun, when you only have to deal with them for week or two. What visitor to Tokyo can't remember the first time they were packed into a train car? It's pretty memorable and gives you a funny story to tell when you get back home. Now trying doing it everyday for two years -- not so much fun anymore.

When we came, we had intended to travel and see more of the countryside on the weekends. But I've found I just don't have the energy. The daily grind just wears me down. So, with my contract coming to a close, we were looking forward to using the last few months before my wife's contract ended to finally make up for some lost time and see Japan.

Back in July I gave my 3-months notice that I didn't intend to renew my contract come this November. But my employer would have nothing to hear of that. Apparently, they need me to stay on through March. I find it flattering that they value my labor enough to want to keep me on for 4 more months, but it shattered our plans of finally getting out of this soul-sucking city and seeing more of Japan.

Probably the average American would have told them to shove it. We're familiar with a system based on at-will employment and autonomy. It would be unthinkable for an employer to tell you cannot quit. They may ask you to stay, even make offers to entice you to stay. But, at the end of the day, you are free to leave.

However, I am a guest in this country. Like a H1B Visa-holder in the U.S., I would have had to leave the country if I lost my employer sponsorship. Luckily, my wife is still employed, so I could have just changed my status to her dependent and stayed. The problem is that landlords in Japan require a guarantor before they will rent you an apartment -- and my employer has been kindly acting as our guarantor.

Which leads to two complications: a) if I left my job I would lose my guarantor; even if we stayed in the country, we would have to move into the housing provided by my wife's employer and b) I feel an obligation to my employer for having been my guarantor for these 2 (now 2.5) years. Add in the fact that I take pride in my work so, if my employer says they need me to stay for 4 more months, I am inclined to see it through. Result: I'll be working for 4 months longer than originally planned.

So we'll be here a bit longer.

I doubt we'll be seeing too much more of Japan, outside of Tokyo, then. Which is really a shame because, based on our limited experience, life in the country is completely different than life in Tokyo. Even Kyoto felt laid back. But Tokyo, Tokyo will never have a warm place in my heart.

There is an old Soul Coughing song that I never really liked called "The Incumbent". But with a little modification, the refrain has really started to speak to me:

Tokyo, Tokyo, I won't go back
Indelible reminder of a steel I lack
I gave you two years, what did you give me back?
A jaw-grind disposition to a panic attack

Anyway, it has been an experience. I'm glad I had the opportunity to live and work in a foreign country; perhaps aiming for the biggest city in the world was a little too high for me. The pressure of big-city life and the hassles of living in an unfamiliar land have just added up to a less-than-wonderful experience. Work has been fine, but I'm looking forward to getting home.

Just four more months.

Moral Majority

2009-10-16T00:33:00.004-07:00

Here I was wishing that more of my fellow Americans would step up and agree that a reasonable baseline of health care should be guaranteed for all of our fellow citizens.

And then I learned about the case of Jamie Leigh Jones. Go ahead, click the link and read about what happened to this poor lady while she worked in Iraq. I'm not going to repeat it here because it makes me sick.

In response, Senator Al Franken offered an amendment to a defense appropriations bill that would ban federal funds to companies that forbid employees from suing for rape.

Not surprisingly, the amendment was added, but it is a sad day that such an amendment is even necessary.

If that was all there was to the story, it would merely be depressing. No, to make things truly sickening you would need to know that 30 Senators voted against the amendment. 30 Senators voted to prevent gang-rape victims from suing their coworkers.

If you would like to know your Senators' stance on defending rape, you can find a list of the 30 pro-rape Senators along with their contact information here. I suppose it shouldn't be surprising that that they are all men.

So now I feel pretty foolish. Here I was hoping my fellow Americans would be Good Samaritans and agree universal health care is a good thing, but apparently we can't even agree rape is wrong.

Who Would Jesus Insure?

2009-10-13T20:00:00.004-07:00

I'm not a religious man, but I do believe in doing the right thing.
Any way you look at it, I cannot see how anyone could argue that offering health care to every man, woman, and child in the U.S. is morally wrong. So why would someone be opposed to such a proposal?

Two of the more common arguments are:
1) The government would run a public health care option incompetently, squandering money and providing sub-par health care coverage.
2) The government would run a public health care program so well that it would put commercial health care companies out of business.

Proponents suggest a third option:
3) The government program would compliment and co-exist with existing commercial offerings.

I'll contend that argument #1 is obviously false. The "public option" being proposed is intended to provide a baseline level of coverage. In other words, it is intended to cap the lower-bound of coverage options. It does not preclude anyone choosing a private health care option. It does not restrict the maximum level of coverage. As such, it can only raise the level of coverage for those currently most poorly served. One would be a fool to choose the public option if a better private option were available; if no better option is available, the public option cannot be inferior to something that does not exist.

Which only leaves the cost component of argument #1. I would like to see the government run the public option in an efficient way, but if it takes an inefficient organization to provide a baseline level of health care to people who have no better option, than that is simply the cost of doing the right thing. There is no moral high ground in putting money before the health of another human being.

As for argument #2, I cannot see the future so I have no way of guessing whether this is true or false. My gut instinct is that it is false. But, if the government were to be capable of running a program so efficiently that we all received better service at lower cost for all possible treatments, I fail to see the problem. Opposing a public option for fear that it will do a better job that corporate options is analogous to corporate welfare for the less-efficient companies. I do not see the moral high ground is putting the health of companies before the health of citizens.

So our possible outcomes are:
#2 is true - we all get better coverage, albeit from a government-run program.
#3 is true - the public option provides a baseline level of coverage but people/companies can still buy health insurance from private providers for a superior level of coverage.

A number of people seem afraid of a single-payer system, as would result if argument #2 were true. However, I emphasize that the only way for #2 to yield a single-payer system is if that single-payer were superior in the marketplace than all other options. If the government were outlawing the competition, that might be a legitimate concern, but there is currently no proposal to do any such thing. As such, gloomy talk of a single-payer system forcing us to endure inferior service to our current corporate plans is nonsense. The only single-payer system on the table is one so superior (and unlikely to happen) that complaining about it seems neurotic.

In summary, there is small but unlikely possibility that everyone in America gets such superior health care that corporate insurers cannot compete. The expected outcome is that every American is guaranteed a minimum level of health care and those of us fortunate enough to be able to afford better health care can still purchase it just as we do today.

Personally, I do not care what happens to my individual health care package. But I firmly believe that ensuring every man, woman, and child in America has access to at least basic health care is the right thing to do. And I am deeply ashamed that a small but vocal contingent of my fellow Americans so despises their fellow man that they want no part in it.

Sniffle Season

2009-10-12T19:49:00.009-07:00

It's fall and the sound of snot is already filling my workplace.

Living in a city the size of Tokyo, you encounter a lot of people on a daily basis. And frankly, people are pretty dirty. It probably doesn't help that there is no custom to cover one's mouth when they cough or sneeze here. I've seen people pick their nose on the train and then immediately re-grab a pole or strap too. As a result, I make it policy to not touch anything.

I'm not ragging on Tokyo here. Frankly, I was appalled at how dirty San Francisco was when I first moved there. I later came to understand it was an "enlightened dirty", something you have to develop an appreciation for. Tokyo, being a far larger city than San Francisco, is correspondingly more enlightened.

So, with the H1N1 virus looming large in the headlines, my workplace has instituted policies including putting hand sanitizers at all of the entrances to "prevent carrying the virus in". We have also been instructed to stay at home if we are running a fever or otherwise suspect we may have contracted swine flu.

Now there is something that has always puzzled me: if you are feeling sick, why would you go into work to get all of your coworkers sick too? This was a complaint I've been harboring for years. Some fool thinks their work is so important that they come into the office and get 5 other people sick. If those five people have any sense, they'll stay at home to prevent further spreading illness, thereby reducing productivity 500% more than if the first person had just stayed home in the first place! But people don't think that way. They think just about their own deadlines or how to make themselves look better to their boss. It is the tragedy of the commons: in trying to sustain their own productivity, the productivity of the company as a whole -- or even the economy as a whole -- takes a hit. In their effort to wring half of their usual productivity in between bouts of coughing or sneezing, they risk spreading their funk to five, ten, a hundred others.

Please stop.

That said, the life of a Japanese salary man is pretty competitive. It is my experience that paid sick days are unheard of -- if you take a day off work, that comes out of your vacation time. Worse, if you take a day off, you "fall behind" your peers. You lose brownie points with your boss. When it comes times for promotion, it is Brownie Point Redemption Day and no one wants to be denied entrance to the pearly gates of management for another year of salary man purgatory.

Which explains why there are so many sick people on the train during the morning commute and why there are so many sick people in my office.

I had one coworker not come in today because H1N1 has been spreading around his daughter's elementary school and apparently made it back to his house. But in my giant open-plan office of 150+ people, there are at least ten more here and sniffling today. And fall is just starting.

I'm actually pretty concerned about coming down with H1N1 myself before the winter is over. It isn't that I'm so much worried about the illness itself -- like the SARS scare of a few years ago, it sounds like H1N1 is more bark than bite. What I'm concerned about is all of the bureaucracy that I would need to deal with. You see, my company has actually instituted a special policy just for H1N1 allowing us to stay home without consuming vacation days if we get a letter from the doctor proving we have H1N1. However, we can only come back to work once we have another letter from the doctor saying we're cured.

I never liked going to the doctor in the U.S., I really don't like doing it in a foreign country. Twice. For a disease they can't cure, no less.

That said, this seems like a nice safe policy to protect the company from H1N1-infected workers thinking they are so important that they need to come in anyway. They are telling us to stay home; even allowing us a special paid leave to do so.

I don't think it'll work, though, for all the same reasons workers always come in to spread their virus-infested cough spew. They should always stay home when the alternative is to risk getting everyone else sick too. But they need the brownie points. It is much easier to put on a mask and just pretend it is a cold than it is to deal with the doctor visits and company bureaucracy. And if they come in, they get the face time and bonus "perseverance" points that managers remember.

My forecast: a train-full of H1N1-carriers dutifully trudging into work, snorting and sniffling all the way. Shoot, come to think of it, even if they were clinic-bound, they'd be on the train sharing their germs.

Monjayaki

2009-09-10T19:50:00.008-07:00

Last night I went out with some coworkers to have some monjayaki in Tsukijima. The Tsukijima business association apparently decided some time back to draw people to their town by making it the center for monjayaki restaurants in Tokyo. There are currently over 60 monjayaki restaurants within walking distance of the Tsukijima station. The one we went too was called Oshio.

We ended up ordering 5 different dishes:

mentaiko-mochi monja - Fish eggs and chewy rice.

mapo tofu monja - Spicy chinese-style beef and tofu.

curry seafood monja - Indian dry curry with shrimp and shellfish.

super-spicy monja - Not sure, but had a pretty strong kimchi flavor.

"apache" monja - Butter, potato, and a big slab of fatty beef.

The picture at the top is the super-spicy monja.

This was the first time I had ever had monjayaki. The consistency is fairy liquid with chunks of whatever the main ingredients are. It solidifies just enough on the grill that you can scoop up with a little spatula-like metal utensil.

Yeah, it has the look and consistency of barf.

But surprisingly, it tasted pretty good. The super-spicy monja, was hands-down my favorite of the bunch. The mentaiko wasn't really to my liking, but at least it tasted better than it looked.

Politics as Usual

2009-07-28T18:41:00.006-07:00

Everyone has their own perspective on the world and their own circumstances influencing their beliefs. What seems right and just to me can seem horrible and unfair to someone else. The founding fathers of the United States of America recognized this truth; in Federalist #10, James Madison wrote that the various competing regional, religious, and economic interests, factions, and parties would be the guarantor of American freedom and justice.

Everyone in America is entitled to their own opinion, representing their own interests. People can try to drum up support for their own ideas by sharing them with others and these ideas compete for mind share with the ideas of others among the American populace.

This is how politics in America works; it is like capitalism applied to the ephemeral world of ideas. And just like capitalism in the market, it should not be surprising that purveyors of ideas often have their own profit motives.

I suppose that statement is somewhat redundant because I believe, to most, the word "politics" is assumed to be the marketing of ideas with underlying profit motives. In which case, to be American is to be constantly embroiled in politics -- to be constantly inundated with ideas, sometimes of dubious motive.

Which is why it is imperative that we each form our own ideas, reflecting our own self-interests. I would dare say that each and every American has their own circumstances, their own obstacles, their own concerns, as well as their own hopes and dreams. And sometimes people's interests conflict.

So ask yourself: if you could change America in a way that would make your life better, what would you do? If you could improve absolutely anything, what would you do? How would you convince others to help you out? If people don't support your idea, what would you do?

Now ask yourself: what are others doing to try to influence you?

Subtle error in RFC 3665 "SIP Basic Call Flow Examples"

2009-07-27T18:52:00.002-07:00

RFC 3665 is simply a collection of SIP call flow examples for reference to implementers. However, I noticed what appears to be a small error in section 3.2 "Session Establishment Through Two Proxies" with the digest values.

In step F2 "407 Proxy Authorization Required Proxy 1 -> Alice", the proxy responds with a Proxy-Authenticate header containing the nonce value "f84f1cec41e6cbe5aea9c8e88d359". However, in F4 "INVITE Alice -> Proxy 1", where the user agent should send back the same nonce value to the proxy server along with the calculated digest, it instead sends the nonce "wf84f1ceczx41ae6cbe5aea9c8e88d359". Clearly, this is a typographical error as the nonces are identical except for the leading "w".

A cursory scan of the rest of the examples in RFC 3665 did not turn up any more mismatched nonce values. The "wf84f1ceczx41ae6cbe5aea9c8e88d359" value appears in other call sequences involving proxy authentication while the "f84f1cec41e6cbe5aea9c8e88d359" value never appears again, implying the nonce value with the "w" character prefixed should be the "correct" value.

Interestingly, while the other nonce values used in examples appear to be 32-character hexadecimal representations of 128-bit values, the "wf84f1ceczx41ae6cbe5aea9c8e88d359" value does not fit this pattern as it is both 33-characters long and "w" is not a valid hexadecimal digit. Which leads me to believe that the "f84f1cec41e6cbe5aea9c8e88d359" value is actually the correct nonce value in the "407 Proxy Authorization Required Proxy 1 -> Alice" call flow and that the other instances where the "w"-prefixed value appears are just the same typographical error replicated via copy-and-paste.

What I don't know, and would be interested in finding out, is whether the calculated digest result value examples were calculated with the "f84f1cec41e6cbe5aea9c8e88d359" nonce or with the "w"-prefixed version. Unfortunately, the RFC does not tell us what password values they used to calculate the digests, so I cannot answer that question myself.

Calculating MD5 of binaries without debug symbols

2009-07-27T00:24:00.003-07:00

If you compile a binary with gcc with debugging information enabled (-g), the MD5 of the resulting binary will change depending on the name of the directory you compile it in. Which means that if two developers compile the same source code with the same options on the same machine, only they do it in their own home directories, the MD5 of the resulting binaries may differ.

However, as soon as you strip the binaries, their MD5s will be the same.

Which leads me to this little tool I whipped up to compare two binaries without stripping them:


#!/bin/sh
# Display the MD5 of a file, ignoring any debugging symbols in
# binaries.

# The strip(1)/objdump(1) commands for removing debugging
# symbols do not support writing to stdout so we need to
# allocate a temp file to write the stripped binary too.
tempfoo=`basename $0`
TMPFILE=`mktemp -q /tmp/${tempfoo}.XXXXXX`
if [ $? -ne 0 ]; then
    echo "$0: Can't create temp file, exiting..."
    exit 1
fi

while [ "$1" != "" ]; do
    # The following line is a cheezy way to accurately
    # reproduce the same error messages as md5(1) when a
    # specified file is unreadable.
    md5 "$1" > /dev/null
    if [ $? -eq 0 ]; then

        # Try to strip symbols from the file on the
        # assumption it is a binary and, if successful,
        # compute the md5 of the stripped file.  Note that
        # objcopy is the same as the strip(1) command.  If
        # objcopy failed to parse the file (i.e. because it
        # is not in ELF format), simply compute the md5 of
        # the whole file since there are no debugging symbols
        # to strip.
        m=`(objcopy -g "$1" $TMPFILE >/dev/null 2>&1 && \
            md5 -q $TMPFILE;) || md5 -q "$1"`

        # Output the result in a md5(1)-compatible format.
        echo "MD5($1) = $m"
    fi
    shift
done

rm $TMPFILE

su Forces Terminal Echo to On

2009-07-02T23:42:00.004-07:00

I just discovered an interesting quirk of the su command, at least on FreeBSD: it forces terminal echo to on and you cannot turn it back off. Interestingly, sudo does not have the same behavior. Here is a sample terminal session showing the behavior (input that was not echoed is shown in red):


[14:54:35] myhost:~ $ stty
speed 38400 baud;
lflags: echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
[14:54:38] myhost:~ $ stty -echo
[14:54:57] myhost:~ $ stty[enter] speed 38400 baud;
lflags: -echo echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
[14:55:00] myhost:~ $ sudo -s[enter] Password: mypassword[enter]
[14:55:15] myhost:~ # stty[enter] speed 38400 baud;
lflags: -echo echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
[14:55:20] myhost:~ # exit
[14:55:25] myhost:~ $ su[enter] Password: rootpassword[enter]
myhost# stty
speed 38400 baud;
lflags: echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
myhost# stty -echo
myhost# stty
speed 38400 baud;
lflags: echoe echok echoke echoctl pendin
iflags: -ixany -imaxbel
oflags: -oxtabs
cflags: cs8 -parenb
myhost#

As you can see, the -echo flag was preserved by the sudo session, but not be the su session. Furthermore, it was not possible to re-disable terminal echo within the su session -- it simply ignored the directive.

I suspect the cause is somewhere in the PAM libraries that su uses to authenticate the user. However, I would have expected sudo to use the same PAM libraries, so I cannot explain the difference in behavior.

Not-so-Cool Biz

2009-06-28T23:58:00.006-07:00

It is still the rainy season here in Japan but the temperature is slowly creeping up as we head towards summer. Today's high is forecast to be 30 degrees (86 degrees Fahrenheit).

Which wouldn't be too much different than the weather this time of year in my home town in Virginia except for two things:

In Virginia we commute by car and cars have air-conditioning; in Japan most people commute by walking or riding a bike to the train station and taking a crowded train.
Cool Biz

Today I'm going to focus on #2. If you aren't familiar with "Cool Biz", there is a nice article over on Slate about this Japanese campaign to reign in energy usage by making everyone uncomfortable.

The gist of Cool Biz is to crank up the temperature in offices to a balmy 28 degrees Celcius (82.4 degrees Fahrenheit) and allow people to "dress down" to compensate. By "dress down", we're talking not having to wear a tie.

I've heard the figure 4 degrees bantered around as the difference not wearing a tie can make. But when I tried researching where that number came from, I cannot find any research measuring the relative apparent temperature with and without a tie. The number (which I presume is in degrees Celsius) seems to appear only in articles about Cool Biz and only as a number the Japanese government cites.

Not surprisingly, the large company I'm working at here in Tokyo is following the government's recommendation and has the thermostats set for 28 degrees. And I can tell you first-hand that the heat is incapacitating.

I was doing some research on how to deal with the heat and ran across some tips for living without air conditioning. For a wussy American such as myself who has been spoiled by comfortable working conditions for my whole life, 82.4 degrees might as well be no air conditioning. The tips:

Turn on fans to keep the air circulating.

We have 2 fans in a room with over 100 people in it.

Set ceiling fans to turn clockwise (counter-clockwise looking toward it).

Not really applicable in an office environment.

Keep curtains or blinds closed during the day.

Blinds are kept open all day.

Use light colored curtains to deflect heat.

The blinds are white. So if we were allowed to close them, it might help.

Keep windows and doors closed tightly to keep the house cool.

Check.

If there is a breeze or if outside air feels cooler than inside, open the window bringing in the cooler.
Keep sunny windows closed if there is no breeze.
In late afternoon, as soon as outside temperatures feel lower than inside temperatures, open all windows and doors.
In the evening, when it’s cool outdoors but still hot indoors, place fans in front of open windows to draw in the cool air.

I work in a fairly old office building that still has windows you can actually open. However, we are forbidden from opening them. Rumor has it that a couple years back someone snapped and try to jump out a window. Since then, we've been strictly forbidden from opening the windows. I suspect it was the heat that drove him to it.

Keep as many windows as possible open during night to take advantage of the cool night air to lower inside temperatures.
In the morning, close windows as soon as the outside air begins feeling warmer than inside air.

Besides not being feasible in an office environment, see the notes regarding the previous tips.

Limit strenuous physical activity until evening.

Check.

Drink lots of cold water.

And check.

When hot, use a water spritzer to spray yourself. The evaporating water will make you feel cool. (For fun, you can also spritz the water upwards and feel it fall down on you. The coldness can be quite shocking.)

I work in an open office floor plan with people sitting 2 feet from me in each direction and no walls. This might work if they are amicable to the idea too.

Tie a wet bandanna around your neck (this is something I do for my dog to prevent overheating and heatstroke during the summer).

I think we have a winner. I think I may have to try this one when the summer heat really sets in. Maybe I can start a fashion trend of white buttoned-up dress shirt, dress slacks, and a wet bandana. If I combining it with spritzing water, I can call it "wet biz".

Whats and Whys

2009-06-08T00:13:00.004-07:00

I just realized something about myself today: when someone is talking to me, I'm not only listening to what they are saying, but also trying to deduce why they are saying it. Not consciously, of course, or else I would have realized that I'm doing it long before now. :)

In hindsight, I think this has been mostly a good thing. As a software engineer, when a request comes in for feature "X", rather than just understanding that the customer wants feature "X" and immediately starting to work on solving that problem, somewhere in the back of my brain I'm thinking about why the user asked or "X". What are they probably trying to accomplish? Is there perhaps a simpler way to achieve the same result that we can present as a counter-proposal? Is this something other customers might need too? If they are asking for feature "X" now, then they'll probably need feature "Y" at some point in the future too.

Which is to say that I think this has made me a better software engineer. However, sometimes it takes me a night or two of "sleeping on it" before I start being aware of the implications of the request. I've gotten into the habit of splitting my project to-do list into "things needed to do what the customer specifically asked for" and "things needed to do what the customer probably wants" categories. Obviously, the items in the list needed to satisfy the customer's explicit requirements always take priority. But I'll freely add items to the "customer will probably want" list as they occur to me, to be implemented in down time between projects or assigned as introductory tasks to junior engineers to get them familiar with the code base. From experience, I've found that this process results in more generic solutions to problems that make the product more resilient to changing requirements and adaptable to new customers.

On the flip side, I've gotten myself in trouble a few times by reading to much into my wife's words. I don't think I'm unique in that regards either. :)

Today I realized that I am sometimes perplexed by requests/statements from my Japanese co-workers. Of course there is the language barrier which means that it takes a lot more effort on my part to understand what is being said. But sometimes I know I comprehend what they are saying, but I still have an uneasy, discontent feeling. That is when I realized that my brain is subconsciously trying discern the "why" behind the statement and failing, causing the uncomfortable feeling.

As for why it is failing, I'm still unsure. But now that I think I know what is going on, I can work to address it. I suspect it may have nothing to do with the fact that the statements are in Japanese, but rather be a symptom of information asymmetry inherent in working in a large company. In which case, it would explain why I always seem to flourish in small company environments where information is more evenly distributed, hence making it easier to deduce the "why" behind new tasks.

A Score I'm Not Proud Of

2009-05-29T00:09:00.004-07:00

I was doing a little research to write a post comparing incentives in the Japanese and U.S. workplace. In the process, I ran across the Zung Self-Rating Depression Scale. I know I'm not happy here in Japan, but I thought I would take the online test just to see how bad it was.

I got a 75 out of 100. That would be "severely depressed"...which is quite an eye opener for me.

The funny thing is that, compared to the people I see on my commute, I think I seem pretty well-adjusted. I mean, Japan has some of the highest incidence rates of suicide and alcoholism amongst industrial countries. I guess I can see why. At least I have something to look forward to: I can go home and get back to a normal life. But I feel for the poor Japanese salarymen I see slumped over on the train platform nursing a beer and staring blankly off into the distance...this is as good as it gets; there is nowhere else to go.

Anyway, I'll try to get back to writing my observations on incentive systems for a future post. For now, I'll leave with the same comment I give everyone who asks about Japan: it is a nice place to visit, but I wouldn't want to live here.

See Me at Interop Tokyo 2009

2009-05-10T22:55:00.005-07:00

In an effort to put a face behind its technology, NTT Communications will be showcasing portraits of some of their engineers at their booth at Interop Tokyo this year. And unless I had spinach in my teeth, there is a good chance my portrait will be there this year....even if I'm not. Six members of the team that runs the servers behind the HOTSPOT service, myself included, were gathered after lunch today to have our pictures taken. It wasn't anything fancy: first, they each of us pose for individual pictures, then did a few pictures of us in groups of three, and finally one big group photo of us at a conference table.

So if you find yourself at Interop Tokyo 2009 this upcoming June, swing by the NTT Communications booth. Above the booth you should see a number of portaits of engineers. I'll be the stupid-looking one.

Racism in Japan

2009-03-11T23:25:00.005-07:00

I generally try to keep myself upbeat and try to maintain a positive outlook, but the racism in Japan really wears me down. The "帰れ(go home)", "死ね(die)", and host of other comments muttered in passing or said behind my back have really dampened my excitement about living in Japan. While less blatant, the rampant tendency for strangers to clear their throats or spit when they become aware of my presence suggests the racism runs pretty deep.

In hindsight, learning Japanese may have been a mistake; if you can't understand the language, you don't know when people are making rude comments.

Even as a white male, I do my best to not draw attention to myself. I've seen the looks of distrust and fear my fellow commuters give to foreigners of middle-eastern or African decent on the train. I can only imagine the death by the thousand cuts that everyday life in Japan must be for them.

My poor 5'2" wife isn't immune either. She's fed up with the way little old ladies stare at her and awkwardly avoid her like she was some sort of circus freak.

My coworkers often ask me why my wife and I don't go out and see more of Japan on the weekend. Frankly, at the end of a grueling work week, I don't have any urge to stand on the train some more. But more important, the last thing either of us wants is our weekend spoiled by bigots. Just going to the store can be emotionally draining. Trying to rent a room in the country can be outright impossible (70% x 37.8% = ~26.5% of hotels surveyed illegally refuse to serve foreigners; they cite "language barriers" -- apparently confusing nationality with language ability).

The almost-daily little hints, most of which seems trivially minor in isolation, add up to a roar of "you're not welcome here".

Before we came to Japan, we were familiar with the complaints many foreigners living in Japan have. The comment sections of many popular news sites aimed for expatiates living in Japan are filled with disillusionment. I always figured it was because the people hadn't taken the effort to learn the language, or had unreasonable expectations, or were being obnoxious in public...it had to be something they were doing wrong. Maybe it was; maybe I've made the same mistakes.

Since coming to Japan, I have come to appreciate Dave Aldwinckle's complaints and the hard work he has been doing to try and bring the injustices in Japan into the forefront. Whenever I get worked up enough about something that I want to bitch about it on my own blog, I just need to go hit his debito.org to commiserate.

When I was sitting in the comfort of the U.S.A., Debito's stories seem farfetched and, frankly, unbelievable. More than once I thought he was making a mountain out of a molehill. However, I now realize that he doesn't have to go digging to find examples of racism, discrimination, injustice, and hypocracy...it turns out there is just a lot of material to pull from here in Japan.

Unfortunately, while brave individuals like Debito are trying to recitify the situation, apologists still abound. Guides for foreigners coming to Japan are filled with such insightful advice as:

NOTE: At certain points through this and other articles we note that in some cases foreigners may be refused entry to particular hotels or rental of accommodation. This is not intended to imply any form of prejudice; it is merely a statement of the facts. Almost exclusively this is due to the very low number of foreigners in Japan, and general ignorance among the Japanese regarding foreigners. Smile, persevere, and try to be a good ambassador not just for your own country but for all non-Japanese in general.

(taken from http://educationjapan.org/jguide/accommodation.html)

That's right, denying you access to shops just because you are a foreigner isn't any form for prejudice. Nope. People may be ignorant and racist, but if there is one they are not: they are not prejudiced. So just suck it up and deal.

The implicit message: "no one wants you here and if you don't like it, well go home."

Anyway, I don't want to give the wrong impression: I've met a lot of nice people in Japan (and Japanese living overseas). If it weren't for the great folks I have (and have had) the pleasure to work with, I probably wouldn't still be here. That said, I am looking forward to going home. Honestly, many people I have met seem to have no comprehension of how intolerant their countrymen are or, as the quote above demonstrates, don't even recognize actions as being discriminatory in the first place.

So until I can go home, I guess I'll have to continue to find solice on Debito's blog.

Happy Nerdigras

2009-03-05T20:04:00.007-08:00

I won't go into the origins since I didn't come up with them, but March 5th has been identified as the Square Root of Christmas and March 14th is Pi day. The discoverer of the Square Root of Christmas has expanded on his work to further dub the week between the two geeky dates as Nerdigras.

I guess this may become only an American holiday, as our nerd brethren in the rest of the world insist on silly dd/mm/yyyy or yyyy/mm/dd orderings for representing their dates.

In any event, in celebration, the commenters over on Ned Batchelder's blog are composing an official Nerdigras carol. Here, I reproduce in its entirety, the 012-days of Nerdigras (sung to the tune of the 12-days of Christmas):

On the oh-first day of Nerdigras, my imaginary girlfriend gave to me,
A hash map in a B-Tree.

On the oh-second day of Nerdigras, my imaginary girlfriend gave to me,
Two Turtle functions,
And a hash map in a B-Tree.

On the oh-third day of Nerdigras, my imaginary girlfriend gave to me,
Three unmatched parens,
Two Turtle Functions,
And a hash map in a B-Tree.

On the oh-fourth day of Nerdigras, my imaginary girlfriend gave to me,
Four calling stacks,
Three unmatched parens,
Two Turtle Functions,
And a hash map in a B-Tree.

On the oh-fifth day of Nerdigras, my imaginary girlfriend gave to me,
Five golden ratios!
Four calling stacks,
Three unmatched parens,
Two turtle functions,
And a hash map in a B-tree.

On the oh-sixth day of Nerdigras, my imaginary girlfriend gave to me,
Six trees sp-laying,
Five golden ratios!
Four calling stacks,
Three unmatched parens,
Two turtle functions,
And a hash map in a B-tree.

On the oh-seventh day of Nerdigras, my girlfriend gave to me,
Seven pointers dangling,
Six trees sp-laying,
Five golden ratios!
Four calling stacks,
Three unmatched parens,
Two turtle functions,
And a hash map in a B-tree.

On the oh-tenth day of Nerdigras, my imaginary girlfriend gave to me,
Eight bits a-zeroed,
Seven pointers dangling,
Six trees a-splaying,
Five golden ratios,
Four calling stacks,
Three unmatched parens,
Two turtle functions,
And a hash map in a B-tree.

On the oh-eleventh day of Nerdigras, my imaginary girlfriend gave to me,
A printout of Graham's Number
One corrupted byte
Seven pointers dangling
Six trees sp-laying
Five golden ratios!
Four calling stacks
Three unmatched parens
Two turtle functions
And a hash map in a B-tree.

On the oh-twelfth day of Nerdigras, my imaginary girlfriend gave to me,
[insert lyric here]
A printout of Graham's Number
One cornipted byke
Seven pointers dangling
Six trees sp-laying
Five golden ratios!
Four calling stacks
Three unmatched parens
Two turtle functions
And a hash map in a B-tree.

(Song by Jonathon Duerig, Robert K., et al)

Procreation Propaganda

2009-02-15T17:28:00.003-08:00

The current fashion in Japan is babies.

For the past 2 or 3 years, there has been a concerted effort on the part of the Japanese government to fight Japan's shrinking population by encouraging more women to have kids. At first, they tried the obligation and duty angle: news was that the Japanese people needed to produce more children to maintain Japan's status as the second wealthiest country in the world. Then, the government started providing more services to ease child rearing; some of these were long overdue, such as maternity care programs and day-care centers for working parents.

Let be reiterate that because the concept may be unbelievable to readers from modern nations: until just recently, day-care centers were few and far between in Japan; what few there are were very expensive and, until deregulation in 2000, run by the government. The concept of a babysitter is still foreign here. Of course, things are changing, but Japan has gotten an awfully late start.

I wish I had taken a picture of it, but I recall a poster on the train a few months back that sent the clear, if not implicit, message that if you have kids then people will be nice to you. In a country in which people rarely acknowledge your existence, that's a pretty bold statement. We're going to skip the courteous stage and jump right from the treating-you-as-an-obstacle stage to the being-kind-to-you stage...all you need is a kid.

There has been much internal discourse in Japan about the ramifications of a shrinking population. By and the large, the message is "have more children" because "shrinking is bad". I have not read the multitude of books on the subject, but the message sent by the in-train posters, celebrity magazines, and "talent"-riddled Japanese television shows is "kids are cool".

Don't get me wrong: kids are cool. What is uncomfortable to me is the idea of the government intervening in my personal life to support an official policy to counter the shrinking population.*

The pretext to all this commotion is that a shrinking population is bad. However, Professor Akiko Matsutani disagrees and in 2007 wrote a book titled "Shrinking Population Economics" in which he claims the shrinking isn't bad, but rather provides an opportunity for Japan to improve the quality of life for its smaller population. In his book, he argues that Japan should not focus on producing a new army of alcohol-numbed workaholics, but rather let workers enjoy more time with their family, take up hobbies, and unwind on holidays.

Which actually brings the argument around full circle. Perhaps the 12-hour+ work days, 3+ hour commutes, semi-regimented drinking parties, etc. with little or no time left to spend at home are as much to blame for the current "baby crisis" as anything. You can't be a family if you are never home. If the only time you see your spouse is for a few minutes as your drag your groggy butt back home after a long day at the office, is it really any surprise you don't have more kids.

The government solution: use propaganda to encourage people produce another generation of workaholics with no time or energy to have children themselves. Professor Matsutani's solution: make life enjoyable, kids or not. Life is to be enjoyed, not frittered away in some Orwellian dystopia. And the irony is, happier couples are more likely to have children.

* I use the phrase "my personal life" empathetically with the Japanese people. As a foreigner in Japan, the government couldn't care less whether I have kids or not.

~~~

As a timely addendum to my ramblings:
Canon has taken the initiative to send its workers home at 5:30pm (only twice a week, mind you). But it sends them home with homework: to make babies. I wonder if they get performance reviews.

In Love with iTunes Movie Rentals

2009-02-11T17:55:00.003-08:00

I've been meaning to try it for a while, but for the first time, on Tuesday night, we rented a movie from iTunes. Yet again, it was another impressively easy-to-use product from Apple.

Living in Japan, we often don't get movies for months after they come out in the states. And when they do finally come out here, they usually either dubbed into Japanese or have subtitles added. Frankly, dubbing sucks in more ways than I care to list. The subtitles aren't so bad, but they do detract from the movie a little. The most important selling point to us, though, is that we don't have to get dressed and walk 30+ minutes to the video rental store (and another 30+ minutes to return it when we're done).

So Tuesday night we basked in the warm glow of my wife's 24-inch iMac and watched WALL-E. Yes, not a new release, but I had only seen it on the airplane and my wife hadn't seen it all yet. It took all of 30 seconds to rent through the iTunes Store and the download began immediately.

On the downside, it took 2 hours to download the movie. While we could have started watching it immediately (while it continued to download in the background), since the movie was only about an hour and 20 minutes long, we didn't want to risk playback catching up to the download and ruining our movie-watching experience with "buffering" breaks. So, we watched a couple of episodes of The Daily Show online to let the download get a little over an hour head start.

With an hour left to finish downloading the movie, we started watching WALL-E full screen on the computer. The image quality was fabulous; there were no visible encoding artifacts at all. The sound was similarly superb. Even with the download continuing in the background, the movie played smooth with no hiccups. It was exactly what I would have expected from renting a title on physical HD-DVD or Blu-ray media.

Overall, the 1 hour lead time before we could start watching the movie was a little disappointing. But being able to sit comfortably at home and not hoof it down to the video store more than made up for that little setback. Next time, we'll probably either download the movie a day or two ahead of time or my wife will start the download when I leave the office so it is ready to play when I get home. Other than that, we were extremely pleased with the convenience, the quality, and the ease-of-purchase.

Apple nailed the user experience aspect again; there wasn't a single moment where I felt like we had to sacrifice something in order to gain convenience by purchasing online. The rental fee was commensurate: $4 to rent a recent release. I seem to recall Blockbuster charging $4-$5 to rent recent releases. So for roughly the same price as an old-school movie rental, I gained convenience but lost nothing. Even without our unusual circumstances, that looks like a decent deal to me.

So Apple iTunes' movie rental feature gets 4 thumbs up. As expatriates, we'll throw in 4 big-toes too, for a total of 8 digits up.

In case you are curious, there are services vaguely similar to NetFlix here, but they all require a) a Japanese credit card and b) postal delivery. I'll save my rant regarding trying to get a credit card in Japan for another day, but suffice to say we don't have one and aren't likely to get one during the remainder of our stay here.

Damn spammers

2009-02-02T21:21:00.001-08:00

As much as I don't like it, spammers have forced me to disallow comments from anonymous posters.

It's the End of the World as We Know It

2009-01-22T16:53:00.002-08:00

And I Feel Fine.

Living in Japan for the past year, I largely missed out on the 24-hour coverage of the presidential election. In fact, even though I registered to vote absentee, I never did get my ballot. In any event, I never got that caught up in the presidential election. To be honest, though, while I was impressed with Obama's message of optimism after 8 years of fear, I didn't have a whole lot of faith he would follow through with his promise of change.

So while I was happy the Obama won the election, more for what it said about my fellow Americans than what it said about Obama, I was hesitant to believe in "change" until I saw it. In particular, I wanted to see Obama make his first act as president be to close the Guantanamo Bay prison. That, more than anything to me, would signal the Obama is serious about restoring American ideals.

Well, it wasn't his first act, but my eyes teared up this morning when I read the news: Obama orders Gitmo to close. 3 days into his presidency and he's committed to shutting down that black-eye of injustice and putting America back on the high road of morality.

In the words of the 2nd century philosopher Hierocles,

We ought always to deal justly, not only with those who are just to us, but likewise to those who endeavor to injure us; and this, for fear lest by rendering them evil for evil, we should fall into the same vice.

As an American, today is truly a proud day.

Open Plan Offices

2009-01-19T01:00:00.003-08:00

The office that I work at, like many other offices in Japan, is what they call an "open plan office". If you search google images for "open plan office", you'll see page after page of images of hip-looking, airy spaces with lots of natural lighting and no walls. Japanese open plan offices are similar, but without the hipness, airy spaces, or natural lighting. Just long tables full of people sitting at computers under row-after-row of fluorescent lighting. My office consists of over 100 people sitting in 1 giant room. Cozy might be a good word to describe it.

And let me tell you, when the air conditioners shut down at 6pm, it gets mighty toasty -- even in the dead of winter.

I searched google images for some representative samples of Japanese open plan offices. You don't have to look hard, a simple search for "オフィス" ("office" in Japanese) yields almost nothing but open plan office pictures. Oddly, the pictures have tiny 1-foot tall dividers separating facing people for some semblance of personal space; I don't have that luxury at my office.

Anyway, having worked in this environment for over a year now, I took special interest in a recent study from Australia's Queensland University of Technology. In that study, they found that:

In 90 per cent of the research, the outcome of working in an open-plan office was seen as negative, with open-plan offices causing high levels of stress, conflict, high blood pressure, and a high staff turnover.

...

The high level of noise causes employees to lose concentration, leading to low productivity, there are privacy issues because everyone can see what you are doing on the computer or hear what you are saying on the phone, and there is a feeling of insecurity.

Now that is something I can relate to. I have found it impossible to concentrate on anything in this environment and my productivity has plummeted compared to the comfortable offices (or even cubes) I was used to in the U.S.

I can only hope that the real research and development institutions in Japan have the good sense to give their staff decent working conditions. Otherwise, I cannot fathom how Japan can hope to compete in knowledge industries.

Merry Christmas

2008-12-25T17:55:00.003-08:00

While there are tons of decorations in Tokyo, Christmas isn't actually a holiday in Japan. As such, I had to work on Christmas. While I wasn't able to spend Christmas with my real family, I did spend some quality time with the people who, over the course of the past year, I've come to be pretty close to.

Yep, I'm hijacking a light-hearted Christmas post to talk about trains in Tokyo.

I didn't actually take this picture, but it is pretty representative of how I spend 2 and half hours of my day, five days a week (Christmas included). Occasionally, especially in the evenings, it is so crowded I can't move or, worse, can't even stand over my own center-of-balance (meaning I'm just propped up by the surrounding crowd).

While I'm getting used to it, this was quite the contrast to the comfort of CalTrain, where I could almost always get a seat even when it was crowded. I've always been a big proponent of mass transit, but I now have I new appreciation for the comfort of one's own car. Even if you are stuck in traffic, at least you still have some personal space to call your own.

The Gay Chikan

2008-12-01T21:04:00.004-08:00

Japan has a particular problem with gropers on trains. So bad, in fact, that 2/3 of all women in their 20s and 30s report having been molested or groped on the train. Considering that most women pass through their 20s and 30s at some point in their life, I imagine it is safe to assume that almost every women has been groped at one point or another.

To alleviate the problem of gropers, or chikan as they are known in Japanese, many train operators in Japan have implemented women-only cars to segregate the perpetrators from their potential victims. The chikan are still there, but now they have no choice but to keep their hands to themselves. In theory. In practice, the women-only cars are overcrowded (perhaps a further testament to how pervasive the groping problem is) and the pervert-to-woman ratio in the normal cars is inflated.

I don't know what the solution is, but I can sympathize with the plight of Japanese women.

'Cause I'll be damned if I haven't been felt-up by the same (male) chikan three times now. This morning was the third incident and he was about to receive the brunt of a year of pent-up stress in the form of all the beating this scraggly nerd could muster. But reason got the better of me and a gruff throat-clear and repositioning of my bag was enough to get him to back off. You see, the reality of the situation is:

I've never fought anyone in my life. I'm a wuss. At 5'10 and 140lbs I've probably got 6 inches of height and 30lbs in my favor, but still, I've never kicked anyone's ass in my life. Frankly, I don't really know where to start. I'd be flailing there in my nerd rage kicking and screaming like an idiot.
He has good plausible deniability. The trains are crowded; he was reading a book with one hand while feeling around for my package with the other. Now I know there was enough room that he didn't need to stand awkwardly close to me and position himself so his free arm was a close as possible, and I know that his free hand was not simply resting by his side but rather working its way my direction. But once I confront him, all that goes out the window. The train moves on and there I am, in my broken Japanese, trying to explain that this sicko was purposefully grabbing my crotch. It is just his word against mine. Gay chikans don't seem to get much coverage here...surely I'll just be seen as a confused foreigner.
Had I beaten the shi*t out of him, I'd be the criminal. Honestly, this is probably true in the U.S. too. After all, feeling a guy up on a train may be a crime, but so is assaulting someone. It doesn't help that the popular image going around Japan the past few years is that most criminals are foreigners. Let's play pretend and say that I somehow managed to inflict severe injury on that pervert. If you are the police officer confronted with a bloodied and beaten Japanese man and an incomprehensible foreigner, who would you handcuff and haul off to jail? Yeah. I don't want to go to prison in Japan.

So all I could do was try to deter the guy. Luckily, he got the message but I don't really know what I can do if the pervert persists.

As a bit of history, I mentioned this was the 3rd time this guy has groped me. The first time, the train was really pretty full and, frankly, I was in denial at the time. It is pretty common for your hands to get pinned in awkward positions on a full train and he was being just subtle enough that it took me a while to figure out that it wasn't accidental. I was pissed, but thoroughly at a loss of what to do about it. That was almost a year ago.

The 2nd time, it was really obvious what he was up to. The train wasn't particularly crowded, but when he got on he positioned himself awkwardly close to me and turned such that his hand was next to my crotch. Within seconds he was feeling around while feigning reading a book. I turned to avoid him but he kept at it until I gave him a stern look and a gruff grunt. At which point he actually walked away and stood somewhere else in the train. That was about 8 months ago.

And then there was today. I had actually forgotten about this creep, but as soon as I saw him positioned awkwardly in front of me I knew what he was up to. I was channelling all of my pent-up frustration and anger to serve him up the best beating I could, but I knew I shouldn't. Luckily, clearing my throat was enough to tell him to go away and, like the previous time, he walked away and rode the rest of the way to Shinjuku at a safe distance from me. But god damn I was pissed. Pissed at this guy for repeatedly targeting me and pissed at the system for making it so hard to stop this jerk once and for all.

And this is exactly what almost every woman in Japan is forced to put up with.

At least a woman can grab his hand and call out "chikan" and have people believe her (at least in theory, this translation of a guide for woman on how to deal with chikans speaks volumes on the Japanese attitude toward the problem). But, seriously, I don't think I'm going to get the same support when I try to drag that gay chikan off the train. And, for some reason, I don't foresee the women-only car concept being extended to solve the gay chikan problem too.

RFC868 UDP Time Protocol Client

2008-11-18T01:10:00.004-08:00

Here is an RFC 868 UDP Time Protocol Client implementation in less than 30 lines of python.
I know...just what you always wanted. It is a long story, but I needed a time protocol client (that would run on Windows) for testing a service we're developing at work. There are tons of implementations of the TCP version of the protocol, but I couldn't find a UDP implementation for Windows to save my life. Python to the rescue.

from socket import *
from struct import unpack
from time import ctime, sleep
from sys import argv

argv = argv[1:]
if len(argv) == 0:
   argv = [ 'time-nw.nist.gov' ]

s = socket(AF_INET, SOCK_DGRAM)
s.settimeout(5.0)

for server in argv:
   print server, ":",
   try:
       s.sendto('', 0, (server, 37))
       t = long(unpack('!L', s.recv(16)[:4])[0])
       # Convert from 1900/01/01 epoch to 1970/01/01 epoch
       t -= 2208988800
       print ctime(t)
   except timeout:
       print "TIMEOUT"
   except:
       print "ERROR"

s.close()
sleep(2)

In case you are curious, the sleep(2) at the end is there so our testers can simply click the icon on their desktop and have time to actually see the results before Windows closes the console window.

By default, it queries the time-nw.nist.gov server, but you can specify any number of servers to query on the command-line.

A Calculus Lesson

2008-11-16T20:03:00.007-08:00

I don't know what triggered it, but this morning I recalled a lesson I learned in high school Calculus class. You see, I was in one of those "advanced" AP Calculus classes, one that was supposed to prepare you for the Calculus AB AP Exam (i.e. the hard one). I don't know if that means anything more these days, but at the time, it was the class all the good students took.

But here is the rub: I was a terrible student. Seriously. This was senior year; while my classmates were keeping tabs on their class ranking to see who was in which position for the top 10, I just kind of floated along and daydreamed about computer programming. I think I had something of a C average in that class; I wasn't even in the top 25% of by graduating class, much less worried about being in the top 10.

Anyway, I never really bought the whole school competition thing. Don't get me wrong: I like learning stuff, but grades were always an afterthought.

The particular incident I remembered this morning, however, was a "challenge" assignment in my Advanced Calculus class. The teacher gave us an assortment of problems that he claimed he also used in his college classes. These problems were supposed to be difficult. We had a week or two to work on them, at the end of which we would turn them in and go over them. I don't recall whether they were to be graded or not, but I latched onto the challenge.

I think there were about 12 or so problems, a few of which weren't anything particularly challenging and everyone got those figured out pretty quick. A few more took a couple of days of trying to arrive at a solution. But the last few I recall being really hard. Each day, before class, my classmates would compare solutions to see if we all arrived at the same answers (we had no answer key).

To tell the truth, I rarely participated in these exchanges. The problem was that some students figured out they could be remoras to the ambitious students' sharks. While a few students compared their solutions and competed to be the first to solve each problem, others would take notes and write the solutions on their own papers.

Now, to me this seemed like cheating at the time. I thought the "smart kids" were getting taken advantage of. I suspect they knew others were copying their answers; maybe they liked the attention, I don't know. Nonetheless, I didn't want to be part of it. I tried solving each problem the best I could. If I was wrong, no big deal...I'd learn the proper solution when the challenge was over. The grade didn't matter, it was the challenge that was exciting. And I didn't want the accomplishment of solving a problem to be sullied by the doubt of having copied it from one of the good students.

Anyway, so things went for most of the assignment. It got down to just one problem that I didn't have a solution for. Again, I don't remember the details, but it had something to do with finding the area of a sphere inscribed in a cone inscribed in a cylinder with the dimensions of the cylinder given in some form. It turns out, that I was not the only one stuck on this problem. No one in my class had solved it.

Well I'll be darned if one morning the solution didn't come to me while I was in the shower. I think we only had a day or two left on the challenge assignment and I was thrilled that I might actually be able to solve all the problems. I think it was in Spanish class that I frantically jotted down my solution and solved to arrive at an answer. I used my intermediate results for the dimensions of the outer cylinder and the inscribed cone to calculate their respective volumes and confirmed that the volume I calculated for the inner sphere was less than the cone it was inscribed in and that the volume of the cone was less than the cylinder. In other words, if nothing else I knew my answer at least made sense.

So I kept my mouth shut in Calculus class that day. If anyone else had managed to solve it, they were apparently doing the same. I started getting excited at the thought that me, the lowest-ranked student in my class, may have actually done something no one there could do.

But the excitement didn't last. You see, the last day before the deadline I blabbed. No one had figured it out, the star students were getting frantic. And someone came over to my desk and asked me if I had figured it out. I hesitated for a second. With only one day left, the odds were I was going to be the only person to solve this particular problem.

And that is when it occurred to me: life isn't a contest. I could keep my solution to myself and be #1 for a day (which is saying something since I graduated 112th out of a class of 420 or so) or I could share the wealth so-to-speak. I didn't show them my answer, though. I spent the 10 minutes or so we had before class to explain how to solve the problem: what things you needed to calculate and how to obtain the parameters needed to calculate them. With that, I knew they could solve it and could still have a clear conscious knowing I didn't just give someone the answer.

Sure enough, by the end of class they had a solution and came back over to compare their answer with mine. Sure enough, they matched. They were happy to have it solved; I felt good having helped someone.

I won't say it wasn't a little disappointing to see them them immediately go back and share the answer with all of the other star students. I suspect almost everyone in the class ended up the answer for that very difficult problem. I'd be surprised if most of them were aware that I had solved it for them, but that was actually kind of fun in and of itself. More importantly, I knew at least one person solved the problem themselves, though, and I knew that they did it based on what I had told them. That felt really good.

Fast-forward 10 years and I found myself working at a Bay Area startup called Backplane with a bright guy named Matt Dillon. Not the actor, the software engineer. I remember a story that the CFO related to me about Matt's previous company, an Internet Service Provider named Best Internet. According to the story, Matt had found Best Internet with a few others and was largely responsible for the technical infrastructure. The servers and whatnot were all hosted in a data center and apparently Matt would, without hesitation, explain the details of how everything worked to any passerby who took an interest. This CFO, who was a childhood friend of Matt's, warned Matt that he shouldn't be giving away company secrets like that.

To which, Matt responded: if they know enough to understand what it is I'm telling them, then they already know enough to build it themselves anyway.

I think that is more-or-less the same thing I realized in my Calculus class so many years before, but he actually put it into words that have stuck with me ever since.

The insight, I think, is not that all information should be free, but rather that there are some truths that there is no benefit in hiding. While some processes, facts, or figures may hold special strategic value, an environment in which all information, no matter what its worth, is horded can be stifling to innovation.

In other words, you can assign a value to all information and a threshold on what you are comfortable sharing. The best engineers I have ever met tend to have pretty low information-sharing thresholds while management tends to err on the high side. The motivations are obvious. Matt, for example, is clearly an engineer while Backplane's CFO was coming from a management perspective.

What I discovered for myself, many moons ago in Calculus class, is that valuing information is a challenging thing. In my case, the information I possessed -- the solution to that problem -- was of questionable value to begin with (I didn't even know whether it was right) and the information would have zero value the following day. Furthermore, my threshold for sharing that information was unjustifiably high -- I didn't stand to gain or lose much if anything by sharing it. To make an unbiased assessment, the only obstacle I had to overcome was my own ego.

And that is the only thing I remember to this day about my high school Calculus class.

We call those meeting rooms

2008-10-23T03:24:00.003-07:00

If you regularly hold meetings in the office "break room", it isn't much of a break room. That is all I have to say.

Firefox advertising

2008-06-17T09:29:00.003-07:00

I couldn't believe my eyes, but this morning I saw this ad for firefox on the train during my commute:

It is being shown on the in-train televisions on the JR Chuo-sen line during rush hour.

Translated, the text is roughly:

If you are using the Internet, use Firefox.
It's very very fast
I'm already hooked
The fox is crazy fast

Oktoberfest

2008-05-28T00:48:00.007-07:00

I don't drink much, but I couldn't help but notice that the 2008 Hibya Oktoberfest is under way across the street from my office.

That's right: they have Oktoberfest in Japan.
In May, no less.

The Perfect Web Interaction Language

2008-05-25T21:48:00.005-07:00

I just noticed that my good friend JJ recently posted a rant regarding the mess that web application development has become. I've ranted about the same thing a couple of times in the past, but JJ's words carry more weight since he is a web developer by profession.

Most of this mess stems from the fact that user interaction via the web is defined by at least 5 different standards: HTTP, HTML, CSS, ~~Java~~ECMAScript, and DOM. HTTP is a protocol; all of the rest are languages, each of which having completely different syntax from all of the rest. On top of that, each standard has multiple different versions. More importantly, each implementation of these standards has varying levels of compliance, effectively creating an ever-changing target to develop for.

Take HTML for example. It is a mess: each web browser has historically rendered the same HTML content slightly differently, supported different tags, and been accommodating of syntax errors to varying degrees. To move the web to a single, consistent, language for describing the content of web pages, the W3C wrote a new specification: XHTML.

Now, good web developers ensure that they always output conforming XHTML, but not everyone does. But then, not every browser accepts conforming XHTML, so there is good reason not to use it (and, as JJ notes in his rant, browser developers have recommended sticking with HTML 4.01 to "ensure the widest browser and search engine support").

This lesson of XHTML is important: it doesn't do any good to make a new standard because you cannot force all browser implementations to support it. In fact, it only makes the problem worse because it simply adds one more possible input a browser may be reasonable expected to accept, hence lengthening the already-too-long list of standards browsers must support (and developers will be expected to know).

CSS, ECMAScript, and DOM are all similar: each browser implements a different set of features that partially overlap each other and partially overlap the standards themselves. An experienced developer is expected to know (and keep up with) what subset of features are safe to use on all targeted browsers and which have equivalent functions and what hacks are necessary to trigger those functions on the implementing browsers. The fact the people even talk about targeting browsers shows just how far we are from having any consistent standard we can depend on to work no matter what browser the document is viewed in.

Now add to this the fact that the HTTP protocol itself, the protocol that is intrinsically linked to HTML, is 176 pages long and (from my experience) poorly understood by most web application developers. Or consider for a moment the fact that you even need to be fluent the mechanics of how the content is delivered to the user (and how input is delivered from the user) in order to write even a simple application. This is Joel's Law of Leaky Abstractions writ large.

Now, let's take a cue from John Lennon and imagine we could start from scratch -- what would the "perfect" environment for web application development look like?

At this point, the problems with the current model are pretty well understood. What if we pretend there is no baggage and we could just start over, developing a perfect language for describing user interaction. Imagine we could develop and deploy a new "browser" that understood this new language and implemented it perfectly to our specifications. Think about it. Seriously.

Not a re-implementation of HTML or XHTML. Not a better JavaScript. Not a better CSS. Those are all thinking too small. How about a single language the describes all aspects of user interaction with a single syntax?

You may say I'm a dreamer, but I'm not the only one.

In fact, as I mentioned above, the W3C has been making motions in this general direction for years. They proposed XHTML and CSS as replacements for the mess that is HTML. But if XHTML and CSS are still not properly implemented by all browsers almost 10 years after their introduction, why should we expect anyone to implement our "ideal" all-encompassing interaction specification language?

My suggestion: we don't.

May I suggest instead, a translator. One that takes a document written in our web interaction language as input and outputs all the hairy HTML, CSS, ECMAScript, etc. that is necessary to get the predominant browsers to behave the way specified by that document. That is, admit defeat and relegate the current web development paradigms to mere implementation details.

In fact, this may not be as crazy as it sounds. There are a number of XHTML templating toolkits for python that can be configured to convert generated XHTML documents into plain old HTML 4.01 documents that all browsers support. Consider extending this concept beyond just output generation, but to the whole of web application interaction.

Think of it as writing a compiler in which the output "machine code" is HTML, CSS, ECMAScript, and the server-side logic.

Is this pie-in-the-sky dreaming? Sure. Is it a hard problem? Sure. Does it risk running afoul of the Law of Leaky Abstractions? You betcha.

But codifying web best practices and cross-browser support behind a consistent high-level language and compiler is priceless. Just imagine the beauty, the joy, that web application development could be. Imagine how much more productive you could be if you weren't coding CSS hacks and having to test on 5 different browsers after each change.

Yeah, it's crazy. But what is the alternative? Convincing every browser vendor in the world to (finally!) implement standards correctly, convincing every person in the world to upgrade to the standards-implementing browsers and ensuring they stay up-to-date as bug-fixes are released for those browsers? Now who is talking crazy?

Turned 0x20

2008-04-11T02:17:00.003-07:00

Wow, I just noticed that it has been months since my last blog update. Turns out that with the long commute, I don't have nearly the amount of free time I used to have. It probably doesn't help that I have to stand/walk for the entire 3-hour round-trip commute; I'm just exhausted at the end of the day.

Anyway, I'm just writing a post to say I turned 0x20 today. At least that is what I told my wife to say when she had her birthday back in February...she's having more trouble with getting old(er) than I am. She didn't like 0x20 because it looked "too young" so we compromised on 26 (duodecimal). She thought that felt more like it.

I'd just assume to tell people I'm 100000, but I don't think too many people would find that funny.

Less code

2007-12-20T20:39:00.000-08:00

I was just reading Steve Yegge's rant against code size and realized that he managed to put into words exactly the feelings that have been drawing me to python in recent years. In particular, I managed to mostly skip the Java step in my journey from Pascal, through assembler, up to C, and then the leap to high-level languages including perl, and more recently python. I don't really know why, but Java never felt "right" -- for anything. To this day, I can't think of too many applications that I would say Java was the best tool for the job. For which, I think Steve hit the nail on the head when he writes:

Java is like a variant of the game of Tetris in which none of the pieces can fill gaps created by the other pieces, so all you can do is pile them up endlessly.

Hallelujah, brother.

Anyway, I strongly agree with Steve's general points about the merits of small code bases, but I won't go so far to say that smaller is necessarily always better. Python hits a sweet spot for me (at least for now) between compactness and comprehensiveness. Certainly a good number of problems could be expressed more succinctly in a functional language such as Erlang or Haskell, but you lose readability. In fact, as elegantly as many problems can be expressed in a functional language, they quickly start to look like line noise when the problems exceed textbook examples.

Programming language preferences aside, what I agree with most from Steve's blog post was not so much that more succinct languages are better, but that less code is better. His post is written so as to suggest that Java itself is a problem -- which may certainly be true -- but he doesn't clarify whether he thinks it is Java the language, or Java the set of libraries.

Python, for example, combines a great set of standard libraries with a language syntax that makes it easy to use those libraries. All the lines of code hidden away in libraries are effectively "free" code. You don't have to manage their complexity. Give me a language that makes leveraging as many libraries as possible painless, then I can glue them together to make great programs with low apparent complexity. In reality, the lines of code might be astronomical, but I don't have to manage the complexity of all of it -- just the part I wrote -- so it doesn't matter.
Python does a great job here, whereas Java (and C++'s STL) largely get it wrong.

In particular, I would argue that, in addition to python's straightforward syntax, the fact that so many of python's libraries are written in C is a large factor in why they are so easy to use. There may be a huge amount of complexity, and a huge number of lines of code, in the C implementation of a library. However, the API boundary between python and C acts a sort of line of demarcation -- no complexity inherent in the implementation of the library can leak out into the python API without the programmer explicitly allowing it. That is, the complexity of libraries written in C and callable from python is necessarily encapsulated.

As a personal anecdote, in one project I work on, we use ctypes to make foreign function calls to a number of Windows APIs. One thing that really bothers me about this technique is that I find myself re-implementing a number of data structures in ctypes that are already defined in C header files. If I make a mistake, then I introduce a bug. Ironically, since I could leverage more existing code, often times there would be fewer lines of code and less complexity had I just used C to call the APIs. Of course, other parts of the program would become hugely unwieldy, but the point of this anecdote is that libraries (more specifically, being able to leverage known-good code) can be much more effective in reducing code than the implementation language.

So long as the implementation language isn't Java. Java just sucks. :)

Lies, Damn Lies, and ... Economics?

2007-12-11T23:42:00.000-08:00

Today I'm going to venture out of any field that I have the slightest expertise in and flounder about in the field of basic macro-economics.

But before I demonstrate my utter lack of knowledge, I'm going to touch on a subject that I at least have some familiarity with -- mathematics. I'm going to share with you a math problem that I cannot solve. It looks something like this:

Given:


     A           A'
    --- = 111 = ----
     B           B'

and


    B = B' * 1.41

What is the ratio between A and A'?
Elementary algebra would seem to imply the answer should be 1.41:


     A     A'
    --- = ----
     B     B'

    A * B' = A' * B

    A * B' = A' * B' * 1.41  ==>  A = A' * 1.41

By itself, I would have expected this problem would be trivial.
However, what perplexes me is that the observed value for the ratio between A and A' is nowhere near 1.41, but rather approximately 1.0.

And here is where I demonstrate my lack of understanding in the field of economics...

The thing that has been puzzling me for years (even before the recent foreign exchange craze) is that the U.S. dollar has experienced fairly consistent inflation for these 13 years while the Japanese Yen has experienced almost none, yet the exchange rate remains the same.

You see, A' / B' is the average value of the Japanese Yen in U.S. dollars for November 2007. And A / B is the average value of the Japanese Yen in U.S. dollars for April 1994. Both ratios just happen to be approximately 111. (Source: Board of Governors of the Federal Reserve System)

The ratio B' / B is the purchasing power of the U.S. dollar in 2007 relative to dollars in 1994. That is, one 1994 dollar (B) is worth 1.41 2007 dollars (B'). (Source: Bureau of Labor Statistics)

The ratio A' / A is the purchasing power of the Japanese Yen in 2007 relative to yen in 1994. It so happens that one Japanese yen buys the same amount in 2007 that it did in 1994. (Source: Japanese Ministry of Internal Affairs and Communications)

How can this be?
How can two currencies can have different inflation rates, but yet maintain the same conversion rate?

I'm no economist, but I cannot help but wonder if the answer lies outside of math and in the realm of human irrationality. Or that the CPI values uses to calculate the purchasing power of a currency are inconsistent and/or flawed. Actually, I know that CPI calculation methods differ amongst countries, but for some reason I expected Japan to have used to the calculation method that U.S. does. I admit I haven't investigated that explanation yet.

Does anyone have a better explanation (preferably one that does not violate basic math principles)? I would love to put this puzzle to rest.

Python: asserting code runs in specific thread

2007-12-03T18:38:00.000-08:00

My buddy JJ's post to the BayPiggies mailing list reminded me of a little snippet I wrote a while back that others might find useful as well. Personally, I avoid threads like the plague, but if you are forced to use them it is generally handy to keep accurate tabs on how you use them. In particular, as JJ suggested in his post, it is a good idea to assert that code is called in the thread context you expect it to be called in. This can go a long way toward avoiding one of many classes of hard-to-find logic bugs multi-threading invites. Anyway, on to the code...

def assertThread(*threads):
    """Decorator that asserts the wrapped function is only
       run in a given thread
    """
    # If no thread list was supplied, assume the wrapped
    # function should be run in the current thread.
    if not threads:
        threads = (threading.currentThread(), )

    def decorator(func):
        def wrapper(*args, **kw):
            currentThread = threading.currentThread()
            name = currentThread.getName()
            assert currentThread in threads or name in threads, \
                   "%s erroniously called in %s thread " \
                   context" % (func.__name__, name)
            return func(*args, **kw)

        if __debug__:
            return wrapper
        else:
            return func
    return decorator

You can restrict execution to one or more threads, each specified by either the thread object or thread name.

Note the trick at the end to make the decorator effectively a no-op in production. Using this decorator around your functions and methods helps you spot logic errors during development without impacting the performance of your production code. Of course, if you are of the school that assertions should never be disabled, feel free to replace the final if __debug__:/else block with an unconditional return of wrapper.

My first OpenSearch plugin

2007-11-22T03:05:00.000-08:00

Well, yesterday I crawed out of my cave and discovered OpenSearch, the XML-based description format for search engines. Both Firefox 2 and IE 7 can import search engine settings from OpenSearch description files and add the search engine to their respective search bars. In fact, there are already description files for just about every search engine you can think of.

There is a great guide to writing your own OpenSearch description and how to link to the description file such that Firefox or IE automatically discover it. The specification is relatively straightforward so, excited, I decided to whip up an OpenSearch plugin for Kelly & Cristin's Wine Reviews.

I only encountered two obstacles:

Getting an icon for my search plugin. I didn't already have a shortcut icon for my site that I could simply re-use, so I had to first make an icon.
I typed the XML document by hand; I left out the final slash in the OpenSearch XML namespace declaration. Firefox will not recognize the file as an OpenSearch document unless the namespace is exactly right. This should have been quick to diagnose except that Firefox gives you no error message to aid in debugging - it simply refuses to load the file.

The text of my OpenSearch description is here. But if you visit the web site using Firefox 2.0 or later, you'll notice that icon next to your search bar gets a blue highlight; if you click on the icon, you'll see an option to add my Wine Review search to your search bar.

Anyway, without the half hour it took to figure out why Firefox wouldn't recognize my OpenSearch plugin, it would have only taken a few minutes to add a search plugin and plugin auto-detection code (one link tag in the HTML head) to my web site. I don't seriously expect anyone to want to be able to search my silly wine review site from their browser search bar, but certainly was neat to be able to add it.

Back on-line

2007-11-20T17:12:00.000-08:00

This is just a quick note to mention that I finally got my Internet access hooked up this past weekend. After picking up a 100V power supply in Akihabara, my little Soekris 4801 is back up and serving the posi.net domain again.

I'm still getting settled into my new routine at work, but I hope to have time to start regularly writing posts again soon...

Unix storage solution

2007-11-07T05:21:00.000-08:00

I've been lax in updating my blog lately, mainly due to lack of Internet access at my apartment since we've moved to Tokyo. It is nice and all that I have a variety of high-speed Internet service providers to choose from, but it would be really great if any of them could actually hook up my service is less than a month. :(

Anyway, until I get my own Internet service, I've been "borrowing" access via an open AP with a signal just strong enough that I can associate if I stand right next to my window. Sitting is no good; I have to have the laptop pretty high in the air to get a signal. As you can imagine, I've been reluctant to get on the Internet any more than I have to.

That said, I couldn't pass up the opportunity to share a surprising Unix storage solution I discovered today. Actually, my wife picked it up unintentionally while shopping at the Don Quixote across the street...

I kid you not, these are cheap little plastic refrigerator storage boxes. I couldn't believe my eyes when she showed them to me. Now if I could only find ones that said "FreeBSD" and "Linux" I'd be set on souvenirs the next time I go back to the U.S.

posi.net going offline

2007-10-15T15:15:00.000-07:00

My personal domain, posi.net, will be going offline for the next few days while I move. This means that any mail sent to the domain will bounce (I'm not setting up a backup MX on purpose, since 99.99% of my mail is spam). It also means the Cristin and Kelly's absolutely fabulous wine reviews, my 地球ラジオ archive, and other miscellaneous amusements will be temporarily offline too.

My last BayPiggies

2007-10-12T09:52:00.000-07:00

Well, last night was my last BayPiggies meeting for a few years. The speakers were Mikeal Rogers & Adam Christian from the Open Source Applications Foundation, presenting their new automated web U.I. testing framework, Windmill.

It was a pretty neat demo; if I actually did any serious web development, I would consider using it (or the preexisting Selenium tool) for testing. Being that it was a python user's group rather than a product show-and-tell, I would have liked more details about how Windmill worked, especially from the python side, but I am assured it is "really really really really cool". ;) I guess I'll have to find time to read the code myself.

Anyway, afterwards I got to hang out with my buddies JJ and Zach for a while and chat over a burger. I'm afraid mine was not particularly stimulating conversation, but I'm really looking forward to finding out what exactly it is JJ has been coding up over at MultiCosmic. He assures me it will go live in a few days, and that I'll have to wait to learn what it is when everyone else does. :) I just know it involves FaceBook.

I've only been going to BayPiggies for the past couple of months, but I'm going to miss the monthly chance to learn about some new tool or technology utilizing python. More importantly, I'm going to miss the opportunity to hang out and hear the state-of-web-development address from JJ (he admitted he loves Rails, by the way :) ). Hopefully, I'll still see Zach from time to time in Tokyo.

Speaking of which, I have yet to find a web page for a Python user's group in Tokyo; pyJug doesn't appear to actually hold casual group meetings just weekend retreats and workshop-style meetings. That sounds like fun, but not quite the casual get-together I've gotten used to with BAFUG and BayPiggies meetings.

Python: A better wx.ListCtrl

2007-10-09T20:53:00.000-07:00

I'm not going to repeat the entire post here, but I would like to direct your attention to my friend and former coworker Zach's recent blog post regarding implementing a better ListCtrl via wxWidgets' virtual ListCtrl.

For those familiar with wxPython, what Zach has done is combine wx.ListCtrl and wx.lib.mixins.listctrl.ColumnSorterMixin into a single easy-to-use class. Except, rather than implement it as an entirely new class, he implements a function that transmutes a generic ListCtrl into his new & improved ListCtrl. That advantage here, as he points out, is that you don't need to modify any XRC files to gain the new functionality.

The post is on NTT MCL's recently-introduced company blog which, unfortunately, doesn't appear to accept comments (and says "Japan Window" for some strange reason). As such, I'll point out that Zach also occasionally posts to his own personal blog, which is also worth checking out.

Sold my car

2007-10-09T20:01:00.001-07:00

We sold my car today. It was a 1996 Ford Escort.

I realize this isn't particularly interesting to anyone else, but since it was the first car I ever bought on my own -- heck, I even moved it with me from Virginia -- it is kind of significant to me.

Anyway, I bought it for $9800 back in 1995 and it is still running well at 154,000 miles. I sold it today for $700 (thanks to craigslist!). That means, excepting gas and maintenance costs, the depreciation was $9100 over 12 years or approximately $760/year. Given that the car got 32 miles to the gallon new and has lately been averaging around 29, it even got good gas mileage (by today's low standards, anyways).

In fact, if I assume an average of 30 MPG over the life of the car, and an average of $2.00/gallon, that comes to about $10,300 in fuel costs. You may think $2.00 a gallon seems low, but a good number of those miles were driven in Virginia, where the price of gas fluctuated between $0.85 and $1.35 before we moved to California. The price of gas has certainly been higher here in California, but I've been riding CalTrain and/or walking to work since long before the recent surge in gas prices, so I think $2.00 a gallon may be pretty close to our average per-gallon price.

Ignoring maintenance costs, which weren't extraordinary, that is $19,500 to own and fuel my Ford Escort for just over 12 years. I'd say it was all around a good purchase. I certainly won't badmouth Ford cars anymore. A hybrid will cost you more than that just to own, not to mention fuel costs.

It certainly didn't turn any heads, but then again, I never once worried about anyone stealing it, dinging it, or hitting it. In fact, I would say that having a dumpy car has its own value -- in peace of mind.

The price of spam

2007-10-02T11:53:00.000-07:00

Like anyone, I get lots of spam. Having kept the same e-mail address since 1995 and having that e-mail address posted all over the web may contribute to absolutely inordinate amount of spam I receive. I get over 1000 pieces of spam mail a day.

To make matters worse, I run my own mail server on a Soekris 4801 at home. I use postfix running on FreeBSD 6.2 with SpamAssassin to identify spam. My spam situation is desperate, so anything that looks remotely like spam gets immediately sent to the bit bucket. Still, over 50 spam mail a day make it to my Inbox.

My Soekris box is my mail server, file server, firewall, and PPPoE tunnel end-point (for my DSL connection). I also run a low-traffic web site off the box. That said, between pppd, the postfix and spamd processes, receiving and processing spam consumes almost 100% of the CPU all day, every day. My load average rarely dips below 1.0 and, at the heaviest times, the inbound mail queue grows to a few hundred messages.

Now, this isn't the fastest machine in the world. But when I started my first ISP back in 1995, we ran dozens of web sites (admittedly, mostly static content) off a single 100Mhz Pentium server with 128MB of RAM. Our entire 27000+ newgroup Usenet feed was hosted on another 100MHz Pentium server, also with 128MB of RAM. But in 2007, I'm here to tell you that it takes a 266Mhz Pentium-class machine (with the same 128MB of RAM) running 24/7 just to deliver mail for two e-mail accounts.

So I can tell you personally that the price of spam in 2007 is roughly one Soekris 4801 plus disk space. It may not seem like much in comparison to today's top-of-the-line computers, but it is enough to make me sick just thinking about.

Moving to Tokyo

2007-09-21T18:46:00.000-07:00

If anyone reads my blog, they would have noticed that I haven't posted any good programming tidbits here in a while. The reason is that I've been busy making arrangements for my move to Japan.

Next month (October 16th to be exact), my wife and I are moving to Tokyo where I will start my new job at NTT Communications' headquarters in Hibiya. My last day here at NTT MCL will be October 5th.

I've worked at NTT MCL for over 5 and half years, making it the longest I have ever worked at a single company. I've really enjoyed working at NTT MCL, so I'm a bit sad to leave. Luckily, my job at NTT Communications will entail further development and maintenance of the servers we developed at NTT MCL for the HOTSPOT wireless network, so it looks like I'll keep in regular contact with my current co-workers. By the way, if you find yourself in Tokyo, or just about any major city in Japan, you should purchase a 1-day pass for HOTSPOT. You get wireless Internet access at a huge number of locations for just 500 yen (about 4 dollars). Having worked on HOTSPOT for over 5 years, I was shocked when I discovered T-Mobile charges $10/day for a similar service in the U.S.

Also, I can't stress enough how nice of a place NTT MCL is to work at. There are lots of positions open, the work environment is relaxed and relatively low-stress, and you'll get to work on a variety of different projects. I know I am going to miss working here, but I simply cannot pass up this opportunity to work abroad for a few years. Hopefully, they'll take me back when I return to the U.S.

Sony getting out of Cell production

2007-09-17T14:00:00.000-07:00

According to Asahi Shinbun, Sony will be selling their semiconductor production facilities, including those for the production of their Cell processor, to Toshiba for approximately 100 billion yen (~870 million dollars). The sale is expected to happen next spring. There is speculation that Sony does not foresee new applications for their Cell processor and is seeking to reclaim a large amount of their investment capital.

Sony's semiconductor subsidiary, Sony Semiconductor, is selling the LSI (Large Scale Integration) facilities in their Nagasaki Technology Center on the island of Kyushu. Besides being the production facilities for the Cell processor, the plant also makes image processing chips for game devices.

Sony will continue to develop the Cell processor, but they are reconsidering producing the chips themselves. Instead, the will be putting their effort into next generation audio/video devices. In particularly, they intend to put emphasis on the production of CMOS sensors like those used to record images in digital cameras.

Keepon

2007-09-03T02:39:00.000-07:00

I saw this last night on a Japanese TV show. Apparently, a U.S. and Japanese researcher have been working on a little robot called "keepon", experimenting with human interaction. Here is a short movie demonstrating some basic emotive interaction:

The American researcher filmed a demo video of the little robot responding to a song by a L.A. band called "Spoon" and uploaded it to YouTube:

Apparently, the video was so popular that the band caught notice and contacted the researchers about using the robot in one of their official music videos. The Japanese researcher appears in the video with the robot (the video was filmed in Tokyo):

Like probably just about everyone else who has seen the video, I checked to see if I could buy one. You can't.

Update 2007/09/04 1:30am (JST):: As with everything else you can think of, Wikipedia already has an entry about Keepon. Sometimes I wonder why I bother writing anything. You could replace my entire blog with nothing but links to Wikipedia articles. :)

Python: Reconstructing datetimes from strings

2007-09-02T18:52:00.000-07:00

Previously, I posted a small snippet for converting the str() representation of a python timedelta object back to its equivalent object. This time I'm going to address doing the same for datetime objects:


def parseDateTime(s):
    """Create datetime object representing date/time
       expressed in a string
    
    Takes a string in the format produced by calling str()
    on a python datetime object and returns a datetime
    instance that would produce that string.

    Acceptable formats are: "YYYY-MM-DD HH:MM:SS.ssssss+HH:MM",
                            "YYYY-MM-DD HH:MM:SS.ssssss",
                            "YYYY-MM-DD HH:MM:SS+HH:MM",
                            "YYYY-MM-DD HH:MM:SS"
    Where ssssss represents fractional seconds.  The timezone
    is optional and may be either positive or negative
    hours/minutes east of UTC.
    """
    if s is None:
        return None
    # Split string in the form 2007-06-18 19:39:25.3300-07:00
    # into its constituent date/time, microseconds, and
    # timezone fields where microseconds and timezone are
    # optional.
    m = re.match(r'(.*?)(?:\.(\d+))?(([-+]\d{1,2}):(\d{2}))?$',
                 str(s))
    datestr, fractional, tzname, tzhour, tzmin = m.groups()

    # Create tzinfo object representing the timezone
    # expressed in the input string.  The names we give
    # for the timezones are lame: they are just the offset
    # from UTC (as it appeared in the input string).  We
    # handle UTC specially since it is a very common case
    # and we know its name.
    if tzname is None:
        tz = None
    else:
        tzhour, tzmin = int(tzhour), int(tzmin)
        if tzhour == tzmin == 0:
            tzname = 'UTC'
        tz = FixedOffset(timedelta(hours=tzhour,
                                   minutes=tzmin), tzname)

    # Convert the date/time field into a python datetime
    # object.
    x = datetime.strptime(datestr, "%Y-%m-%d %H:%M:%S")

    # Convert the fractional second portion into a count
    # of microseconds.
    if fractional is None:
        fractional = '0'
    fracpower = 6 - len(fractional)
    fractional = float(fractional) * (10 ** fracpower)

    # Return updated datetime object with microseconds and
    # timezone information.
    return x.replace(microsecond=int(fractional), tzinfo=tz)

Last time Lawrence Oluyede was kind enough to point out that the dateutil module can likely do this and a lot more. However, I'm trying to stick to modules in the base library. Of course, it wouldn't be a bad thing if dateutil were to make it into the base library....

Speaking of which, the snipped above relies on the FixedOffset tzinfo object described in the documentation for the datetime.tzinfo module. Being that the documentation is part of the standard python distribution, I guess you could call that code part of the base library, even if you can't import it. :|

Update 2007/09/03 1:39pm (JST):: Fix example format examples in doc-string per comment from David Goodger.

Bought a Mac

2007-08-17T18:49:00.000-07:00

My wife and I have been considering buying a new computer for a couple of months now; yesterday we bought a new 24" iMac. The odd thing is, even though my wife was comfortable on a Mac and I was drawn to the BSD underpinnings, we had decided against getting a Mac largely on the grounds of the expense (and the inability to easily uninstall software). Instead, we had settled on a small form-factor Dell. That is, until we visited the Apple Store on University Ave. Sunday afternoon.

We were in the neighborhood to pick up some prints at Wolf Camera and I wanted to stick my head in the Apple Store to check out the new Aluminum iMacs. They are gorgeous. I've never seen a screen quite like it. But the kicker was when my wife asked whether the $1799 price on the card next to the computer was for the computer we were looking at.

It wasn't. We were gawking at the 20" iMac; the placard was for the 24" iMac next to us. Now that was an impressive machine. And what was more amazing was that the $1799 price tag was on-par, if not cheaper, than the equivalent Dell desktop we had been considering! That's right, we could get the gorgeous iMac for the same price as the bland Dell we were planning to buy.

I have read claims in the past the recent-model Macs were cheaper than equivalent PCs, but there it was staring me in the face. Just to prove I'm not making this up, I just re-spec'ed the Dell machine we were going to buy to compare it to the Apple machine we did buy. As of 2007/08/17:

Component	Dell Optiplex 745 Small Form Factor	Apple iMac 24"
CPU	Intel Core 2 Duo Processor E6600 (2.40GHz, 4M, 1066MHz FSB)	Intel Core 2 Duo Processor T7700 (2.40GHz, 4M, 800MHz FSB)
OS	Windows Vista Ultimate, with media, 32 Edition, English	Mac OS X v10.4.10 Tiger
Memory	1.0GB DDR2 Non-ECC SDRAM, 667MHz, (1DIMM)	1.0GB DDR2 Non-ECC PC2-5300 SDRAM, 667MHz, (1SO-DIMM)
Hard Disk	250GB SATA 3.0Gb/s	320GB SATA 7200-rpm
Removable Disc	8X Slimline DVD+/-RW	Slot-loading 8x SuperDrive (DVD±R DL/DVD±RW/CD-RW)
Video Card	256MB ATI Radeon X1300PRO	256MB (GDDR3) ATI Radeon HD 2600 PRO
Speakers	Dell™ A225 Speakers	Built-in stereo speakers
WiFi	Linksys WUSB54GC Wireless-G USB Adapter	AirPort Extreme wireless networking (802.11n)
Price w/o Monitor	$1,198	N/A
Display	UltraSharp 2407WFP-HC 24-inch Widescreen Flat Panel LCD $569	24-inch widescreen TFT active-matrix LCD
Total	$1,767	$1,799

Anyway, we left the Apple Store to head over to Fry's to do some shopping. But while we were at Fry's, my wife looked over at me and "we should get the Mac". She didn't have to say that twice. In my 15 years or so in the computer industry, I have never before seen a machine that I really wanted to buy. So we bought of copy of Parallels Desktop at Fry's and headed back over to the Apple Store.

Unfortunately, we were too late. They close at 6:00pm on Sundays. We tried again Monday night but both the University Ave. and Stanford Shopping Center stores were sold out of the 24" iMacs. We went ahead and ordered on on-line, but we were going to have to wait two weeks for it to even be shipped. So yesterday morning we made it up to the University Ave. store at 10:15am (they open at 10:00am) and bought the last 24" model in that day's shipment. That's right: sold out in 15 minutes. Needless to say, we cancelled our on-line order.

I was up to 4:00am last night setting up Parallels to run Win2k for some of our old software (namely, Microsoft Money and our favorite game: Settlers 3), installing Firefox and Thunderbird, importing our photos into iPhoto, etc. That screen is simply awesome. I don't think I've had this much fun since I was a kid. Certainly not with a computer.

License for code posted on my blog

2007-08-16T14:56:00.000-07:00

It occurred to me that while many people post code on their blogs, they seldom grant anyone else license to use their code. Google encourages their BlogSpot users to mark their posts with the Creative Commons License, but does not enforce users to do so nor do many blogs I've seen follow this recommendation. Now, the poster may know they have no intent to sue anyone over the use of their snippet, but how can the reader be sure?

To address the problem, I'm hereby declaring that all code posted on my blog (kbyanc.blogspot.com) is covered by the license below unless otherwise indicated in the body of the post itself:

Copyright (c) 2007, Kelly Yancey

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

I hope that others will follow suite and clarify the license terms of use for content posted on their blogs too. Use a different license if you like (it is your code after all), but don't be fooled into thinking that simply posting code on your blog makes it open source.

And before someone writes that I should just declare the code to be public domain, I should point out that you can't do that (at least it probably won't become public domain until I'm so long dead that the idea of using the code I posted at the turn of the century seems quaint).

Python: Reconstructing timedeltas from strings

2007-08-15T12:14:00.000-07:00

For some reason, the date/time objects implemented by the datetime module have no methods to construct them from their own string representations (as returned by calling str() or unicode() on the objects). It so happens that reconstructing a timedelta from its string representation can be implemented using a relatively simple regular expression:

import re
from datetime import timedelta

def parseTimeDelta(s):
    """Create timedelta object representing time delta
       expressed in a string
   
    Takes a string in the format produced by calling str() on
    a python timedelta object and returns a timedelta instance
    that would produce that string.
   
    Acceptable formats are: "X days, HH:MM:SS" or "HH:MM:SS".
    """
    if s is None:
        return None
    d = re.match(
            r'((?P<days>\d+) days, )?(?P<hours>\d+):'
            r'(?P<minutes>\d+):(?P<seconds>\d+)',
            str(s)).groupdict(0)
    return timedelta(**dict(( (key, int(value))
                              for key, value in d.items() )))

But the other types are not quite so easy. Next time, I'll post my implementation for reconstructing datetime objects from their string representation.

NTTMCL hiring again

2007-08-07T00:04:00.000-07:00

I guess it comes as no surprise since it seems like everyone is hiring these days, but NTTMCL is hiring again. Of particular interest (to me at least, since it is replacing my position) is the Unix Kernel Developer. I know, you wouldn't know it from my blog posts, but originally I was hired as a FreeBSD kernel developer. Since I am being transferred to Tokyo in a couple of months, and they don't really have the infrastructure in the office over there for kernel development, NTTMCL is hiring a kernel developer to replace me here in the Bay Area.

I don't know who wrote the job description, but there appear to be some typographical errors in it. For example, "enhanced 3 switch" should probably read "enhanced layer 3 switch" and "stabilize the developed wireless network system" should probably be "stabilize the existing wireless network system". Did I mention that NTTMCL is a subsidiary of a Japanese company? We have lots of non-native speakers.

Anyway, besides the kernel developer position, we also have a position open in our security research group, and two positions in our IP technologies group (formerly the IPv6 group) [1, 2]. If you are really into researching and playing with the latest hardware, we have a position in our Business Development group that will get to check out all of the latest gear that NTT Communications might be interested in.

We're a relatively small research and development subsidiary of NTT; currently we have about 35 people, the majority of which are software engineers of some sort. We have offices in both San Mateo, CA and San Jose, CA. The work environment is kind of a mix of big-company and startup: we're small and work with many cutting-edge technologies, but our income is stable being that we are a wholly-owned subsidiary of one of the largest ISPs in the world. There are no stock options, but there are also no 18-hour work days.

Anyway, I really like it here. In my 5 and half years here, I've worked on everything from FreeBSD kernel programming low-level networking, and implementing high-performance Un*x daemons (all in C, of course), to perl and embedded perl interpreters, to web application development (both with and without AJAX), to developing GUI-based applications on Windows using python. If you really like learning and applying new technologies, this is a great place to do so without having to worry about where your next paycheck is coming from.

If that sounds good to you too, send a copy of your resume to jobs@nttmcl.com.

Update 2007/08/16 07:07pm:: I got most of the errors in the job description for the kernel programmer corrected with H.R.; I haven't checked the others, though. Speaking of which, we're up to seven open positions now. Bring some friends! :)

One-liner to crash IE6

2007-08-06T13:44:00.000-07:00

A Japanese fellow going by the name Hamachiya2 has stumbled upon one line of HTML/CSS code that crashes IE6. The magic line is:

<style>*{position:relative}</style><table><input></table>

You can try it yourself at: http://hamachiya.com/junk/ie_crash.html.

Of course, if you are running IE6 or anything that embeds IE6 as a component, you can expect it to crash. All other browsers appear to render the code just fine.

I think I may have just found a new signature. :)

Update 2007/08/07 01:16pm:: Zeth wrote in that he noticed that IE7 pukes on the same line of HTML; it just waits until you try to visit a different page before it crashes.

Python: Typed attributes using descriptors

2007-08-02T23:08:00.000-07:00

A few weeks a saw a post by David Stanek on Planet Python regarding using python descriptors to implement typed attributes. At the time, I needed something very similar and I've been trying to find something descriptors were good for (besides re-implementing the property builtin) so I decided to give his trick a try. The only problem was, I was coding on CalTrain at the time so I couldn't access his blog to reference the code in his post. The worst part was that I struggled with problems that, as it turns out, were pointed out in the comments to his posting (specifically, attributes implemented via descriptors would erroneously share state between instances of the classes the attributes were assigned to). By the time I got to the office and could consult David's blog post, this is the implementation I had working:

from collections import defaultdict

class TypedAttr(object):
    """Descriptor implementing typed attributes, converting
       assigned values to the given type if necessary

    Constructed with three parameters: the type of the attribute,
    an initial value for the attribute, and an (optional)
    function for converting values of other types to the desired
    type.
    
    If the converter function is not specified, then the type
    factory is called to perform the conversion.
    """
    __slots__ = ('__type', '__converter', '__value')
    def __init__(self, type, initvalue=None, converter=None):
        if converter is None:
            converter = type
        self.__type = type
        self.__converter = converter
        initvalue = self.convert(initvalue)
        self.__value = defaultdict(lambda: initvalue)

    def convert(self, value):
        if not isinstance(value, self.__type) \
           and value is not None:
            value = self.__converter(value)
            assert isinstance(value, self.__type) \
                   or value is None
        return value

    def __get__(self, instance, owner):
        if instance is None:
            return self
        return self.__value[instance]

    def __set__(self, instance, value):
        self.__value[instance] = self.convert(value)

With this, I could write my classes like so:

class Example(object):
    Name = TypedAttr(str)
    Type = TypedAttr(str, "cheezy example")

Mainly I'm using the typed attributes for data validation in objects populated from values supplied by an untrusted source. It would be really nice if I could compose descriptors to build more complex managed attributes (sort of like you can with validators in Ian Bicking's FormEncode package). Then I could make a descriptor, for example, Unsigned or NotNone and compose them with TypedAttr like so:

class CompositionExample(object):
    Name = TypedAttr(NotNone(str))
    Age = Unsigned(TypedAttr(int))

I'll admit I haven't put a whole lot of thought into it yet, but at first glance it appears that it would be impossible to compose descriptors in python. I would love to be proven wrong.

Python: Serializer benchmarks

2007-07-23T00:41:00.000-07:00

I am working on a project in which clients will be submitting more data than my current server implementation knows what to do with. The reason the current implementation doesn't use all of the submitted data is that I don't yet know what the quality of the data will be until the client is deployed in the wild. I want to record all of the submitted data, though, in the expectation that a future implementation will be able to use it. So I was considering formats for logging the submitted data such that it would be easy to parse in the future.

Since I'm already storing a subset of the submitted data in a database, the most obvious solution is to make a table of submissions which has a column for each submitted data element. However, it turns out that this is quite slow and given that I'm not sure how much of the extra data I'll ever need or when I may update the server implementation to use it, I hate to pay a hefty price to store it now. For now, I can consider the data write-only. If and when I need to use that data, then I can write an import script that updates the server database using the saved data.

So I've been considering simply logging the submissions to a file. It is considerably faster to append to a flat file than it is to write to a database -- which makes sense since the database supports read/write access, whereas I only need write-only access for now.

The next question is what format to write the data to the log file. I have a python dictionary of the submitted data; at first I considered writing the dictionary to the log file in JSON format. The JSON format is relatively easy to convert to/from python data structures and python has quality implementations to do it. Furthermore, unlike the pickle text format, it is trivial to visually interpret the serialized data. This latter point is also important to me since I need to be able to judge the quality of the data in order to discern what portions I can use in the future.

However, to my chagrin, it turns out that the JSON module I have been using, simplejson, is slower than I had imagined. Profiling of my server implementation found that, after the database update logic, serializing the submitted data into JSON format was my second largest consumer of CPU cyles. I hate the thought of wasting so much time logging the data when it is an operation that is essentially pure overhead.

Hence I started considering other serialization formats, benchmarking them as I went. Here are the results of my benchmarks:

Serializer	Run 1 (secs)	Run 2 (secs)	Mean (secs)
pyYAML 3.05	21953.18	25482.61	23717.89
pySyck 0.61.2	3107.06	2805.38	2956.22
pprint	2364.91	2368.42	2366.67
pickle	1509.31	1665.16	1587.23
pickle/protocol=2	1359.40	1330.71	1345.05
simplejson 1.7.1	710.78	604.13	657.46
cPickle	159.27	172.26	165.77
repr	73.50	77.24	75.37
cjson 1.0.3	63.94	74.28	69.11
cPickle/protocol=2	50.97	57.72	54.34
marshal	12.52	13.32	12.92

All numbers were obtained using the timeit module to serialize the dictionary created by the expression "dict([ (str(n), n) for n in range(100) ])".
The tests were run under Python 2.5 (r25:51908, Mar 3 2007, 15:40:46) built using [GCC 3.4.6 [FreeBSD] 20060305] on freebsd6. The simplejson, cjson, pyYAML, and pySyck modules were installed from their respective FreeBSD ports (I had to update the FreeBSD pySyck port to install 0.61.2 since it otherwise installs 0.55).

I guess I should not have been surprised, but it turns out that simply calling repr() on the dictionary is almost 9 times faster than calling simplejson.dumps(). In fact, taking repr() as a baseline (100%), I calculated how long each of the other serializers took relative to repr():

Serializer	Mean (secs)	Relative to Baseline
pyYAML 3.05	23717.89	31469%
pySyck 0.61.2	2956.22	3922%
pprint	2366.67	3140%
pickle	1587.23	2106%
pickle/protocol=2	1345.05	1785%
simplejson 1.7.1	657.46	872%
cPickle	165.77	220%
repr	75.37	100%
cjson 1.0.3	69.11	91.7%
cPickle/protocol=2	54.34	72.1%
marshal	12.92	17.1%

The numbers in the last column are how much longer it took to serialize the test dictionary using the given serializer than it was using repr().

So now I'm thinking of sticking with JSON as my log format, but using the cjson module rather than simplejson. cPickle's latest binary format (protocol=2) is even faster, but I would lose the ability to visually scan the log file to get a feel for the quality of the data I'm not currently using.

Now, before I get a horde of comments I should point out that I am aware that simplejson has an optional C speedups module. Unfortunately, it does not appear to be installed by default on either FreeBSD (my server) or on Windows (my current client). I wouldn't be the least bit surprised if the C version of simplejson is just as fast as the cjson module, but it doesn't matter if it isn't installed. As such, it looks like I'll be switching to cjson for my JSON serialization needs from now on.

Update 2007/07/25 07:07pm:: In response to paddy3118's comment, I added benchmarks for the python pprint module to the tables above.
Update 2007/07/27 12:26pm:: In response to David Niergarth's comment, I added benchmarks for pyYAML 3.05 and pySyck 0.61.2.

Python: Mapping arguments to their default values

2007-07-22T16:51:00.000-07:00

Using inspect.getargspec or the getargspec implemention in my previous post, we can build a dictionary mapping a callable's argument names to their default values:

def getargmap(obj, default=None):
    """Get dictionary mapping callable's arguments to their
       default values

    Arguments without default values in the callable's argument
    specification are mapped to the value given by the default
    argument.
    """
    spec = getargspec(obj)
    argmap = dict.fromkeys(spec[0], default)
    argmap.update(zip(spec[0][-len(spec[3]):], spec[3]))
    return argmap