HTML encoding of form inputs

I suppose this is common knowledge amongst professional web developers but I just discovered myself that if a user enters characters into a HTML form input that is not representable in the character set of the page the form is in, browsers will HTML-encode the non-representable characters when the form is submitted. I just spent over an hour assisting a coworker to track down a bug in one of our web applications that was due to this poorly-documented -- but reasonable -- behavior.

I say "reasonable" because, as obscure as it is, this is really the best thing I think a browser can do given the situation.

To recap, here is the scenario:

• You have a web page with a form in it that is served using some locale-specific encoding. In our case it was Shift-JIS, but the default ISO8859-1 encoding leads to the same problem.
  
• The user enters text into a form input field that is not representable in the displayed page's character set or encoding. For example, entering Cyrillic characters into a form displayed on an ISO8859-1 page.
  
• When the user submits the form, the browser tries to convert the inputs to the encoding of the page. Any character not representable in that page's character set or encoding has its Unicode character code point encoded as an HTML numeric character reference (e.g. DŽ).
  
• The web application or CGI receiving this input needs to a) know the character encoding of the page that was used to submit the form data so it knows how to interpret the data as characters and b) be prepared to convert any embedded HTML numeric character references back to their corresponding characters.
  

I like that last part where web applications (or CGIs) have to know the encoding of an HTML page served to the client in order to be able to properly parse input from that client. This fact shatters any remaining fantasies I had of HTTP being stateless.

Anyway, the real surprise is that a web application or CGI needs to be prepared to unencode HTML entities in form input. I quick check of perl's CGI.pm and python's cgi module indicates that neither of them do entity decoding of inputs automatically. And considering that information on the web regarding this behavior is sparse , I suspect that most web developers are unaware of it. At the time of writing, I can only find two references [1][2] that document HTML character reference encoding in the scenario described above.

Luckily, there is a really simple solution: always serve pages in UTF8 encoding and always expect form input to be in UTF8 encoding. One of the many great things about UTF8 encoding is that all characters are representable, so you never have to worry about the browser resorting to HTML character reference encoding.

Sharp pointy sticks

This past Saturday my wife and I bought recurve bows with the intent to make a hobby out of archery. It turns out there are a number of parks with free-to-the-public archery ranges in the Bay Area. A couple of the ranges we have found are:

• Huddart County Park in San Mateo county near Woodside.


• Stevens Creek Park near Cupertino

We actually got started a couple of weeks ago when we went for a free lesson from the Kings Mountain Archery Club. That was a lot of fun and our instructors were very friendly, helpful, and patient. They give free two-hour introductory lessons about once a month; information about how to signup is available on their web site.

We picked up our bows and arrows at a shop intimidatingly called Predator's Archery down in Gilroy. Much like our experience with Kings Mountain Archery, the staff at Predator's Archery were pretty friendly and helpful and they offer a great "starter package" which includes everything you need as well as 5 lessons.

In other news: Subversion still sucks

OK, I've had 4 months to get used to Subversion now. And it is growing on me. Or perhaps it is Stockholm Syndrome. But there are still a lot of annoyances.

After being bashed as a "troll" by one of Subversion's authors after daring to suggest it wasn't all ponies and rainbows, I thought I would check to see if others where sharing my pain transitioning from CVS to Subversion.

Not surprisingly, I did. I found a wonderful summary of all the frustrations I've been experiencing, thoughtfully compiled by no less than David O'Brien of the FreeBSD community.

I would add to his list, as my friend John pointed out in comments to my previous post, that it is really annoying to have to depend on external tools (ironically, CVS) to see commits across branches.

Serving file downloads with non-ASCII filenames

Recently, while helping out one of my coworkers, it came to my attention that there is no universally-agreed on way to download a file to a web browser while suggesting a filename that contains non-ASCII characters.

The common way to tell a browser to download a file (rather than try to display it in-browser) is to include a Content-Disposition header in the HTTP response; the header's value should be "attachment". Additionally, the server can include a filename parameter in the Content-Disposition header as a suggestion to the browser for what filename to save the file as.

As a bit of history, the Content-Disposition header was originally defined in RFC 1806 which was obsoleted and replaced by RFC 2183. However, the Content-Disposition header was originally defined for use in MIME messages and, while RFC 2616 (HTTP 1.1) makes reference to the Content-Disposition header, it does so only to note that:

Content-Disposition is not part of the HTTP standard, but since it is widely implemented, we are documenting its use and risks for implementors.

Luckily, while not officially standardized for use in HTTP, the Content-Disposition header is "widely implemented" indeed; it seems that all modern browsers implement the header. If the web server responds to a request with application/octet-stream data and a Content-Disposition header of "attachment", your browser will display the familiar "Save As..." dialog. If the server included a filename parameter in that Content-Disposition header, your browser will likely pre-fill the filename input field of the "Save As..." dialog with the specified filename.

But here is where things start getting murky.

RFC 2183, skirts the issue of international filenames by disclaiming responsibility:

Current [RFC 2045] grammar restricts parameter values (and hence Content-Disposition filenames) to US-ASCII. We recognize the great desirability of allowing arbitrary character sets in filenames, but it is beyond the scope of this document to define the necessary mechanisms.

So, as long as the downloaded files' names are always representable in the ASCII charactet set, any browser should properly display the filename (although I've seen rumors that some browsers, such as IE, do enforce a limit on the length of the filename). However, I work at a Japanese company, making products largely for the Japanese market, so we don't have the privilege of assuming the whole world is ASCII.

By the way, in case you are curious, even iso-8869-1 (latin1) isn't consistently supported across browsers so Europeans are left high-and-dry too.

You are probably thinking, like I was, that surely this is a solved problem. And actually, it is. Kind of. The Content-Disposition header originates with the MIME protocol which, since the publication of RFC 2231 in 1997, now supports non-ASCII character encodings for header values. So, for example, the filename "foo-ä.html" can be represented in the Content-Disposition header like so:

Content-Disposition: attachment; filename*=UTF-8''foo-%c3%a4.html

The problem is that few browsers actually implement this RFC 2231 syntax. For example, Firefox 3.6 and Opera 10 appear to support the RFC 2231 syntax. On the other hand, for Internet Explorer, Microsoft's developers choose to simply perform URL-style percent-decoding and then interpret the result as bytes of UTF8-encoded characters. So a server would need to send the Content-Disposition header as

Content-Disposition: attachment; filename="foo-%c3%a4.html"

for an MSIE user to see "foo-ä.html" in the "Save As..." dialog.

Despite requests for IETF working group members to fix it, Google's Chrome browser also does not comply with RFC 2231, preferring to follow Microsoft's lead and use simple URL-style percent decoding.

As a result, there is no consistent cross-browser way to suggest a non-ASCII filename for a file download. I'm sure it doesn't help that the Content-Disposition header has never formally been part of the HTTP specification, but yet it is used by all major browsers to implement file download functionality.

Julian Reschke has compiled a test suite and publishes a nifty page illustrating all of incompatibilities between browsers regarding handling of the Content-Disposition header. In addition, as part of the IETF Network Working Group, he is working on an RFC to formally define the interpretation of the Content-Disposition header in the HTTP context.

Unfortunately, because the ambiguity has been left unresolved for so long, some web servers have adopted the MSIE/Chrome encoding technique for their non-ASCII filenames. Actually, my gut feeling is that probably most have, although I don't have any hard numbers to back up that claim. The good news is that since the MSIE/Chrome encoding is only used for parameters in the form filename="..." while the RFC 2231-style encoding used by Firefox, Opera, and Julian's proposal uses filename*=... it is possible for the two to coexist in the same Content-Disposition header (note the presence of the * in the RFC 2231 format to differentiate it).

In fact, probably the most important section of Julian's proposal is section 4.2 where defines the HTTP client's behavior when the server responds with both filename=... and filename*=..., allowing for an easy upgrade path for MSIE and Chrome.

For now, however, Julian's test results show that when presented both traditional and extended formats, only Firefox and Opera will select the extended filename*=... format.

This opens an opportunity for those of us that need to serve file downloads containing non-ASCII filenames: we can include the filename in non-standard encoding supported by MSIE and Chrome first in the Content-Disposition header, followed by the filename in extended RFC 2231 encoding. According to Julian's tests, MSIE and Chrome will always take the first parameter while Firefox and Opera will properly selecte the extended-syntax parameter, no matter what order it appears in.

For example, if the server includes the header:

Content-Disposition: attachment; filename="foo-%c3%a4.html" filename*=UTF-8''foo-%c3%a4.html

all four major browsers should properly display the filename "foo-ä.html" in the "Save As..." dialog. Unfortunately, WebKit-based browsers, like Apple's Safari browser, would display the raw percent-encoded value "foo-%c3%a4.html" as the filename. At least for now, though, I'm afraid this is the best we can do.

Sharon Jones and the Dap Kings

This past Friday we had the chance to see Sharon Jones and the Dap Kings at the Warfield in San Francisco. The music was fun and Sharon was an amazing performer and entertainer. In addition to her own high-energy dancing around the stage, she invited a few of the audience onto the stage to dance with her. This was sort-of a crap-shoot as at least one person's dance skills ranked somewhere between Elaine from Seinfeld and epileptic seizure victim. However, one guy who referred to himself as merely "The Gator" got up on stage in full 70's era leisure suit and that boy could dance. Sharon even turned the spotlight over to him and let him go to town.

I'll be damned if someone didn't manage to get a video of it and has already uploaded it to Youtube:



The opening act was The Heavy, whose is perhaps best known for their song "How Do You Like Me Now?" that Kia uses in car ads featuring bad-ass stuffed animals. Except for the horns, I wouldn't have thought that they would have been a good match for Sharon Jones and the Dap Kings. But they were another high-energy act that really got the crowd riled up for the main act.

Overall it was a great show. We had some punk kids causing trouble in front of us, but eventually security took care of them and we were able to enjoy the rest of the show.

Experiences with Clipper

While I was away in Japan, Caltrain and the other Bay-Area transit agencies teamed up to introduce the Translink Clipper payment system. I was excited at the prospect of an easy-to-use IC-card payment system like the Suica system I enjoyed using in Tokyo, and I was tired of having to buy new 8-ride tickets every few days, so I got a Clipper card and tried it out.

The card is easy enough to use, but the experience refilling my balance on the card has been less than stellar. I'm not about to tie my Clipper card to my credit card account, effectively turning it into a backdoor for a thief to clean me out (the "autoload" feature they push on their website). I was hoping to be able to refill it manually using an automated machine, just like I always did with my Suica card.

It turns out that there are a whopping 7 "Add Value" machines in the entire Bay Area: two in San Rafael, one in Sausalito, one in Oakland, two in downtown San Francisco, and one on the freaking Golden Gate Bridge (I guess for the toll-booth workers?). Fortunately, for those of us whose commute doesn't include one of those 7 locations, you can also add to your Clipper card balance at most Walgreens locations.

And here is where my experience with Clipper takes a nose-dive. First, not all Walgreens locations are equipped to add money to your Clipper card balance. There is, however, a map of the locations on the Clipper website. Unfortunately for me, the Walgreens one block from my office in San Mateo is not on the list and, when asked in person, I received the same look one would expect had I asked them if sold live Jabberwockies.

So, each time I need to refill the balance on my card, I hop in the car and drive two miles to the Walgreens nearest my home. Two miles isn't particularly far, but it does feel a little odd to have to drive somewhere so I can pay for public transportation. Anyway, I only have to do it once every couple of weeks, so it isn't a big deal; just a minor annoyance.

What is more annoying about the experience is that apparently only managers have the specialized training necessary to work the Clipper card "add value" device they keep behind the counter. So the poor checkout clerk has to page for the manager who, after finishing his dooby, mosies up to the front counter and fumbles with the machine until (hopefully) it actually has the balance on it I paid for. This past time, my wife and I had to repeatedly tell other customers that they might want to get in the other line because we were going to be a while. Ten minutes in fact.

The good news is that, with the name change from Translink to Clipper, it looks like they have added an option to refill your card balance online. I'm planning on trying that the next time I need to update my balance. I might miss the biweekly visit to the bloodshot-eyed Walgreens manager, though.

Crappy Code Hopscotch

I'm officially coining the term "crappy code hopscotch" to refer to the stupid games you have to play to workaround crappy code. I guess it could equally well refer to that feeling of being surprised by the effects of crappy code in any otherwise simple task, which might not be altogether unlike the feeling of unpleasant surprise you would get if someone were to throw a pile of dog poo in your hopscotch square.

The term popped into my head today while doing some MySQL wrangling; I was testing a stored function that called LOWER() on the results of CONCAT_WS(). Sounds simple enough: lower-case the result of concatenating strings with a separator. Check this output from MySQL 5.1:

mysql> SELECT LOWER(CONCAT_WS(' ', 'MySQL', 'scores', 'a', 0));
+--------------------------------------------------+
| LOWER(CONCAT_WS(' ', 'MySQL', 'scores', 'a', 0)) |
+--------------------------------------------------+
| MySQL scores a 0                                 |
+--------------------------------------------------+

Silent lower-case fail.

The problem, it seems, is that CONCAT_WS() doesn't convert the numeric argument to a string, but rather decides to convert *all* of the parameters to BINARY types and, as a result, returns a BINARY value. To it's credit, at least LOWER() is documented as being a no-op on BINARY values, hence the useless output shown above. What amazed me is that not just the undocumented, unintuitive behavior of CONCAT_WS() but that MySQL did not emit a single warning when LOWER() returned a value without, you know, lower-casing it.

So out of nowhere, I find myself playing crappy code hopscotch. I can explicitly either cast the numeric argument to CONCAT_WS() to a string or else let CONCAT_WS() return a BINARY value and explicitly convert that back to a string before passing it to LOWER().

Two crappy boxes to pick from and I got to put my foot in one of them.