Other professions get to use a variety of big words. Physicians, for [...]]]> People from Computer Science backgrounds often use the word “orthogonal.” I think that’s partly because it concisely describes an often-important concept, but mostly because it’s one of the few impressive words in our vernacular. (“Idempotent” is my other favorite CS term.)

Other professions get to use a variety of big words. Physicians, for example, have a rich, domain-specific vocabulary. So do economists. Lawyers have perhaps the best lexicon; not only do they get to use big, fancy words, but half of those words are in Latin.

Thus it was a pleasant surprise to read that a lawyer had adopted our humble “orthogonal” and earned kudos from the Justices of the United States Supreme Court for it.

“Slow”? What does that mean?

It means that when I start up Firefox, even if it’s already in the filesystem cache, the app’s Mac dock icon completes two full bounces before the browser window appears. Safari’s doc icon completes approximately [...]]]> I’ve found myself using Safari more and more, instead of Firefox, because Firefox is slow.

“Slow”? What does that mean?

It means that when I start up Firefox, even if it’s already in the filesystem cache, the app’s Mac dock icon completes two full bounces before the browser window appears. Safari’s doc icon completes approximately half of one bounce before its window appears. The just-released Chrome for Mac also displays its window in half a bounce.

Each icon bounce takes approximately one second. Thus I’ve developed a preference not to use Firefox–an otherwise excellent product–because it takes 1.5 seconds longer to start than the competition.

I’ll typically keep my web browser running for hours or even days after starting it up, so Firefox’s extra startup time is immaterial when amortized over the time between restarts. That’s my quantitative assessment, anyway. In practice, though, I tend to click the Safari icon because, qualitatively, it’s the fast one.

In software, speed matters.

There was one technical challenge: the project was using nginx as a webserver, and none of [...]]]> I recently helped an old friend with some prototyping and strategy for a stealth-mode project he’s doing. He wanted to use MobWrite, a framework that allows multiple users to do real-time, collaborative editing of content in a web page.

There was one technical challenge: the project was using nginx as a webserver, and none of the existing web gateway implementations for MobWrite were well-suited for use with nginx. MobWrite has a back-end server that speaks a custom protocol, and it needs a bit of additional software to translate HTTP requests from web browsers into the server’s protocol. The MobWrite distribution contains PHP, JSP, and Python pages that do this translation.

It is possible to add support for any and all of those languages to nginx, but doing so adds substantial complexity. Specifically, the event-driven design of nginx works well only with plugins that do their work in small, event-driven steps and do not block for I/O or lengthy computation. PHP, mod_python, and JSP, in contrast, all are based on a programming model where the code that handles the request expects to run synchronously until completion. Running such code inside the single-threaded, asynchronous nginx could allow a long-running PHP/Python/JSP request to block all other requests. One common solution is to use an nginx gateway plugin like mod_scgi that dispatches requests for these synchronous languages to a separate pool of processes. This enables a single nginx server to handle both synchronous PHP/Python/JSP requests and asynchronous static-content or HTTP-proxy requests without either type of request starving out the other. However, adding a separate process pool for the synchronous requests means adding configuration and tuning complexity, thus diminishing one of the inherent advantages of using nginx.

Determined to have my cake and eat it too–i.e., to support MobWrite and other types of content in a single nginx server while still enjoying nginx’s single-process simplicity and efficiency–I made a new nginx plugin called nginx-mobwrite. This plugin translates incoming HTTP requests from MobWrite clients into the MobWrite internal format and proxies them asynchronously to the MobWrite server. The nginx architecture includes an “upstream” framework that manages requests to back end servers as part of nginx’s event loop. The nginx-mobwrite uses this framework to dispatch translated requests to the MobWrite server and send the responses back to clients in an asynchronous manner, without blocking any other requests the nginx server might be handling at the same time.

The nginx-mobwrite plugin is available as open source, under v2.0 of the Apache License, on GitHub.

I [...]]]> I decided recently to try installing Windows 7 on my iMac, in a Boot Camp dual-boot configuration alongside OS X. I ultimately got it working, but I ran into one perplexing problem along the way. Here’s a summary of the symptoms and the solution, in case the information is helpful to other people.

I used the installation process described here. After I clicked the “Start Installation” button in step 8, the computer rebooted and greeted me with a blank black screen with a blinking white underline cursor in the upper left.

A bit of web searching led me to a solution that worked: I had to reset the computer’s PRAM. The PRAM apparently is an area of nonvolatile memory containing information about USB devices such as keyboards and mice. The full process I had to use was:

• Power off the computer.
• Unplug the keyboard and plug it into a different USB port.
• Turn on the computer.
• Immediately press the Command, Option, P, and R keys and hold them down until the system restarts.

I grabbed the .tar.gz distribution of the latest MySQL 5.1 release and installed [...]]]> I wanted to try out Ruby on Rails with MySQL on OS X 10.6. I ran into a few common problems along the way, so I am documenting the process here in case it might be helpful to others.

Step 1: Install MySQL

I grabbed the .tar.gz distribution of the latest MySQL 5.1 release and installed it in a nonstandard location on my computer. “Nonstandard” in this case means “somewhere other than /usr/local/mysql.” This detail would be important in a later step.

Step 2: Install the Ruby mysql gem

I’d installed a 64-bit version of MySQL, so the installation process for the mysql gem was:

sudo env ARCHFLAGS="-arch x86_64" gem install -t mysql \
  -- --with-mysql-config=/path/to/my/mysql/installation/bin/mysql_config

For a 32-bit MySQL installation, replace the “x86_64″ with “x86″ in the ARCHFLAGS.

Step 3: Set the DYLD_LIBRARY_PATH

I omitted this step initially, and when I tried to run “rake db:migrate” for my Rails application, I got an error message:

uninitialized constant MysqlCompat::MysqlRes

In the environment of each user account where you plan to run Ruby apps that depend on MySQL, do this:

export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/path/to/my/mysql/installation/lib

Note that most of the command-line tools supplied with binary builds of MySQL appear to be statically linked with libmysqlclient. This means that these tools may work flawlessly even if you’ve neglected to set the DYLD_LIBRARY_PATH. The Ruby mysql gem, however, is dynamically linked to libmysqlclient, so it is essential that the MySQL lib directory be included in your DYLD_LIBRARY_PATH if MySQL is installed in a nonstandard location.

For most other Unix systems, the environment variable to set is LD_LIBRARY_PATH rather than DYLD_LIBRARY_PATH.

Twitter Data Analysis: An Investor’s Perspective

As the author notes, it was possible to deduce so many key metrics because “Twitter uses auto-incrementing ID numbers (1,2,3,4…) for both users and tweets.”

Auto-incremented ID numbers are a common web development practice. When you run an [...]]]> This post on TechCrunch contains quite a nice analysis of Twitter’s user behavior:

Twitter Data Analysis: An Investor’s Perspective

As the author notes, it was possible to deduce so many key metrics because “Twitter uses auto-incrementing ID numbers (1,2,3,4…) for both users and tweets.”

Auto-incremented ID numbers are a common web development practice. When you run an online business, you and/or your users create data–hopefully lots of data. Each record of data needs a unique identifier that you can use to go find that bit of data for later use. Automatically-incrementing integer IDs are a common implementation choice because they have some desirable properties:

• Compactness – Shorter URLs are generally a good thing. In addition, IDs that can fit into a 32- or 64-bit integer representation allow for efficient storage in a database and efficient manipulation within most programming languages.
• Uniqueness – It is trivially easy to generate unique IDs with an auto-incrementing approach: keep track of the last ID you generated, and add one.
• Efficiency – Generating the next sequential ID value doesn’t take very long. In more precise terms, it takes O(1) time; the speed with which you can generate the next integer is a constant, and this operation will not get slower as your data size grows.
• Convenience – Most databases have a built-in feature that can auto-generate auto-incrementing IDs.

Despite these virtues, auto-incrementing IDs can reveal information about the rate at which you are accumulating customers or content. If your user IDs are sequentially generated, for example, a competitor need only sign up for a new account on your website once a day to determine how fast your user base is growing. You may not mind that risk; the engineering team at Twitter probably discussed this issue when designing their data model and consciously decided not to obscure their ID values.

In cases where you do want to avoid using auto-incrementing ID values, some alternative ID generation strategies are:

• Random – Generate a random number every time you need a new ID. Guaranteeing uniqueness requires a bit more work with this approach, though.
• Random increment – When generating a new ID, instead of adding one to the previously generated ID, add a random number. This approach will exhaust the range of your chosen integer representation more quickly than the traditional auto-increment, though, so it is perhaps best used with a 64-bit representation.
• UUID – Use UUIDs instead of integers as IDs. UUIDs have the properties of efficiency and uniqueness (even with random UUID generation, there are enough bits to make duplicates very rare) but are less compact than a typical auto-incrementing numeric ID. A UUID occupies 36 bytes in human-readable form or 16 bytes (128 bits) in binary form.

As described earlier, I tried the OpenID plugin for WordPress, but that plugin required a PHP extension not supported by my web hosting provider.

I subsequently found the WordPress RPX plugin that supports OpenID. RPX is an outsourced authentication service run by a [...]]]> This blog now allows readers to post comments if they authenticate via OpenID.

As described earlier, I tried the OpenID plugin for WordPress, but that plugin required a PHP extension not supported by my web hosting provider.

I subsequently found the WordPress RPX plugin that supports OpenID. RPX is an outsourced authentication service run by a company called JanRain. The WordPress RPX plugin calls the RPX servers, which in turn do the OpenID handshaking with one’s OpenID provider.

Conveniently, RPX also supports authentication via one’s Facebook login. In all cases, the user’s password is sent only to the identity provider (Yahoo, Google, Facebook, etc) and is never shared with the RPX servers or my own site.

There is a plugin for WordPress that adds OpenID support. I downloaded it, installed it, and was soon up and running with a non-working OpenID implementation.

When I [...]]]> I wanted to add OpenID support to my blog so that people could authenticate using their credentials from other sites (Yahoo and Google, for example) to post comments.

There is a plugin for WordPress that adds OpenID support. I downloaded it, installed it, and was soon up and running with a non-working OpenID implementation.

When I entered the service provider endpoint URI for one of my OpenID-providing accounts into the right field in the WordPress login form, the WordPress OpenID plugin replied with a cryptic error message: “Could not discover an OpenID identity server endpoint at the url.”

After a couple of hours of troubleshooting, including adding debugging code to the OpenID plugin, I found the root cause of the problem. OpenID uses a standard called XRDS. XRDS is based on XML. The OpenID plugin for WordPress supports two specific XML-supporting extensions for PHP (the programming language in which WordPress and its plugins are written), one named “domxml” for PHP4 and another named “dom” for PHP5. My web host provides PHP, but not either of those XML extensions.

On the bright side, my web host’s PHP installation does appear to have a different XML-supporting extension for PHP, one called simply “xml.” If I succeed in modifying the WordPress OpenID plugin to work with that as a third option, I’ll submit a patch to the maintainers.

The printer support in 10.6 already includes [...]]]> For a couple of years, I’ve used an HP all-in-one printer/scanner device. Up through OS X 10.5, HP provided a scanning application that made it reasonably easy to scan paper documents into multi-page PDFs. That application, however, stopped working under OS X 10.6 Snow Leopard.

The good news

The printer support in 10.6 already includes a built-in scanning application. In the Finder, navigate to your home directory, then Library -> Printers. Click on the entry for your scanner, and you’ll get a window with Printer and/or Scanner tabs (depending on the capabilities of the device). From the Scanner tab, you can scan documents into PDF or various image formats.

The bad news

If the document you want to scan is more than one page long, this Scanner app will produce a multi-page PDF only if you use the document feeder on the scanner. The document feeder is a fine solution if the documents are printed on only one side of the paper. If you have two-sided documents to scan, and your scanner doesn’t have a duplex feeder, you can use the flatbed scanning mode to scan a page at a time–but you’ll end up with a collection of separate PDFs, one per page.

Solution

I haven’t found a way to produce a multiple-page PDF in one step without either 1) getting a different scanning application or 2) getting a scanner with a duplex document feeder. I have, though, managed to automate the process of concatenating multiple PDFs into a single document.

The first step is to create an application that concatenates PDFs. This is much easier than it sounds:

1. Start up the Automator application
2. Automator will ask you to “choose a template for your workflow.” Pick the one named “Application.”
3. Drag and drop Actions from the menus on the left to create a workflow that looks like this:
   
   
4. From the Automator, save this workflow somewhere. You now have an application that concatenates PDFs.
5. Optionally, put this application in your Pictures folder, or put it somewhere else and make a symlink to it in your pictures folder.

Now that you have a PDF-concatenator application, you can use it from the Finder by selecting the PDFs that you want to concatenate and then dragging-and-dropping them onto the app.

Usage

When you scan documents from the Printer/Scanner app, they end up by default in your Pictures folder with names like Scan.pdf, Scan 1.pdf, etc. I created an empty file called Scan.pdf in that directory to make sure that all my actual scans would be named sequentially as Scan 1.pdf, Scan 2.pdf, and so on.

Based on this setup, my procedure for scanning a multi-page document into a single PDF using the flatbed scanner is:

1. Open up the Printer/Scanner window and use it to scan each page.
2. In the Finder, go to the Pictures folder.
3. Select the PDFs representing the scanned pages.
4. Drag and drop the selected files onto the PDF-concatenator application’s icon.
5. The combined document will open in Preview. From Preview, save it as whatever file name you like.

That last step–saving the file from Preview–is essential. The Automator action that does the actual concatenation puts the temporary file in a randomly-named file under /var/folders. My automator script, though, moves this temporary file to the Trash after opening it in Preview. If the document you’ve just scanned contains sensitive data, such as a bank statement, I recommend that you 1) from Preview, save your document someplace secure, like an encrypted disk image, and 2) from the finder menu, use the Secure Empty Trash operation to delete the temporary file.