World Wide Web Consortium Lists: 400,000 emails

HTML 4.0, XML, PNG, CSS, DOM, and XQuery: These are but a few of the technologies to come out of the World Wide Web Consortium, commonly referred to as the W3C. We're proud to announce that MarkMail (which by the way uses all of those technologies!) has loaded the full W3C public mailing lists. They start in 1994 and cover 400,000 emails across 200 mailing lists.


With such a long and deep history it's fun to do a little archaeology: You can find the first mention of XML back in 1996. I tried to find the formal "XML 1.0" announcement and saw there wasn't one, but on launch day (February 10, 1998) you can find people complaining about rendering issues with the spec. Isn't that always the way with mailing lists? By the way, it's fun to use XML to search on the birth of XML.

Google first came up as a topic in August 1998, back when its domain ended "stanford.edu". That beats any other list by 5 months. The first mention of XQuery didn't come until January 2001, well after xml-dev and other lists were talking about it. I expect there's more chatter in the private W3C archives.

Finally, the first mention of MarkMail came in December 2007. And what a great post it was! :)

Loaded Perl: 530,000 emails

Perl is the duct tape of the internet. Created by Larry Wall in 1987 and made famous with his Programming Perl "camel book" published by O'Reilly, it's the tool sysadmins use to keep things running.


We're proud to announce we've finished loading the Perl.org mailing list history into MarkMail. A total of 530,000 emails across 75 lists. The lists don't go back to 1987 (boy that'd be cool if they did). But that's all right; who really needs tech support against Perl 1.000?

What we have here is traffic starting with the migration to the Perl.org setup in 1999:


Enjoy! And if anyone has earlier archives, let us know.

New Search Feature: "opt:nostem"

In the science of Information Retrieval there's a constant tug of war between precision and recall. As Wikipedia defines the terms, precision is the fraction of the documents retrieved that are relevant to the user's information need, and recall is the fraction of the documents that are relevant to the query that are successfully retrieved. Or as I define the terms, precision is how much of what you wanted you actually got, and recall is how much of what you got is what you wanted.

MarkMail increases recall by running stemmed searches. This loosens the query constraint so that searching for proxies will match proxy as well. Sometimes this is good, and sometimes we hear from users who don't like the behavior all that much! They want more precision.

So we're happy to announce a new feature, opt:nostem, that when added to the search string turns off stemming for that query. You can try it for yourself:

http://markmail.org/search/?q=proxies
http://markmail.org/search/?q=proxies+opt%3Anostem

Average Load Time: 0.1 Seconds

There are many challenges in running a high-traffic web site. Performance is a challenge we particularly focus on at MarkMail because users get frustrated if they have to wait more than a second for a reply.

The challenge in maintaining performance increases as more of a site's content gets built dynamically -- meaning on the fly in response to user requests rather than ahead of time where it can be directly served (like a McDonalds hamburger).

With MarkMail we build every page dynamically using XQuery. Even a page that at first blush seems as if it could be pre-built, like an individual email message, we actually build dynamically because we want to highlight the search terms from your query.

All this is why I was so happy to notice that Alexa.com calls us a "Very Fast" site...

• Markmail.org has a traffic rank of: 128,666 (UP 745,248)
• Speed: Very Fast (99% of sites are slower), Avg Load Time: 0.1 Secs

Here's some background on how Alexa tracks performance.

New Feature: Top 10 expands to Top 100

Every time you do a search on MarkMail the leftmost pane shows you the top 10 lists, senders, attachments, and message types for all emails matching your query. OK, it's not always 10 that you see. Sometimes it's more, sometimes less. Exactly how many you see depends on your browser size. But even if you're the proud owner of one of those new 17" MacBook Pro laptops with the 1920x1200 screen, the view maxes out around 25.

We've added a new feature to help improve this. When there are more values than will fit in the selection box, you'll see a "View more" link in the top right corner.


Clicking on "View more" shows the top 100 in an overlay.


Clicking on any of the values in the overlay will limit your search, same as clicking on a value in the short list. Enjoy!

A Place for Xen

At MarkMail you can now find your Zen. Or, to be more accurate, you can find your Xen.

Xen is an open source "hypervisor" (similar to VMWare) that enables operating system virtualization. It's supported by Citrix and used by Amazon EC2, among others.

I can joke about finding Xen at MarkMail because we recently loaded a bit over 100,000 messages from the Xen community. If you're into virtualization, enjoy!


If you want to compare VMWare with Xen, you'll find some good discussion in the archive.

PostgreSQL: More Traffic than MySQL (and a first Google spotting)

When we announced back in December we'd loaded the MySQL database mailing lists, we heard from several people who asked us to load the PostgreSQL lists also. We said we'd be happy to, and MarkMail now has 635,000 PostgreSQL emails loaded and searchable.

Comparing PostgreSQL and MySQL is kind of interesting. With all the talk about the LAMP (Linux/Apache/MySQL/PHP-Perl-Python) architecture you'd think MySQL had a lock on the open source database market, but based on simple message traffic analytics, PostgreSQL has a much higher level of community involvement. Looking at January 2000 onward, the MySQL lists have amassed 340,000 messages with about 3,000 new messages each month:


In the same time period, the PostgreSQL lists have hit 583,000 messages with 7,000 new each month:


I wouldn't have thought it, but there it is.

Also in the PostgreSQL lists we find the very first mention of Google in all of the messages loaded so far! The first Google sighting was on the pgsql-interfaces list, January 28, 1999, in a post by James Thomson:

"I've been using the Oracle Pro*C precompiler manual. I don't have the URL here at work but I found an online copy using www.google.com"

The first mention in another community happened on the xml-dev list a couple months later, March 10, 1999, in a post by Andrew McNaughton:

"You need a new search engine. I've recently been using www.google.com with results an order of magnitude better than what I got from altavista (though altavista still has it's place for more complex query definitions)."

Here's the query if you want to look for yourself:

http://markmail.org/search/google+order:df


Maybe as we load more community archives we'll get even earlier sightings.