BadCheese.com

Ok, as I write my crawler, I've noticed that there are many pieces to the whole crawler puzzle.  I've got the part that pulls pages off the internet as fast as possible (that part works great), but that part is the easy part.  The hard part is doing something useful with the data once I've fetched it.  The first step is pulling URLs from the html for future crawling.  Storing/de-duping/… of urls is a whole other problem with doing a huge crawl, so I'll cover that later.

When I was working with the http://archiveteam.org guys pulling down geocities urls, I used my old crawler.  It just pulled urls and didn't think very hard about things.  However, after the fact, I realized that crawlers convert relative urls (i.e. "blah.html", "../blah.html", "../dir2/blah.html", "/blah.html") to absolute urls (i.e. http://blah.com/blah.html).  Many crawlers do this internally and the code to do it is not found easily on the internet and I think that I know why.

The reason there isn't just a regex or some code out there to convert relative urls to absolute urls is because urls are not always the same.  Computers don't write web pages most of the time, people do, so there are mistakes in them.

If a file (blah.html) is in a directory called "a", the absolute url could be http://blah.com/a/blah.html  However, if blah.html has links in it to "../../blah.html" – that's not a valid url, so it's catchable.  "../blah.html" is fine.  "../b/blah.html" is fine, "c/blah.html" is fine.  Get it?

But what happens if an original url is http://blah.com/a (no trailing slash).  Is the crawler pulling http://blah.com/a ("a" is the name of the file) or http://blah.com/a/index.html or http://blah.com/a/index.php or http://blah.com/a/index.htm ???  Also, if there are relative urls in that file, like "blah.html", does that mean http://blah.com/blah.html or http://blah.com/a/blah.html ???  The answer is that you have to crawl both.  The web server will serve up content in either case, and not all people put trailing slashes in their urls if they mean a directory, so you can't assume that you know which one is best, so you need to crawl both cases.  That's why there's no code out there to figure it out.  There's no one, good answer.

http://www.adjix.com has figured out the solution to bit-rot with url-shorteners. They have all of the standard stuff with regards to the normal url-shorteners, but they do one thing interesting. They allow for the user to back up their shortened url links to an amazon S3 bucket. What does this do? It allows you to OWN your shortened links! Yea, I set up a CNAME of url.badcheese.com to point to my amazon S3 bucket, gave adjix permission to write to my bucket and now when I make a shortened url, it fetches an actual file off of S3 and does the redirect through javascript. Why would someone like to own their shortened urls? Well, bit-rot http://en.wikipedia.org/wiki/Bit_rot is a problem with url-shorteners. If a url-shortener goes away, your tweets, IMs and all of your content looses a LOT of meaning. Take for example, the tweet: "Check this out: http://tinyurl.com/xxxx". Meaningless without the following page, right? If I own my own shortened urls, then I can take those shortened urls and move them around from machine to machine and do whatever I want to with them. I know that they'll always be there because I own them, and if I want to remove one, I just nuke it. Simple enough!

Anyway, hats off to the guys at http://www.adjix.com , also - I sent in a bug report last night at 11:30pm MT, and they answered and fixed the bug by the time I got up in the morning, so they're midnight hacker types and could really be onto something! I'm a big fan already.

Google Stock Chart on Google Finance: http://url.badcheese.com/57m5

http://Reteltechnologies.com - mostly just uses something like amazon's mechanical turk to have people watch surveillance videos and flag when something happens. Good for guys who own a 7-11 store, but I wouldn't use it.

http://everlater.com - travel site that combines many social sites to tell a story of your trip (the guys were very good at selling the product, but I don't think that I'd use it)

http://timzon.com - was just a customer support tool, I wouldn't use it.

http://TakePublishing.com - itunes for comic books. I'm not a huge comic book fan, but it has potential.

http://NextBigSound.com - for big record labels or bands to monitor which social site they need to focus their marketing on.

http://vanillaforums.com - I would use it! Great forum software - used by tons of big-boys.

http://sendgrid.com - Better 'transactional' email, like the email that you get from facebook and twitter (aka, Mr Jones is now following you on twitter). I don't need that.

http://SpryPlanner.com - Manager software for high-level overview of a software development package/group. Integrates into nagios, github, etc ... Not something I need.

http://Mailana.com - the least impressive tool, like linked-in, but for the major social sites. Combines all of your friends and makes meta-friend circles out of all of your 'best' friends and allows for you to search within those circles.

http://Rezora.com - The real estate email marketing tool. Impressive tool, but I don't have a need for it.

I must spend 10-12 hours on a computer every day most of it online in one form or another.

I've got a quad-core computer with 4GB of ram, a $300 Nvidia video card, the latest browser (Firefox 3.5) which has a turbo-speed JavaScript engine in it, but for some reason, my browser seems to hang every few minutes while waiting for something to happen and crappy resolution internet video is choppy and difficult to bear.  I must waste 30-60 minutes a day just waiting for my browser to wake up from doing something behind the scenes.  To put it bluntly, this pisses me off.

I'm an old-school Unix guy.  I use pine for my email reader.  I use xterms instead of the nicer gnu terminals, I turn off all of the fancy special effects in the window manager, …  I prefer a minimalistic, simple, not overly complicated interface to do my work.  When I put the network and supporting infrastructure for the company that I work for, I chose as simple as possible so that anyone can jump in and understand things if that needed to happen.  I do these things for many reasons, but the main reason is that complexity costs resources.  Simple = fast and manageable.  With today's computers, everything should be lightening-fast and waiting should be a thing of the past, but it isn't.  It seems that I'm waiting just as much nowadays as I was with my 4.7 MHz IBM PC XT back in the late 70's.  Sure, things have changed, but today software seems like it's written just slow enough to stay within human tolerances to being slow.  People optimize just enough so that it's acceptably fast, and no more.  To put it bluntly, this also pisses me off.

I want to build a website that competes with the big-boys and has all of the bells and whistles as the other guys.  It will be pretty and serve a function, but it will have one big difference.  It'll be lightening-fast!  It won't hang the user's browser doing things that could be done differently.  It won't thrash the user's swap just to go to the second page of search results.  It will transfer just enough data to get the job done, no more.  The JavaScript that it uses will be clean and fast and will not cause the browser to thrash or swap.  The infrastructure on the server-side will not use the latest/greatest technology just because it exists and might make things 2% easier to write or maintain.  I'll write the backend code as optimized as I possibly can.  I'll write C code when it needs to be tight, fast and small.  Use IP numbers instead of hostnames to avoid the 100ms DNS query.  Use a BerkeleyDB instead of SQL wherever necessary.  Send compressed, pipe-delimited text as the protocol instead of bloated XML.  Sure, it'll be harder to write.  Sure it'll be a little less flexible, but it'll be easy to maintain, easy to understand and will take fewer resources and run fast on slower (or practically any) hardware.  The experience will be a happy one and people like simple, happy, fast experiences.

I use Perl for some tasks, but if I write a script that I'll call thousands or millions of times, I start to see that Perl's startup time is a large part of the cost of my entire procedure.  Re-writing the simple task in C cuts down on the startup time (C is almost instantly doing real work after startup when Perl/python/java are still loading, parsing, allocating, etc …) and reduce the overall task my a very large margin.  This is a perfect example of programming ease versus optimal code.  I can whip out a Perl script in a minute to do something, but if I use it a lot, Perl is not the most optimal.

Another example is desktop apps versus browser apps.  I prefer desktop apps because of the speed and flexibility to the more modern browser-based apps of today.  Sure, they're great for some applications, but anything that requires any speed or major resource will still work WAY better on a user's desktop than in any browser-based application.  Let's see a browser-based AutoCAD or 3dsmax application – forget it.  It'll never happen.

I'm going to stick with the good old Unix philosophy – do one thing and do it well.  Trying to be the end-all, be-all application that is everything for everyone is not going to happen or make anyone very happy … ever … even in another 20 years when computers become even more unbelievably powerful.