Fun job (in) search
Via Anil’s daily links, Yahoo!’s looking for a search crawler engineer. The part of the description that caught Anil’s attention was websites such as online stores, discussion forums, and blogs
, but the part I like is:
Client software must be able to create logins, store and return HTTP cookies, support secure network connections, and run client-side scripting to access and discover content on systems such as these.
Writing a crawler that can make its way through a user registration form, then log in to a secure site, interpret Javascript, and suck everything down, sounds like a whole lot of fun. And a whole lot of potential to blow up, wreck sites, and crawl things it shouldn’t. If they can make it work, though, it would sure be a great reason way to distinguish themselves from the herd of second-tier search engines.
No comments yet.