Fun job (in) search

Via Anil’s daily links, Yahoo!’s looking for a search crawler engineer. The part of the description that caught Anil’s attention was websites such as online stores, discussion forums, and blogs, but the part I like is:

Client software must be able to create logins, store and return HTTP cookies, support secure network connections, and run client-side scripting to access and discover content on systems such as these.

Writing a crawler that can make its way through a user registration form, then log in to a secure site, interpret Javascript, and suck everything down, sounds like a whole lot of fun. And a whole lot of potential to blow up, wreck sites, and crawl things it shouldn’t. If they can make it work, though, it would sure be a great reason way to distinguish themselves from the herd of second-tier search engines.


No comments yet.

Name (required)
E-mail (required - never shown publicly)
Your Comment (smaller size | larger size)
You may use <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <del datetime="" cite=""> <dd> <dl> <dt> <em> <i> <ins datetime="" cite=""> <kbd> <li> <ol> <p> <pre> <q cite=""> <samp> <strong> <sub> <sup> <ul> in your comment.