We need to further automate our back-end crawler via EC2 & Rackspace. We also need to bidder to take over the crawling tasks and monitor the progress of the servers for a complete crawl of a few sites. More details to come in the private messages after bidding.
Thank you!