Check for Valid URLs Rapidly - Using Rotating IP and Proxies - Mac OS
$30-250 USD
Pagado a la entrega
I need a piece of software, preferably desktop based for Mac OS, that cycles through an increasing integer and enters it into a URL. The software should check if the URL leads to a valid product page, and if so, saving the integer in a list within the software's GUI.
The general template would require the base site address, and the starting integer. Both numbers defined by the user. For instance:
http://(sitenamehere)/cart/(integer here):1
So (sitenamehere) would be the site we're scraping info on, and (integer here) would be the number we are starting at. The software should increase by one integer for every check, and should use Rotating IPs and proxies to avoid bans. I would like to software to perform these checks as quickly as possible, looking to check about 800k-1 million product pages per day.
Nº del proyecto: #11128270
Sobre el proyecto
7 freelancers están ofertando un promedio de $152 por este trabajo
Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database Más
because I have developed the scraper before. for yellowpages, justdial and dx.com. auto increment in pages to scrape next page data and proxy script is implemented. I can provide you the software after minor changes. b Más