Hello,
This sounds like a very interesting task, though I would prefer more information and specifics. I am experienced in the lane of web scraping and accurately transporting the results out to excel/json on a large and automated scale. My method is using the Scrapy module in Python - doubtful that whatever servers you are running should have much of an issue.
Basically by the sounds of it, you want a ruleset built for each of your partners to cycle through and collect their data. My concern is the speed at which you want it - you make it sound as if you want a lot of sites indexed completely at a very fast speed. If these sites are true partners it shouldn't be as much of an issue, as they could simply allow your specific "bot" as they call them to pull whatever data it needs at any speed. However if we are talking about hundreds of thousands of records or more, the speed of the data would be a little more sluggish.
As for outputting results, that can be customized as well. I am not sure if you are asking for results to be pushed directly to a web interface, but be warned I would have to look into the feasibility of that.
If you are interested please contact me so we can work out some of the specifics of your needs; those will give the greatest indication of the level of effort this project will entail.
Respectfully,
Jake Thompson