Hello, i must find a expert of scrapy framework.
When i run this command: scrapy runspider "[login to view URL]" the script must return the json datas in the std out.
Then i think that is necessary to create a correct pipeline and use a memory area to temporary store the datas and reload it from main script.
Do you undestand the problem?
I have experience developing scrappers for several tasks. This looks like a fun challenge. I do see some concerns loading the whole data set in memory, but they could be solved easily.
Hello. I am an experienced python developer with the scrapy framework.
Can you check pmb for further questions regarding your project.
Regards,
Arthur.