Scrapy Jobs

We are looking for a Python developer to build a Proof of Concept (POC) focused on web scraping with a simple Tkinter-based desktop UI. The role involves scraping data from defined sources, processing it, and displaying/exporting results through a lightweight interface. Key Responsibilities: Develop Python scripts for web scraping Create a basic Tkinter GUI for user interaction Handle data extraction, cleaning, and storage (CSV/JSON) Ensure error handling and basic performance optimization Requirements: Strong knowledge of Python Experience with web scraping libraries (BeautifulSoup, Scrapy, Selenium, etc.) Basic to intermediate experience with Tkinter Ability to deliver a functional POC quickly Duration: Short-term / POC-based Type: Freelance / Contract

I need to lay the digital foundation for our motorcycle-parts store by turning the catalogues of our suppliers into a clean, reusable database. Phase 1 focuses on the full product catalogue of MRM; Phase 2 will add Alessia and any other future sources that use a similar structure. Core tasks • Write reliable scraping scripts (Python + Scrapy/BeautifulSoup/Selenium—choose the stack you master) that log in if required, paginate, and pull every product. • Extract every field we rely on: description, SKU, and the complete compatibility matrix (brand, model, year). • Download all images, rename them by SKU, and deliver them in a single, well-organised directory tree. • Clean and normalise the raw data, then export to a single master table structured by SKU. ...

I need every product currently live on captured and organised into a single CSV that I can import straight into Wix. The file must follow Wix’s bulk-upload template so that titles, descriptions, images, SKUs, prices, weights, categories, vendors and any additional attributes drop into the correct columns without extra mapping. Because the goal is to list these items on another website, I’m not interested in formats designed for Shopify, WooCommerce or Amazon—only a Wix-ready structure will work for me. Please scrape: • Product name • Primary image URL (plus secondary image links where available) • Full description (HTML preserved) • Weight/unit of measure • SKU / product code • Price (regular and sale, if shown) • Category...

I already have a Python scraper that targets a supplier site for bathroom furniture, but it’s throwing errors before it finishes a full pass. I need you to go into the existing code, track down the points of failure, and get the crawler running to completion without time-outs, missing fields, or malformed HTML parsing. The finished scraper must reliably collect every product detail, all variant combinations (sizes, finishes, anything the site offers), and each accessory that belongs with the main item. Once scraped, the data should be exported to a clean CSV structured for PrestaShop’s native import—parent products first, combinations next, accessories linked through the proper reference column. After the CSV looks good, the final step is to load everything into my Pres...

I need a dependable scraper that can crawl an online blog of roughly ten thousand posts and pull down every entry, complete with any comments attached to each post. The final dataset must be delivered in a clean, well-structured XML file because that is my preferred working format, but feel free to include additional JSON or raw HTML copies if they come out of your workflow naturally—extra formats are a bonus, not a requirement. Scope of work • Crawl every live post on the site, following all pagination and in-site links that surface original articles. • Capture full article content and pair the corresponding comments so they stay linked to the right post. • Preserve each post’s core details (title, body, URL slug, and whatever standard metadata your tool ...

I need a reliable script that will pull public-record information from an ordinary county website and reorganize it so I can work with the data quickly—ideally in a clean CSV or Excel file. That date is then run through another website to further qualify it against some requirements. The first site does not provide an export function, so the scraper will have to crawl the relevant pages, capture every field that appears in the public-record tables, and normalise names, dates, and addresses before saving. However, the second site has some export capabilities. Python with BeautifulSoup, Scrapy or a lightweight Selenium setup is fine, as long as the final code is readable and I can rerun it myself whenever new records appear. Please keep throttling, polite headers and retries in mind ...

I’m building a clean, city-level database of healthcare providers and want the information pulled directly from Google results (Maps or Search is fine). The scope is limited to three facility types—Hospitals, Chemists/Pharmacies, and Clinics—across the specific cities or regions I’ll supply once we start. The only fields I need for every record are: • Facility name • Full street address (including postcode) • Primary contact details (phone and, when visible, email) Please deliver the final data as a tidy CSV or Excel file, with consistent column headers and no duplicate entries. Python tools such as Scrapy, Selenium, BeautifulSoup or similar Google-compatible methods are welcome, provided you stay within public-data usage guidelines. I&rsqu...

I’m looking for a small, self-contained application that can pull structured data from one or more websites of my choice and drop it into CSV or JSON on demand. The key for me is flexibility: I want to be able to point the tool at a new URL, tweak the selectors or XPaths, and run it again without having to rewrite code from scratch. Please build it in a language you’re comfortable maintaining—Python with Scrapy or BeautifulSoup is fine, as is a compiled solution—so long as setup is straightforward and there are no paid dependencies. The program has to: • Handle pagination or infinite scroll automatically • Respect and throttle requests to avoid bans • Detect and bypass basic anti-bot measures (simple captchas, user-agent checks) • Let ...

CLARIFICATION: What I am looking to build a tool that scrapes existing content and new content when needed by selecting the websites I want (select all, select some) without duplicates if already scraped. The list of websites to be scraped shall be added progressively by the developer who creates the scripts. So, this is not a one time scraping project. I want to automate the collection of articles that analyse why startup businesses collapse. The first phase covers ten sites—mainly well-known business news outlets and respected entrepreneur blogs—and should gather up to 1,000 relevant pieces in total. To keep the workflow clean, the scraper must create a standalone configuration and output file for each site; once the process proves reliable I will quickly expand the list. ...

I’m standing up a production-grade ETL pipeline that visits a public-records website with Playwright (Python), extracts the legally public data every hour, cleans and normalizes it, then loads the results into Postgres on Supabase. Long-term maintainability and horizontal scalability are the primary goals, so the codebase should be modular, clearly documented, and ready for future contributors to extend without fear of breaking things. Core build expectations • Browser automation: headless Playwright with smart pacing, built-in retry logic, and respect for site rate limits. • Transformation layer: standardization, normalization, plus upfront cleansing and validation before anything ever touches the database. • Storage: well-designed Postgres schema on Supabase, ...

Tôi đang cần một đoạn script tự động thu thập “Thông tin và giá sản phẩm” từ Shopee nhưng chỉ trong một vài danh mục cụ thể (tôi sẽ gửi danh sách chi tiết ngay sau khi bắt đầu dự án). Mục tiêu – Lấy đầy đủ tên sản phẩm, giá hiện tại, giá gốc (nếu có), thuộc tính chính, tên shop, lượt bán, lượt thích, điểm rating và đường dẫn ảnh. – Crawler phải đi hết các trang trong danh mục, không bỏ sót, không trùng dữ liệu. – Kết quả xuất ra CSV / Excel; cột nào ra cột nấy, tiếng Việt không lỗi font. Yêu cầu kỹ thuật – Ưu tiên Python sử dụng Scrapy, Requests + BeautifulSo...

Note: the photo which I have uploaded it’s an chrome extension but I need an application. I need an application that can instantly pull contact details from university websites in any country and drop them straight into an Excel workbook. The scraper must capture the following for each person it finds: email address, phone number, social media profile links, full name, affiliation, department, and the exact profile URL the data came from. My ideal workflow is simple: I point the tool at a single domain or a list of university URLs, press start, and receive a neatly formatted .xlsx file. Because institutional sites vary widely—some rely on pagination, others on JavaScript-rendered staff directories—the program should tackle both static and dynamic pages, handle moderate ...