Automatically scrape the URLs in the Web Scraper every 24 hours | Voters

Automatically scrape the URLs in the Web Scraper every 24 hours

closed

Robin Dirksen

Created by Fleur Nouwens

September 6, 2023

Charl Haas

marked this post as

closed

We’ve received requests for an automated feature to scrape all URLs. However, due to the high server demands this would create, it is not feasible to implement. Manual selection allows us to maintain optimal performance and service reliability for all users.

Alexander Wijninga

marked this post as

in progress

Charl Haas

marked this post as

planned

Robin Dirksen

Supplement:
At Google Merchant Center, it is possible to have a product feed crawled every 24 hours, at a specific time. For example, 5:00am every day. With the watermelon web scraper, this would also be desirable, so that you, as a website administrator, can decide when watermelon will crawl the website again.
If this is not possible to set up, or possibly as an additional function, to be able to call the web scraper via the API to perform a recrawl. This allows the system to automatically instruct watermelon to start the recrawl in the event of a change.