Self-cleaning Web Crawler based on changes in our sitemap
To enhance our AI agent's capabilities, we need an automated and self-cleaning Web Crawler. This Web Crawler should run at a certain frequency, automatically adding new URLs and removing those no longer present on the sitemap. This feature would ensure our AI agent has the most up-to-date information, improving its efficiency and accuracy.
X
Xavier Grene
Is there any update on this feedback ? This seems like a must to have for any Ecom having new SKU's on their site on a regular basis.
M
Melvin van Rookhuizen
Fleur Nouwens an API call from our websites to Watermelon to crawl update changes would be a good addition
Fleur Nouwens
Melvin van Rookhuizen: Thanks for you input!
Fleur Nouwens
Hey thanks for your feedback! Following up on this:
- What specific frequency do you envision for the WebCrawler to run (e.g., daily, weekly)?
- Are there any specific types of URLs or content that should be prioritized or excluded by the WebCrawler?
- How should the WebCrawler handle URLs that are temporarily unavailable or return errors?