Diffbot APIs extract data from web pages: articles, blog posts, products -- or any other page using our Custom API Toolkit. Crawlbot, our intelligent spider, crawls entire sites and returns the full data in clean, readable JSON. Our Bulk API allows for the extraction of thousands or millions of URLs. Be notified whenever a Crawlbot crawl or Bulk API job completes.
You can find more information here.
Your webhook URL will be notified when your Crawlbot crawl finishes.