How to build a basic web crawler or spider to crawl and index webpages?
Do you want to build your own web crawler to download and parse websites? Follow this thread to learn how to develop such an application.
I want to build a web crawler to crawl a certain number of websites to find latest updates, posts from those websites without the use of any feed reader. Is there a way to build such web crawler? Specify complete details about it and the requirements to start. You can suggest me any platform.