Ask Experts » Internet & Browsers » Other Internet topics »

How to build a basic web crawler or spider to crawl and index webpages?


Date: 07 Oct 2014   Posted By: nagabhyru revanth     Group: Internet & Browsers    Category: Other Internet topics   

I want to build a web crawler to crawl a certain number of websites to find latest updates, posts from those websites without the use of any feed reader. Is there a way to build such web crawler? Specify complete details about it and the requirements to start. You can suggest me any platform.

Do you want to build your own web crawler to download and parse websites? Follow this thread to learn how to develop such an application.


Author: [Anonymous]    19 Oct 2014      Member Level: Gold     Points : 2  (Rs 2)    Voting Score: 0

If you know coding or JAVA or PHP then you can build basic web crawler to get particular webpage index. You can Google as "how to build a web crawler" and you will get some results as an idea.

Author: Muddassir    01 Dec 2014      Member Level: Diamond     Points : 1  (Rs 1)    Voting Score: 0

If you want regular updates from a particular website then you should subscribe to their RSS feed. If you want to create a web crawler then you need to learn php and javascript.




Post Answer



You must Sign In to post a response.


Post Answer        

Read related posts: Crawl Rate    SEO    Web Development    





Related Questions
  • Return to Question Index
  • Ask Question to Experts


  • Top Contributors
    TodayLast 7 Daysmore...