You must Sign In to post a response.
  • Category: Other Internet topics

    How to build a basic web crawler or spider to crawl and index webpages?


    Do you want to build your own web crawler to download and parse websites? Follow this thread to learn how to develop such an application.

    I want to build a web crawler to crawl a certain number of websites to find latest updates, posts from those websites without the use of any feed reader. Is there a way to build such web crawler? Specify complete details about it and the requirements to start. You can suggest me any platform.
  • #6741
    If you know coding or JAVA or PHP then you can build basic web crawler to get particular webpage index. You can Google as "how to build a web crawler" and you will get some results as an idea.
    Regards,
    Hafeezur Rahman

  • #7043
    If you want regular updates from a particular website then you should subscribe to their RSS feed. If you want to create a web crawler then you need to learn php and javascript.


  • Sign In to post your comments