Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Code Project
CODE PROJECT For Those Who Code
  • Home
  • Articles
  • FAQ
Community
  1. Home
  2. General Programming
  3. Java
  4. A Web Crawler code...help

A Web Crawler code...help

Scheduled Pinned Locked Moved Java
csharpjavahtmlgraphicsdesign
10 Posts 3 Posters 0 Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S Offline
    S Offline
    sangeeta2009
    wrote on last edited by
    #1

    i have got a code for Web Crawler in java language.... Following is applet code...

    import java.applet.Applet;
    import java.text.*;
    import java.awt.*;
    import java.awt.List;
    import java.awt.event.*;
    import java.util.*;
    import java.util.*;
    import java.net.*;
    import java.io.*;

    /* <applet code=WebCrawler height=400 width=400>
    </applet>
    */

    public class WebCrawler extends Applet implements ActionListener, Runnable {
    public static final String SEARCH = "Search";
    public static final String STOP = "Stop";
    public static final String DISALLOW = "Disallow:";
    public static final int SEARCH_LIMIT = 50;

    Panel   panelMain;
    List    listMatches;
    Label   labelStatus;
    
    // URLs to be searched
    Vector vectorToSearch;
    // URLs already searched
    Vector vectorSearched;
    // URLs which match
    Vector vectorMatches;
    
    Thread searchThread;
    
    TextField textURL;
    Choice    choiceType;
    
    public void init() {
    
    // set up the main UI panel
    panelMain = new Panel();
    panelMain.setLayout(new BorderLayout(5, 5));
    
    // text entry components
    Panel panelEntry = new Panel();
    panelEntry.setLayout(new BorderLayout(5, 5));
    
    Panel panelURL = new Panel();
    panelURL.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
    Label labelURL = new Label("Starting URL: ", Label.RIGHT);
    panelURL.add(labelURL);
    textURL = new TextField("", 40);
    panelURL.add(textURL);
    panelEntry.add("North", panelURL);
    
    Panel panelType = new Panel();
    panelType.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
    Label labelType = new Label("Content type: ", Label.RIGHT);
    panelType.add(labelType);
    choiceType = new Choice();
    choiceType.addItem("text/html");
    choiceType.addItem("audio/basic");
    choiceType.addItem("audio/au");
    choiceType.addItem("audio/aiff");
    choiceType.addItem("audio/wav");
    choiceType.addItem("video/mpeg");
    choiceType.addItem("video/x-avi");
    panelType.add(choiceType);
    panelEntry.add("South", panelType);
    
    panelMain.add("North", panelEntry);
    
    // list of result URLs
    Panel panelListButtons = new Panel();
    panelListButtons.setLayout(new BorderLayout(5, 5));
    
    Panel panelList = new Panel();
    panelList.setLayout(new BorderLayout(5, 5));
    Label labelResults = new Label("Search results");
    panelList.add("North", labelResults);
    Panel panelListCurrent = new Panel();
    panelListCurrent.setLayout(new BorderLayout(5, 5));
    listMatches = new List(10);
    panelListCurrent.add("North", listMatches);
    labelStatus
    
    L 1 Reply Last reply
    0
    • S sangeeta2009

      i have got a code for Web Crawler in java language.... Following is applet code...

      import java.applet.Applet;
      import java.text.*;
      import java.awt.*;
      import java.awt.List;
      import java.awt.event.*;
      import java.util.*;
      import java.util.*;
      import java.net.*;
      import java.io.*;

      /* <applet code=WebCrawler height=400 width=400>
      </applet>
      */

      public class WebCrawler extends Applet implements ActionListener, Runnable {
      public static final String SEARCH = "Search";
      public static final String STOP = "Stop";
      public static final String DISALLOW = "Disallow:";
      public static final int SEARCH_LIMIT = 50;

      Panel   panelMain;
      List    listMatches;
      Label   labelStatus;
      
      // URLs to be searched
      Vector vectorToSearch;
      // URLs already searched
      Vector vectorSearched;
      // URLs which match
      Vector vectorMatches;
      
      Thread searchThread;
      
      TextField textURL;
      Choice    choiceType;
      
      public void init() {
      
      // set up the main UI panel
      panelMain = new Panel();
      panelMain.setLayout(new BorderLayout(5, 5));
      
      // text entry components
      Panel panelEntry = new Panel();
      panelEntry.setLayout(new BorderLayout(5, 5));
      
      Panel panelURL = new Panel();
      panelURL.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
      Label labelURL = new Label("Starting URL: ", Label.RIGHT);
      panelURL.add(labelURL);
      textURL = new TextField("", 40);
      panelURL.add(textURL);
      panelEntry.add("North", panelURL);
      
      Panel panelType = new Panel();
      panelType.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
      Label labelType = new Label("Content type: ", Label.RIGHT);
      panelType.add(labelType);
      choiceType = new Choice();
      choiceType.addItem("text/html");
      choiceType.addItem("audio/basic");
      choiceType.addItem("audio/au");
      choiceType.addItem("audio/aiff");
      choiceType.addItem("audio/wav");
      choiceType.addItem("video/mpeg");
      choiceType.addItem("video/x-avi");
      panelType.add(choiceType);
      panelEntry.add("South", panelType);
      
      panelMain.add("North", panelEntry);
      
      // list of result URLs
      Panel panelListButtons = new Panel();
      panelListButtons.setLayout(new BorderLayout(5, 5));
      
      Panel panelList = new Panel();
      panelList.setLayout(new BorderLayout(5, 5));
      Label labelResults = new Label("Search results");
      panelList.add("North", labelResults);
      Panel panelListCurrent = new Panel();
      panelListCurrent.setLayout(new BorderLayout(5, 5));
      listMatches = new List(10);
      panelListCurrent.add("North", listMatches);
      labelStatus
      
      L Offline
      L Offline
      Lost User
      wrote on last edited by
      #2

      This is far too much code for anyone to read through. Please identifiy the line of code where the error occurs and post only the small amount of code near that line. Have you looked at the documentation to see what an IllegalThreadStateException is, and what may be its causes?

      S 1 Reply Last reply
      0
      • L Lost User

        This is far too much code for anyone to read through. Please identifiy the line of code where the error occurs and post only the small amount of code near that line. Have you looked at the documentation to see what an IllegalThreadStateException is, and what may be its causes?

        S Offline
        S Offline
        sangeeta2009
        wrote on last edited by
        #3

        public static void main (String argv[])
        {
        Frame f = new Frame("WebFrame");
        WebCrawler applet = new WebCrawler();
        f.add("Center", applet);

        /* Behind a firewall set your proxy and port here!
        */
        Properties props= new Properties(System.getProperties());
        props.put("http.proxySet", "true");
        props.put("http.proxyHost", "webcache-cup");
        props.put("http.proxyPort", "8080");

                    Properties newprops = new Properties(props);
                    System.setProperties(newprops);
        

        /**/

                    applet.init();
                    applet.start();
                    f.pack();
                    f.show();
            }
        

        }

        Ok ..just look at this MAIN is there something needed to be done at Properties prop=new Properties(System.getProperties()); Please compile it at your side ....and look at error.. there are some errors which i cannot explain... please just once run it at your side...and put the inputs and see what is that i am missing.... Please help me...

        L 1 Reply Last reply
        0
        • S sangeeta2009

          public static void main (String argv[])
          {
          Frame f = new Frame("WebFrame");
          WebCrawler applet = new WebCrawler();
          f.add("Center", applet);

          /* Behind a firewall set your proxy and port here!
          */
          Properties props= new Properties(System.getProperties());
          props.put("http.proxySet", "true");
          props.put("http.proxyHost", "webcache-cup");
          props.put("http.proxyPort", "8080");

                      Properties newprops = new Properties(props);
                      System.setProperties(newprops);
          

          /**/

                      applet.init();
                      applet.start();
                      f.pack();
                      f.show();
              }
          

          }

          Ok ..just look at this MAIN is there something needed to be done at Properties prop=new Properties(System.getProperties()); Please compile it at your side ....and look at error.. there are some errors which i cannot explain... please just once run it at your side...and put the inputs and see what is that i am missing.... Please help me...

          L Offline
          L Offline
          Lost User
          wrote on last edited by
          #4

          sangeeta2009 wrote:

          Please compile it at your side ....and look at error..

          Well once I had commented out all the irrelevant code I could not see anything wrong; it compiles and runs. However, I don't see what this has to do with your original problem which as I recall, was some sort of ThreadException. My only suggestion is that you use whatever debug tools you have and try to step through your code. Assuming you wrote this code in the first place you should be able to extract small parts of it to test in isolation until you can at least narrow it down to a specific area of code.

          N S 2 Replies Last reply
          0
          • L Lost User

            sangeeta2009 wrote:

            Please compile it at your side ....and look at error..

            Well once I had commented out all the irrelevant code I could not see anything wrong; it compiles and runs. However, I don't see what this has to do with your original problem which as I recall, was some sort of ThreadException. My only suggestion is that you use whatever debug tools you have and try to step through your code. Assuming you wrote this code in the first place you should be able to extract small parts of it to test in isolation until you can at least narrow it down to a specific area of code.

            N Offline
            N Offline
            Nagy Vilmos
            wrote on last edited by
            #5

            Richard MacCutchan wrote:

            Assuming you wrote this code in the first place [...]

            #wistles#


            Panic, Chaos, Destruction. My work here is done.

            L 1 Reply Last reply
            0
            • N Nagy Vilmos

              Richard MacCutchan wrote:

              Assuming you wrote this code in the first place [...]

              #wistles#


              Panic, Chaos, Destruction. My work here is done.

              L Offline
              L Offline
              Lost User
              wrote on last edited by
              #6

              Someone must have written it! :rolleyes:

              1 Reply Last reply
              0
              • L Lost User

                sangeeta2009 wrote:

                Please compile it at your side ....and look at error..

                Well once I had commented out all the irrelevant code I could not see anything wrong; it compiles and runs. However, I don't see what this has to do with your original problem which as I recall, was some sort of ThreadException. My only suggestion is that you use whatever debug tools you have and try to step through your code. Assuming you wrote this code in the first place you should be able to extract small parts of it to test in isolation until you can at least narrow it down to a specific area of code.

                S Offline
                S Offline
                sangeeta2009
                wrote on last edited by
                #7

                Ok if Possible please help me out... I have run that code without any compilation error... But when i input the url int the first Text box.. say: http://java.sun.com in command prompt it give me errors..and doesn't display me any of the Url's related to that site.. Hope you get it...Please help and see if that works on your side... And please all other guys..stop commenting about the code....:mad::mad: I haven't written that code, i have taken it form Sun.com So if Somebody is interested in helping someone whose is in need...

                L 1 Reply Last reply
                0
                • S sangeeta2009

                  Ok if Possible please help me out... I have run that code without any compilation error... But when i input the url int the first Text box.. say: http://java.sun.com in command prompt it give me errors..and doesn't display me any of the Url's related to that site.. Hope you get it...Please help and see if that works on your side... And please all other guys..stop commenting about the code....:mad::mad: I haven't written that code, i have taken it form Sun.com So if Somebody is interested in helping someone whose is in need...

                  L Offline
                  L Offline
                  Lost User
                  wrote on last edited by
                  #8

                  sangeeta2009 wrote:

                  I haven't written that code, i have taken it form Sun.com

                  Well I suggest you go back to Sun and ask them to fix it. I don't have the skills to work on a piece of code this complex, and as you haven't written it yourself I doubt that you have much understanding of it either.

                  S 1 Reply Last reply
                  0
                  • L Lost User

                    sangeeta2009 wrote:

                    I haven't written that code, i have taken it form Sun.com

                    Well I suggest you go back to Sun and ask them to fix it. I don't have the skills to work on a piece of code this complex, and as you haven't written it yourself I doubt that you have much understanding of it either.

                    S Offline
                    S Offline
                    sangeeta2009
                    wrote on last edited by
                    #9

                    Ohhh..!!! That means you cannot help ..except fake posting... If i will be knowing that...it will be better that i will not be here..asking for help!!! well thanks for time wasting... :( :( :mad::mad: :( :(

                    L 1 Reply Last reply
                    0
                    • S sangeeta2009

                      Ohhh..!!! That means you cannot help ..except fake posting... If i will be knowing that...it will be better that i will not be here..asking for help!!! well thanks for time wasting... :( :( :mad::mad: :( :(

                      L Offline
                      L Offline
                      Lost User
                      wrote on last edited by
                      #10

                      sangeeta2009 wrote:

                      That means you cannot help ..except fake posting.

                      On the contrary, I have tried to help, but you seem incapable of doing anything for yourself, even answering a simple question.

                      sangeeta2009 wrote:

                      well thanks for time wasting

                      Yes, I am sorry that I have wasted my time trying to assist you. I thought you were a serious programmer who wanted to learn something. Obviously I was mistaken.

                      1 Reply Last reply
                      0
                      Reply
                      • Reply as topic
                      Log in to reply
                      • Oldest to Newest
                      • Newest to Oldest
                      • Most Votes


                      • Login

                      • Don't have an account? Register

                      • Login or register to search.
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • World
                      • Users
                      • Groups