crawling a data of a website
-
Hi, I was reading how to create a program that crawls website data. one of the example I found was http://www.codeproject.com/KB/IP/Crawler.aspx but I dont have much resource to crawl everything. In a ideal e-shops you have product page and product details page. what I would like to do is crawl each product page and product details page and only write data that interests me to my database. e.g. product name, product price, etc etc. could anyone point me to right direction? Thank you
-
Hi, I was reading how to create a program that crawls website data. one of the example I found was http://www.codeproject.com/KB/IP/Crawler.aspx but I dont have much resource to crawl everything. In a ideal e-shops you have product page and product details page. what I would like to do is crawl each product page and product details page and only write data that interests me to my database. e.g. product name, product price, etc etc. could anyone point me to right direction? Thank you
So you have used someones article to crawl some web sites, this is good. Now you need to customise the article to meet your needs, so inspect the data as it is crawled and decide what to do with it. Something along the lines of if it is a product then turn a store flag on and write the data to the database, identify the token that denotes the end of the product information and turn off the store flag.
Never underestimate the power of human stupidity RAH