Web Site is not Accessible by its IP?

[AliAmjad]

But when the IP address of the Server is correct and sent along with the path where the page is located then why it is not displaying the correct page. How can i handle such a situation because I've to resolve each discovered URL and if the resolver failed to get the correct information then I've to issue the request with the unresolved address which not only consumes the bandwidth but also effects the performance of the crawler. Thanks for your helps man !!!

AliAmjad(MCP) First make it Run THEN make it Run Fast!

[Guffa]

AliAmjad wrote:

But when the IP address of the Server is correct and sent along with the path where the page is located then why it is not displaying the correct page.

Because there can be several web sites that respond to the same IP, and they differ only by the domain name.

Experience is the sum of all the mistakes you have done.

[AliAmjad]

Then what should be the correct approach to access the right thing any article here or anywhere that can help me to solve this issue ???

AliAmjad(MCP) First make it Run THEN make it Run Fast!

[Guffa]

What are you trying to do, really? Why are you "resolving" the URLs? What's wrong with the URL as it is?

Experience is the sum of all the mistakes you have done.

[AliAmjad]

As i told you earlier that I am working on a Distributed Web Crawler and a web crawler must use the Domain Name Service (DNS) to map the web server’s host name into an IP address because if i have discovered say 50 urls from a site e.g. http://www.example.com/ then every request issued to each URL will require a DNS lookup to resolve that domain name which is an overhead a performance reduction or leak etc. what you say !!!

AliAmjad(MCP) First make it Run THEN make it Run Fast!

[Guffa]

Well, there's not much that you can do. A domain name is not the same as an IP. Sometimes you can use the IP instead of the domain name, sometimes not, and only the server knows. DNS services are usually cached, so a repeated DNS lookup will usually only go outside the local computer the first time. Have you actually determined that the DNS lookups is even a peformance problem?

Experience is the sum of all the mistakes you have done.

[AliAmjad]

Yes indeed and I've also read many research paper on this technology and Altavista's Marcator also has this component and their research also shows a significant performance improvement by implementing this very component.

Guffa wrote:

DNS services are usually cached, so a repeated DNS lookup will usually only go outside the local computer the first time.

Can you please tell me or explain it a bit ! Thanks for your help man !!

AliAmjad(MCP) First make it Run THEN make it Run Fast!

[Guffa]

AliAmjad wrote:

Yes indeed and I've also read many research paper on this technology and Altavista's Marcator also has this component and their research also shows a significant performance improvement by implementing this very component.

Well, to get that to work you have to send the request differently. You can send the request directly to the IP, but it has to contain the "unresolved" URL.

AliAmjad wrote:

Can you please tell me or explain it a bit !

DNS in Windows XP[^] Configuring the DNS Resolver Cache in Windows XP[^] Adjust Windows XP DNS Cache Settings[^]

Experience is the sum of all the mistakes you have done.

[AliAmjad]

Thanks a million man for your help !!!!

AliAmjad(MCP) First make it Run THEN make it Run Fast!

[Vasudevan Deepak Kumar]

A single IP can also cater to multiple websites (domains), in the case of shared hosting. Isn't it?

Vasudevan Deepak Kumar Personal Homepage
Tech Gossips
Regional Weblog (in Tamil) :: Voicing for the Society
Yesterday is a canceled check. Tomorrow is a promissory note. Today is the ready cash. USE IT.