Allowing http redirections??
-
I am trying to crawl using the following code snippet. HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url); req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)"; ... By default, the req.AllowAutoRedirect is true and MaximumAutomaticRedirections is 50. When I try to crawl the following URL. http://citeseer.ist.psu.edu/rd/55811103,653,1,0.25,Download/http://citeseer.ist.psu.edu/cache/papers/cs/7145/http:zSzzSzwww.stanford.eduzSzclasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf I get NameResolutionFailure exception. However, I am able to open this URL from the browser and it gets redirected to: http://citeseer.ist.psu.edu/cache/papers/cs/7145/http:zSzzSzwww.stanford.eduzSzclasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf How do I force my C# code to go to the redirected url?