request.GetResponse() - The remote server returned an error: (403) Forbidden

_Q12_

I used various code combinations for web page reading, but i have this error: [The remote server returned an error: (403) Forbidden] My questions: - I use VS2010. It's because my version of VS is so old, i keep getting this error? - Is the same problem for newer versions of VS? - If newer versions have no problem, what can i add to my version? - Or if it's the code problem, what is the correct code? This problem appear for SOME websites. But not on all. this is a sample of code i use:

    string urlAddress = "http://somesite.com";
    string ReadPage()
    {
        string page = "";
        HttpWebRequest request;
        HttpWebResponse response = null;
        Stream stream = null;
        request = (HttpWebRequest)WebRequest.Create(urlAddress);
        request.UserAgent = "Foo";
        request.Accept = "\*/\*";
        response = (HttpWebResponse)request.GetResponse();
        stream = response.GetResponseStream();

        StreamReader sr = new StreamReader(stream, System.Text.Encoding.Default);
        page = sr.ReadToEnd();
        if (stream != null) stream.Close();
        if (response != null) response.Close();
        //-------------------------------------------------------------
        return page;
    }

Thank you.

Dave Kreskowiak

It has nothing to do with Visual Studio and everything to do with your code, and you're assumptions about what you're doing and the site you're doing it with. Why you're setting the UserAgent string and the Accept is beyond me. I also have no idea what you're trying to do so it's impossible to tell you what's going on.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

_Q12_

I can't read html web pages because i get this error. Is now clear enough?

Lost User

Remote server returned a 403; so it does not allow you to do what it is you want.

Bastard Programmer from Hell :suss: If you can't read my code, try converting it here[^] "If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.

_Q12_

i want to read a page and on this line : request = (HttpWebRequest)WebRequest.Create(urlAddress); the compiler stops and gives me this error: [The remote server returned an error: (403) Forbidden] but i just remember that i have .NET 4 and " in .NET 4.0, TLS 1.2 is not supported, but if you have .NET 4.5 (or above) installed on the system then you still can opt in for TLS 1.2 even if your application framework doesn’t support it. " so i have to find a workaround. (this is a comment from an old code i had and i just find it now) - Is there a way to install a newer .NET in my VS2010 ? (the latest .NET that works for VS2019)

_Q12_

the problem is like this: Sometimes, the content of webpage is read!!! But after a couple of readings on the same website, it gets stuck on this error. It's like some cookie is activated or something, and remembers that i was there before and not alowing me to enter anymore. And it's true for any page from that website after this thing happen. The problem is i don't know how to find and set that cookie from c#. I strongly believe is a cookie.

Luc Pattyn

Hi, it probably means the site you're trying to access requires TLS1.2; there are at least two ways to get there: - use a sufficiently recent .NET version (and hence dito Visual Studio); - or use the following hack, which works for me even on .NET 2.0 and Visual Studio 9, presuming your Windows is sufficiently recent (say Win10):

using System.Net.Security;
using System.Security.Principal;
...
// the value for enum SecurityProtocolType.Tls12 is 3072, hence:
ServicePointManager.SecurityProtocol=(SecurityProtocolType)3072;
// now use your regular web code e.g.
using (WebClient wc=new WebClient()) {
// ...
}

:)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Dave Kreskowiak

No, it's not. There are a few ways to "read" a web page, like getting the directory of a website, which most site won't allow. Then there's downloading the page and parsing it. You give a "sample" site, but mention nothing about the site you're actually using so examining the details of the request and any response impossible. Then there's authentication to the site, any cookie management it requires, javascript crap, ... So, NO, it's not clear enough.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

Dave Kreskowiak

For .NET 4.5 and above, you need VS 2012 at a minimum. YOu can get the latest Community Edition (free) from here[^] .

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

Dave Kreskowiak

Your best source of information, and PERMISSION TO DO WHAT YOU WANT, is going to come from whoever owns the website you're trying to read. Hell, they may even have an API you can use to get the data from the site instead of scraping web pages.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

_Q12_

hi Luc, yes you are right. For awhile, some time ago, this little workaround that you are saying here worked fine. But after some 'internet' updates, like the [https] thing and more others that were very in shadow for me, but my code could sense them and i had to came back and guess, figure out, what he wants, until it worked again. And this thing you are specifying here was one of the problems i had to find and resolve. But it worked a limited time. It still works i suppose, but the cause of my recent error it seems to be other than this resolve here. I have windows7 This is what i have too, like you mentioned :

            //initialize WEBPAGE
            //Special wepage Reading  \[using System.Net;\]
            ServicePointManager.Expect100Continue = true;
            ServicePointManager.DefaultConnectionLimit = 9999;
            //in .NET 4.0, TLS 1.2 is not supported, but if you have .NET 4.5 (or above) installed on the system 
            //then you still can opt in for TLS 1.2 even if your application framework doesn’t support it. 
            //The only problem is that SecurityProtocolType in .NET 4.0 doesn’t have an entry for TLS1.2, 
            //so we’d have to use a numerical representation of this enum value:
            ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;
            //instead of: ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls;

Luc Pattyn

where I needed the hack, it still works today. So you must be facing another problem, requiring some other fix. :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

_Q12_

the problem is like this: Sometimes, the content of webpage is read !!!!!!!!!!!!!!!!!! But after a couple of readings on the same page/or/website, it gets stuck on this error. It's like some cookie is activated or something, and remembers that i was there before and not alowing me to enter anymore. And it's true for any page from that website after this thing happen. The problem is i don't know how to find and set that cookie from c#. I strongly believe is a cookie. But if it's something else? Here is why i'm asking.

_Q12_

I also suspect its a cookie problem, as you mention. The page is loading for awhile, like 5 times for example, but after the first occurrence of the error, it never loads anymore. I have to wait like more than 1 day/or a couple of hours, until i try again and the same story repeats. You mention about "examining the details of the request and any response" - give me a tutorial from which i can learn by myself how to do it, please. This web problem is not my strong point but i start to learn it through these errors i get, which sucks since it impedes my normal routine. ...Or summarize here yourself like a quick and dirty.

_Q12_

Dave Kreskowiak, my friend. Is not in my routine to do clever checks ! Especially for web problems. But you suggested me to do a check just now. And i did it. And it seems the code is ok !!! I tested it with another website and it loads fine...hopefully not getting the error after a couple of accessing like i described it already, though i repeated the request a couple of times and it seems ok so far. Yah, shame on me not figuring this out so far. But other than the code working, i still remain with the problem unsolved on the original link (that i cant mention). - How to (generally speaking) check what a web page wants and implement it in c# after that? (avoiding this error i keep receiving?)

Dave Kreskowiak

You don't do "clever checks"? Wow. What you call "clever", I call basic troubleshooting. And the only way to "figure out" what a site wants is to use a web browser to navigate around the site and "cleverly" use a tool like Fiddler (Google it) to see what's passed to the site and what comes back. Keep in mind, the site may also implement filters to prevent you from doing what you're doing, like X number of requests in a certain amount of time. There's nothing you can do to get around that.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

Dave Kreskowiak

_Q12_ wrote:

give me a tutorial from which i can learn by myself how to do it, please.

There isn't any tutorial for this. "This" just comes from a lot of experience and trial and error.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

Dave Kreskowiak

Well, if it's a cookie the site is looking for, you'd see that in a Fiddler trace.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak

_Q12_

i just downloaded Fiddler Web Debugger, and im looking in it. It's a very powerful tool from the first view, but i have no clue how to find my cookie with it. So far, i can see it is auto detecting my pages that are open, but unfortunatly, it open a bunch of other (background? maybe) pages. I look in them but is a bit overwhelming. I also find this neat thing where i can target the window. I also see in Inspectors - Cookies button, but it says "This request did not send any cookie data", and i just log out and log in from the page. Hmmm. Tell me please how to use it in this little thing i want. And thank you so much for such a awesome tool. I love it, even i am very new to it and im looking like a cat at the OZN. :)

Dave Kreskowiak

_Q12_ wrote:

Tell me please how to use it in this little thing i want

You want me to teach you everything you're going to need to know in a couple of forum posts?? :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: I don't have the time to write an entire book.

Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak