request.GetResponse() - The remote server returned an error: (403) Forbidden
-
I used various code combinations for web page reading, but i have this error: [The remote server returned an error: (403) Forbidden] My questions: - I use VS2010. It's because my version of VS is so old, i keep getting this error? - Is the same problem for newer versions of VS? - If newer versions have no problem, what can i add to my version? - Or if it's the code problem, what is the correct code? This problem appear for SOME websites. But not on all. this is a sample of code i use:
string urlAddress = "http://somesite.com"; string ReadPage() { string page = ""; HttpWebRequest request; HttpWebResponse response = null; Stream stream = null; request = (HttpWebRequest)WebRequest.Create(urlAddress); request.UserAgent = "Foo"; request.Accept = "\*/\*"; response = (HttpWebResponse)request.GetResponse(); stream = response.GetResponseStream(); StreamReader sr = new StreamReader(stream, System.Text.Encoding.Default); page = sr.ReadToEnd(); if (stream != null) stream.Close(); if (response != null) response.Close(); //------------------------------------------------------------- return page; }Thank you.
Hi, it probably means the site you're trying to access requires TLS1.2; there are at least two ways to get there: - use a sufficiently recent .NET version (and hence dito Visual Studio); - or use the following hack, which works for me even on .NET 2.0 and Visual Studio 9, presuming your Windows is sufficiently recent (say Win10):
using System.Net.Security;
using System.Security.Principal;
...
// the value for enum SecurityProtocolType.Tls12 is 3072, hence:
ServicePointManager.SecurityProtocol=(SecurityProtocolType)3072;
// now use your regular web code e.g.
using (WebClient wc=new WebClient()) {
// ...
}:)
Luc Pattyn [My Articles] Nil Volentibus Arduum
-
No, it's not. There are a few ways to "read" a web page, like getting the directory of a website, which most site won't allow. Then there's downloading the page and parsing it. You give a "sample" site, but mention nothing about the site you're actually using so examining the details of the request and any response impossible. Then there's authentication to the site, any cookie management it requires, javascript crap, ... So, NO, it's not clear enough.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
i want to read a page and on this line : request = (HttpWebRequest)WebRequest.Create(urlAddress); the compiler stops and gives me this error: [The remote server returned an error: (403) Forbidden] but i just remember that i have .NET 4 and " in .NET 4.0, TLS 1.2 is not supported, but if you have .NET 4.5 (or above) installed on the system then you still can opt in for TLS 1.2 even if your application framework doesn’t support it. " so i have to find a workaround. (this is a comment from an old code i had and i just find it now) - Is there a way to install a newer .NET in my VS2010 ? (the latest .NET that works for VS2019)
For .NET 4.5 and above, you need VS 2012 at a minimum. YOu can get the latest Community Edition (free) from here[^] .
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
the problem is like this: Sometimes, the content of webpage is read!!! But after a couple of readings on the same website, it gets stuck on this error. It's like some cookie is activated or something, and remembers that i was there before and not alowing me to enter anymore. And it's true for any page from that website after this thing happen. The problem is i don't know how to find and set that cookie from c#. I strongly believe is a cookie.
Your best source of information, and PERMISSION TO DO WHAT YOU WANT, is going to come from whoever owns the website you're trying to read. Hell, they may even have an API you can use to get the data from the site instead of scraping web pages.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
Hi, it probably means the site you're trying to access requires TLS1.2; there are at least two ways to get there: - use a sufficiently recent .NET version (and hence dito Visual Studio); - or use the following hack, which works for me even on .NET 2.0 and Visual Studio 9, presuming your Windows is sufficiently recent (say Win10):
using System.Net.Security;
using System.Security.Principal;
...
// the value for enum SecurityProtocolType.Tls12 is 3072, hence:
ServicePointManager.SecurityProtocol=(SecurityProtocolType)3072;
// now use your regular web code e.g.
using (WebClient wc=new WebClient()) {
// ...
}:)
Luc Pattyn [My Articles] Nil Volentibus Arduum
hi Luc, yes you are right. For awhile, some time ago, this little workaround that you are saying here worked fine. But after some 'internet' updates, like the [https] thing and more others that were very in shadow for me, but my code could sense them and i had to came back and guess, figure out, what he wants, until it worked again. And this thing you are specifying here was one of the problems i had to find and resolve. But it worked a limited time. It still works i suppose, but the cause of my recent error it seems to be other than this resolve here. I have windows7 This is what i have too, like you mentioned :
//initialize WEBPAGE //Special wepage Reading \[using System.Net;\] ServicePointManager.Expect100Continue = true; ServicePointManager.DefaultConnectionLimit = 9999; //in .NET 4.0, TLS 1.2 is not supported, but if you have .NET 4.5 (or above) installed on the system //then you still can opt in for TLS 1.2 even if your application framework doesn’t support it. //The only problem is that SecurityProtocolType in .NET 4.0 doesn’t have an entry for TLS1.2, //so we’d have to use a numerical representation of this enum value: ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072; //instead of: ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls; -
hi Luc, yes you are right. For awhile, some time ago, this little workaround that you are saying here worked fine. But after some 'internet' updates, like the [https] thing and more others that were very in shadow for me, but my code could sense them and i had to came back and guess, figure out, what he wants, until it worked again. And this thing you are specifying here was one of the problems i had to find and resolve. But it worked a limited time. It still works i suppose, but the cause of my recent error it seems to be other than this resolve here. I have windows7 This is what i have too, like you mentioned :
//initialize WEBPAGE //Special wepage Reading \[using System.Net;\] ServicePointManager.Expect100Continue = true; ServicePointManager.DefaultConnectionLimit = 9999; //in .NET 4.0, TLS 1.2 is not supported, but if you have .NET 4.5 (or above) installed on the system //then you still can opt in for TLS 1.2 even if your application framework doesn’t support it. //The only problem is that SecurityProtocolType in .NET 4.0 doesn’t have an entry for TLS1.2, //so we’d have to use a numerical representation of this enum value: ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072; //instead of: ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls;where I needed the hack, it still works today. So you must be facing another problem, requiring some other fix. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
-
Hi, it probably means the site you're trying to access requires TLS1.2; there are at least two ways to get there: - use a sufficiently recent .NET version (and hence dito Visual Studio); - or use the following hack, which works for me even on .NET 2.0 and Visual Studio 9, presuming your Windows is sufficiently recent (say Win10):
using System.Net.Security;
using System.Security.Principal;
...
// the value for enum SecurityProtocolType.Tls12 is 3072, hence:
ServicePointManager.SecurityProtocol=(SecurityProtocolType)3072;
// now use your regular web code e.g.
using (WebClient wc=new WebClient()) {
// ...
}:)
Luc Pattyn [My Articles] Nil Volentibus Arduum
the problem is like this: Sometimes, the content of webpage is read !!!!!!!!!!!!!!!!!! But after a couple of readings on the same page/or/website, it gets stuck on this error. It's like some cookie is activated or something, and remembers that i was there before and not alowing me to enter anymore. And it's true for any page from that website after this thing happen. The problem is i don't know how to find and set that cookie from c#. I strongly believe is a cookie. But if it's something else? Here is why i'm asking.
-
No, it's not. There are a few ways to "read" a web page, like getting the directory of a website, which most site won't allow. Then there's downloading the page and parsing it. You give a "sample" site, but mention nothing about the site you're actually using so examining the details of the request and any response impossible. Then there's authentication to the site, any cookie management it requires, javascript crap, ... So, NO, it's not clear enough.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave KreskowiakI also suspect its a cookie problem, as you mention. The page is loading for awhile, like 5 times for example, but after the first occurrence of the error, it never loads anymore. I have to wait like more than 1 day/or a couple of hours, until i try again and the same story repeats. You mention about "examining the details of the request and any response" - give me a tutorial from which i can learn by myself how to do it, please. This web problem is not my strong point but i start to learn it through these errors i get, which sucks since it impedes my normal routine. ...Or summarize here yourself like a quick and dirty.
-
No, it's not. There are a few ways to "read" a web page, like getting the directory of a website, which most site won't allow. Then there's downloading the page and parsing it. You give a "sample" site, but mention nothing about the site you're actually using so examining the details of the request and any response impossible. Then there's authentication to the site, any cookie management it requires, javascript crap, ... So, NO, it's not clear enough.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave KreskowiakDave Kreskowiak, my friend. Is not in my routine to do clever checks ! Especially for web problems. But you suggested me to do a check just now. And i did it. And it seems the code is ok !!! I tested it with another website and it loads fine...hopefully not getting the error after a couple of accessing like i described it already, though i repeated the request a couple of times and it seems ok so far. Yah, shame on me not figuring this out so far. But other than the code working, i still remain with the problem unsolved on the original link (that i cant mention). - How to (generally speaking) check what a web page wants and implement it in c# after that? (avoiding this error i keep receiving?)
-
Dave Kreskowiak, my friend. Is not in my routine to do clever checks ! Especially for web problems. But you suggested me to do a check just now. And i did it. And it seems the code is ok !!! I tested it with another website and it loads fine...hopefully not getting the error after a couple of accessing like i described it already, though i repeated the request a couple of times and it seems ok so far. Yah, shame on me not figuring this out so far. But other than the code working, i still remain with the problem unsolved on the original link (that i cant mention). - How to (generally speaking) check what a web page wants and implement it in c# after that? (avoiding this error i keep receiving?)
You don't do "clever checks"? Wow. What you call "clever", I call basic troubleshooting. And the only way to "figure out" what a site wants is to use a web browser to navigate around the site and "cleverly" use a tool like Fiddler (Google it) to see what's passed to the site and what comes back. Keep in mind, the site may also implement filters to prevent you from doing what you're doing, like X number of requests in a certain amount of time. There's nothing you can do to get around that.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
I also suspect its a cookie problem, as you mention. The page is loading for awhile, like 5 times for example, but after the first occurrence of the error, it never loads anymore. I have to wait like more than 1 day/or a couple of hours, until i try again and the same story repeats. You mention about "examining the details of the request and any response" - give me a tutorial from which i can learn by myself how to do it, please. This web problem is not my strong point but i start to learn it through these errors i get, which sucks since it impedes my normal routine. ...Or summarize here yourself like a quick and dirty.
_Q12_ wrote:
give me a tutorial from which i can learn by myself how to do it, please.
There isn't any tutorial for this. "This" just comes from a lot of experience and trial and error.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
the problem is like this: Sometimes, the content of webpage is read !!!!!!!!!!!!!!!!!! But after a couple of readings on the same page/or/website, it gets stuck on this error. It's like some cookie is activated or something, and remembers that i was there before and not alowing me to enter anymore. And it's true for any page from that website after this thing happen. The problem is i don't know how to find and set that cookie from c#. I strongly believe is a cookie. But if it's something else? Here is why i'm asking.
Well, if it's a cookie the site is looking for, you'd see that in a Fiddler trace.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
Well, if it's a cookie the site is looking for, you'd see that in a Fiddler trace.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiaki just downloaded Fiddler Web Debugger, and im looking in it. It's a very powerful tool from the first view, but i have no clue how to find my cookie with it. So far, i can see it is auto detecting my pages that are open, but unfortunatly, it open a bunch of other (background? maybe) pages. I look in them but is a bit overwhelming. I also find this neat thing where i can target the window. I also see in Inspectors - Cookies button, but it says "This request did not send any cookie data", and i just log out and log in from the page. Hmmm. Tell me please how to use it in this little thing i want. And thank you so much for such a awesome tool. I love it, even i am very new to it and im looking like a cat at the OZN. :)
-
i just downloaded Fiddler Web Debugger, and im looking in it. It's a very powerful tool from the first view, but i have no clue how to find my cookie with it. So far, i can see it is auto detecting my pages that are open, but unfortunatly, it open a bunch of other (background? maybe) pages. I look in them but is a bit overwhelming. I also find this neat thing where i can target the window. I also see in Inspectors - Cookies button, but it says "This request did not send any cookie data", and i just log out and log in from the page. Hmmm. Tell me please how to use it in this little thing i want. And thank you so much for such a awesome tool. I love it, even i am very new to it and im looking like a cat at the OZN. :)
_Q12_ wrote:
Tell me please how to use it in this little thing i want
You want me to teach you everything you're going to need to know in a couple of forum posts?? :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: I don't have the time to write an entire book.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
_Q12_ wrote:
Tell me please how to use it in this little thing i want
You want me to teach you everything you're going to need to know in a couple of forum posts?? :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: I don't have the time to write an entire book.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiakso its complicated, i get it. Though i made some progress. I went on : View Cookie Information | Progress Telerik Fiddler[^] Capture HTTPS traffic from Firefox | Progress Telerik Fiddler[^] and i start made these settings and that involved installing the addon to be able to see the cookies. I installed it and i can see the cookies now ! I succesfully exported and installed a certificate that was required. And now is awesome. hmmm. Now what? You did an excelent job so far. Thank you very much. You can hint me, give me a general idea how to proceed next in this Fidler software. No need to give me baby steps, i get its too much for you. But general and hopefully to the point directives you can easily give (if you say you know this stuff). I want these directives to know in what direction to concentrate my search, because until now im doing it blind. Thank you again for the help so far. I hope i will not get stuck.
-
_Q12_ wrote:
Tell me please how to use it in this little thing i want
You want me to teach you everything you're going to need to know in a couple of forum posts?? :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: :laugh: I don't have the time to write an entire book.
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak{ My AWESOME friend, Dave Kreskowiak! } Here it is, i find my .. cookie! HOOOOOIIIII this is awesome. Sorry for the gray squares but as i promised, i will keep secret the website and my password on it :) heh. image: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/281c49f0-e874-4b83-bff6-9bc6b8526d42/dddeofe-237364f3-c452-4dac-aa60-d30061e27a8d.jpg/v1/fill/w_1485,h_538,q_70,strp/dbenzin83_ziub_6_copy_1_by_q12a_dddeofe-pre.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTA4NyIsInBhdGgiOiJcL2ZcLzI4MWM0OWYwLWU4NzQtNGI4My1iZmY2LTliYzZiODUyNmQ0MlwvZGRkZW9mZS0yMzczNjRmMy1jNDUyLTRkYWMtYWE2MC1kMzAwNjFlMjdhOGQuanBnIiwid2lkdGgiOiI8PTMwMDAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.ciBSaqH4cAj1SJBF6hisTxd9zmDJ4mYypGTCONhtMhY[^] But i must implore you to tell me how to inject it in c# WinApp code (not webapp)! (remember i have an older VS2010 too).So keep in mind these variables for me. - - -
-
{ My AWESOME friend, Dave Kreskowiak! } Here it is, i find my .. cookie! HOOOOOIIIII this is awesome. Sorry for the gray squares but as i promised, i will keep secret the website and my password on it :) heh. image: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/281c49f0-e874-4b83-bff6-9bc6b8526d42/dddeofe-237364f3-c452-4dac-aa60-d30061e27a8d.jpg/v1/fill/w_1485,h_538,q_70,strp/dbenzin83_ziub_6_copy_1_by_q12a_dddeofe-pre.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTA4NyIsInBhdGgiOiJcL2ZcLzI4MWM0OWYwLWU4NzQtNGI4My1iZmY2LTliYzZiODUyNmQ0MlwvZGRkZW9mZS0yMzczNjRmMy1jNDUyLTRkYWMtYWE2MC1kMzAwNjFlMjdhOGQuanBnIiwid2lkdGgiOiI8PTMwMDAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.ciBSaqH4cAj1SJBF6hisTxd9zmDJ4mYypGTCONhtMhY[^] But i must implore you to tell me how to inject it in c# WinApp code (not webapp)! (remember i have an older VS2010 too).So keep in mind these variables for me. - - -
Google: C# HttpWebRequest cookies[^]
Asking questions is a skill CodeProject Forum Guidelines Google: C# How to debug code Seriously, go read these articles.
Dave Kreskowiak -
I used various code combinations for web page reading, but i have this error: [The remote server returned an error: (403) Forbidden] My questions: - I use VS2010. It's because my version of VS is so old, i keep getting this error? - Is the same problem for newer versions of VS? - If newer versions have no problem, what can i add to my version? - Or if it's the code problem, what is the correct code? This problem appear for SOME websites. But not on all. this is a sample of code i use:
string urlAddress = "http://somesite.com"; string ReadPage() { string page = ""; HttpWebRequest request; HttpWebResponse response = null; Stream stream = null; request = (HttpWebRequest)WebRequest.Create(urlAddress); request.UserAgent = "Foo"; request.Accept = "\*/\*"; response = (HttpWebResponse)request.GetResponse(); stream = response.GetResponseStream(); StreamReader sr = new StreamReader(stream, System.Text.Encoding.Default); page = sr.ReadToEnd(); if (stream != null) stream.Close(); if (response != null) response.Close(); //------------------------------------------------------------- return page; }Thank you.
They blocked your IP because your "poking" is being interpreted as hacking. Or they are throttling you. You're assuming only static content. The door is being locked. Verboten. They have your IP address.
Quote:
You tell 'em I'm comin' and Hell is comin' with me, you hear?
The Master said, 'Am I indeed possessed of knowledge? I am not knowing. But if a mean person, who appears quite empty-like, ask anything of me, I set it forth from one end to the other, and exhaust it.' ― Confucian Analects
