Bumping an old post - https web request [modified]
-
I'm still trying to get a block of code that does an https file download working. The vendor (swift.com) says that they took my code and got it working on their end no problem which leads me to believe that the something that is preventing me lives on my side of the request. We have a download scanner that pops up when I load the URL in my browser, so it's possible that I'm getting blocked there but I'm still working with the security team on that one. I'd appreciate it if someone could take a look at my code below and let me know if you see anything that's wrong/missing/poorly formed for an https request. This is being run from a .NET 2.0 console application if that makes any difference. I highlighted the line in the code where the 401 Unauthorized error occurs. I added a few hard returns to the code to keep the page from going WAY WIDE. If you pull the code down from here and past it into VS you might have to remove them.
private bool DownloadFile()
{
private const string URL = "https://www2.swift.com/bicdownload/bicdownloader?
action=getfile&productline=bicdir&product=bicdb&content=full&format=txt&platform=win";
try
{
string bicFileTemp = string.Format("{0}\\BIC-{1}.txt",
ConfigurationManager.AppSettings["BICDLPath"], DateTime.Now.ToString("dd-MMM-yyyy"));
string line;// build credentials for SWIFT authentication System.Net.NetworkCredential cred = new System.Net.NetworkCredential(); cred.UserName = "myUserName"; cred.Password = "myPassword"; WebRequest myReq = WebRequest.Create(URL); myReq.Credentials = cred; // build credentials for WTC Proxy authentication myReq.Proxy = WebProxy.GetDefaultProxy(); myReq.Proxy.Credentials = CredentialCache.DefaultCredentials; // send request and get response* ERROR HERE * WebResponse wr = myReq.GetResponse();
Stream receiveStream = wr.GetResponseStream();
StreamReader sr = new StreamReader(receiveStream, Encoding.UTF8);
StringBuilder bicFile = new StringBuilder();// parse response into file, write file out to \\\\server\\BICDownloads do { line = sr.ReadLine(); bicFile.Append(line); } while (line != null); -
I'm still trying to get a block of code that does an https file download working. The vendor (swift.com) says that they took my code and got it working on their end no problem which leads me to believe that the something that is preventing me lives on my side of the request. We have a download scanner that pops up when I load the URL in my browser, so it's possible that I'm getting blocked there but I'm still working with the security team on that one. I'd appreciate it if someone could take a look at my code below and let me know if you see anything that's wrong/missing/poorly formed for an https request. This is being run from a .NET 2.0 console application if that makes any difference. I highlighted the line in the code where the 401 Unauthorized error occurs. I added a few hard returns to the code to keep the page from going WAY WIDE. If you pull the code down from here and past it into VS you might have to remove them.
private bool DownloadFile()
{
private const string URL = "https://www2.swift.com/bicdownload/bicdownloader?
action=getfile&productline=bicdir&product=bicdb&content=full&format=txt&platform=win";
try
{
string bicFileTemp = string.Format("{0}\\BIC-{1}.txt",
ConfigurationManager.AppSettings["BICDLPath"], DateTime.Now.ToString("dd-MMM-yyyy"));
string line;// build credentials for SWIFT authentication System.Net.NetworkCredential cred = new System.Net.NetworkCredential(); cred.UserName = "myUserName"; cred.Password = "myPassword"; WebRequest myReq = WebRequest.Create(URL); myReq.Credentials = cred; // build credentials for WTC Proxy authentication myReq.Proxy = WebProxy.GetDefaultProxy(); myReq.Proxy.Credentials = CredentialCache.DefaultCredentials; // send request and get response* ERROR HERE * WebResponse wr = myReq.GetResponse();
Stream receiveStream = wr.GetResponseStream();
StreamReader sr = new StreamReader(receiveStream, Encoding.UTF8);
StringBuilder bicFile = new StringBuilder();// parse response into file, write file out to \\\\server\\BICDownloads do { line = sr.ReadLine(); bicFile.Append(line); } while (line != null);The only dodgy bit I can see is the proxy. Can you try connecting directly? If your proxy is not forwarding your credentials correctly, you will get a 401. I forget how, but there is a way to add your username and password to the URI. It's something like https://username:password@www.... Might be worth a try. Nick
---------------------------------- Be excellent to each other :)
-
The only dodgy bit I can see is the proxy. Can you try connecting directly? If your proxy is not forwarding your credentials correctly, you will get a 401. I forget how, but there is a way to add your username and password to the URI. It's something like https://username:password@www.... Might be worth a try. Nick
---------------------------------- Be excellent to each other :)
Thanks for the reply Nick. Tried removing the Proxy but I don't get off the network if I do. Maybe embedding the credentials in the URI will work. I'll give that a go and let you know how I make out.
Mike Devenney
-
Thanks for the reply Nick. Tried removing the Proxy but I don't get off the network if I do. Maybe embedding the credentials in the URI will work. I'll give that a go and let you know how I make out.
Mike Devenney
You'll have to find a computer that isn't behind your proxy. Do you run a DMZ? Or a web server? You can always try from someone's home. I just googled the URI encoding and I got it right :-D http://en.wikipedia.org/wiki/URI_scheme[^] Please let us know how you get on. Nick
---------------------------------- Be excellent to each other :)
-
You'll have to find a computer that isn't behind your proxy. Do you run a DMZ? Or a web server? You can always try from someone's home. I just googled the URI encoding and I got it right :-D http://en.wikipedia.org/wiki/URI_scheme[^] Please let us know how you get on. Nick
---------------------------------- Be excellent to each other :)
And the games continue. My username for the SWIFT site is my password, which includes the @ sign. I got an error back saying that
a port # is expected because a colon was found in the request.
I assume this is happening because of the @ in my username. Would I escape the @ character?
Mike Devenney
-
And the games continue. My username for the SWIFT site is my password, which includes the @ sign. I got an error back saying that
a port # is expected because a colon was found in the request.
I assume this is happening because of the @ in my username. Would I escape the @ character?
Mike Devenney
Not sure about that. Have you tried putting the uri in a web browser? Nick
---------------------------------- Be excellent to each other :)
-
Not sure about that. Have you tried putting the uri in a web browser? Nick
---------------------------------- Be excellent to each other :)
Before I was getting an error about having a colon in the request with no port specified. Now, I get this error from IE in a dialog box entitled Address Bar. Windows cannot find 'https://mdevenney@wilmingtontrust.com:Password@www2.swift.com/bicdownload/bicdownloader?action=getfile&productline=bicdir&product=bicdb&content=full&format=txt&platform=win'. Check spelling and try again. :mad:
Mike Devenney
-
Before I was getting an error about having a colon in the request with no port specified. Now, I get this error from IE in a dialog box entitled Address Bar. Windows cannot find 'https://mdevenney@wilmingtontrust.com:Password@www2.swift.com/bicdownload/bicdownloader?action=getfile&productline=bicdir&product=bicdb&content=full&format=txt&platform=win'. Check spelling and try again. :mad:
Mike Devenney
I don't know how to escape an @ in your username. I just tried putting the uri without username:password into ie8 and a dialog popped up asking for credentials. Could you try this, enter your credentials and see if you get the file? If you do, it means your account and local network are working, which would narrow the problem down to the credentials your proxy is passing when you use WebRequest. Nick
---------------------------------- Be excellent to each other :)
-
I don't know how to escape an @ in your username. I just tried putting the uri without username:password into ie8 and a dialog popped up asking for credentials. Could you try this, enter your credentials and see if you get the file? If you do, it means your account and local network are working, which would narrow the problem down to the credentials your proxy is passing when you use WebRequest. Nick
---------------------------------- Be excellent to each other :)
I'm able to get the file using the URI and entering the credentials into the dialog that pops up. Is there any way to see what the proxy is passing?
Mike Devenney
-
I'm able to get the file using the URI and entering the credentials into the dialog that pops up. Is there any way to see what the proxy is passing?
Mike Devenney
You could try setting
myReq.PreAuthenticate = true;Might work... Nick---------------------------------- Be excellent to each other :)
-
You could try setting
myReq.PreAuthenticate = true;Might work... Nick---------------------------------- Be excellent to each other :)
Sounds like your idea of the proxy server interfering is a winner. I'm talking over my head here but I'll do my best to explain what I just learned. I spoke with two of our architects who said that any https requests have to spoof a validated certificate because our infosec dept strips out the remote host's certificate and inserts one of their own. I have some sample code that one of them uses for a web service that makes https requests. Because it's a web service the code is slightly different but it's the same idea...
ServicePointManager.ServerCertificateValidationCallback
+= new RemoteCertificateValidationCallback(ValidateRemoteCertificate)private static bool ValidateRemoteCertificate
(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors policyErrors)
{
return true;
}By "faking" the true value for the ValidateRemoteCertificate method the authentication succeeds and the connection will open successfully. I'm off to work this into my routine. If/when I get it working I'll post the "correct" code. Thanks for helping me work through this! :thumbsup:
Mike Devenney
-
Sounds like your idea of the proxy server interfering is a winner. I'm talking over my head here but I'll do my best to explain what I just learned. I spoke with two of our architects who said that any https requests have to spoof a validated certificate because our infosec dept strips out the remote host's certificate and inserts one of their own. I have some sample code that one of them uses for a web service that makes https requests. Because it's a web service the code is slightly different but it's the same idea...
ServicePointManager.ServerCertificateValidationCallback
+= new RemoteCertificateValidationCallback(ValidateRemoteCertificate)private static bool ValidateRemoteCertificate
(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors policyErrors)
{
return true;
}By "faking" the true value for the ValidateRemoteCertificate method the authentication succeeds and the connection will open successfully. I'm off to work this into my routine. If/when I get it working I'll post the "correct" code. Thanks for helping me work through this! :thumbsup:
Mike Devenney
GAH :confused: Thought that the certificate was going to be the silver bullet but I still get my now least favorite response: The server returned an error: (401) Unauthorized. I'm headed back to the drawing board. Not defeated yet, but getting there. X|
Mike Devenney