how to open an URL and save the content?
-
Is there any way to make a .net webpage, and in the server script to grab the URL and save the content to actual HTML file?
-
Is there any way to make a .net webpage, and in the server script to grab the URL and save the content to actual HTML file?
Simple answer: yes. In the code behind you will need to utilize a component (or write one) that makes the http request to the server running the .net webform and cache the response. The simplest mechanism would be to open a socket to port 80 (unless you're running https) of the server, send the GET or POST request and cache the response. You'll need to strip off the HTTP headers. eg. send:
GET /some/dotnet/file.aspx HTTP/1.1check the reponse contains:HTTP/1.1 200 OKin the first line. To strip off the HTTP headers in the response, look for the first empty line. The HTML content should be after that.
I'm largely language agnostic
After a while they all bug me :doh:
-
Is there any way to make a .net webpage, and in the server script to grab the URL and save the content to actual HTML file?
Here is a function that lets you do that:
Public Function GetWebPageResult(ByVal webPG As String) As String Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(webPG), HttpWebRequest) Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse) Dim receiveStream As Stream = myHttpWebResponse.GetResponseStream() Dim encode As System.Text.Encoding = System.Text.Encoding.GetEncoding("utf-8") Dim readStream As New StreamReader(receiveStream, encode) GetWebPageResult = readStream.ReadToEnd() ' Use this 'readStream' where ever you want. End FunctionJust call this function and pass the webpage URL to it. It will grab the html content of the page and return it as a string as the result of the function.Pete Soheil DigiOz Multimedia http://www.digioz.com
-
Here is a function that lets you do that:
Public Function GetWebPageResult(ByVal webPG As String) As String Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(webPG), HttpWebRequest) Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse) Dim receiveStream As Stream = myHttpWebResponse.GetResponseStream() Dim encode As System.Text.Encoding = System.Text.Encoding.GetEncoding("utf-8") Dim readStream As New StreamReader(receiveStream, encode) GetWebPageResult = readStream.ReadToEnd() ' Use this 'readStream' where ever you want. End FunctionJust call this function and pass the webpage URL to it. It will grab the html content of the page and return it as a string as the result of the function.Pete Soheil DigiOz Multimedia http://www.digioz.com
Hi, IMO you should Close both the WebResponse and the Stream inside your function. :)
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips: - make Visual display line numbers: Tools/Options/TextEditor/... - show exceptions with ToString() to see all information - before you ask a question here, search CodeProject, then Google
-
Here is a function that lets you do that:
Public Function GetWebPageResult(ByVal webPG As String) As String Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(webPG), HttpWebRequest) Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse) Dim receiveStream As Stream = myHttpWebResponse.GetResponseStream() Dim encode As System.Text.Encoding = System.Text.Encoding.GetEncoding("utf-8") Dim readStream As New StreamReader(receiveStream, encode) GetWebPageResult = readStream.ReadToEnd() ' Use this 'readStream' where ever you want. End FunctionJust call this function and pass the webpage URL to it. It will grab the html content of the page and return it as a string as the result of the function.Pete Soheil DigiOz Multimedia http://www.digioz.com
I'd forgotten about that class, cheers. Been working on winforms too long!
I'm largely language agnostic
After a while they all bug me :doh:
-
Hi, IMO you should Close both the WebResponse and the Stream inside your function. :)
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips: - make Visual display line numbers: Tools/Options/TextEditor/... - show exceptions with ToString() to see all information - before you ask a question here, search CodeProject, then Google
Luc Pattyn wrote:
IMO you should Close both the WebResponse and the Stream inside your function.
LOL... oops.... my bad. :) Yeah, make sure to close both those.
Pete Soheil DigiOz Multimedia http://www.digioz.com
-
Here is a function that lets you do that:
Public Function GetWebPageResult(ByVal webPG As String) As String Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create(webPG), HttpWebRequest) Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse) Dim receiveStream As Stream = myHttpWebResponse.GetResponseStream() Dim encode As System.Text.Encoding = System.Text.Encoding.GetEncoding("utf-8") Dim readStream As New StreamReader(receiveStream, encode) GetWebPageResult = readStream.ReadToEnd() ' Use this 'readStream' where ever you want. End FunctionJust call this function and pass the webpage URL to it. It will grab the html content of the page and return it as a string as the result of the function.Pete Soheil DigiOz Multimedia http://www.digioz.com
Hi Digi, thatnks for the reply. The function is awesome. it works like charm ^^ BTW, i just find out another solution, to share with you guys here =D I used webclient class, there is a readopen() method to achieve this. here is a sample class which i get from msdn. Public Class Test Public Shared Sub Main(args() As String) If args Is Nothing OrElse args.Length = 0 Then Throw New ApplicationException("Specify the URI of the resource to retrieve.") End If Dim client As New WebClient() ' Add a user agent header in case the ' requested URI contains a query. client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)") Dim data As Stream = client.OpenRead(args(0)) Dim reader As New StreamReader(data) Dim s As String = reader.ReadToEnd() Console.WriteLine(s) data.Close() reader.Close() End Sub 'Main End Class 'Test :-D