Anyone done any Web Scraping in C++/MFC

[Bryan Anslow]

Hi, I currently get data from a website like this; Go to the webpage, wait while the page loads, "select" the data in the webpage by dragging my mouse over it, Copy to clipboard, call up my text editor, Paste the data into the editor, save the output, analyze the data using my C++/MFC Analyzer. There must be a way I can get the data without the need to copy/paste etc. I know something about HTTP Connections, sockets, etc., but I am at a loss to know just what parameters need to be passed to the website in the request. I tried just sending the URL of the webpage, which returned a lot of stuff which I did not want. Does anyone have a simple example of how this is supposed to work? Thanks so much, Bryan.

[Lost User]

If you read the HTML output directly then there will be a load of stuff that you are perhaps not interested in. The answer is to find some classes that help you find specific parts quickly, and C++/MFC does not offer the facilities that are available in .NET. Most web scrapers are written in the latter framework using C# or VB.NET. As to what information to send to the website, that depends on the website itself as each one has its own structure.

[David Crow]

Bryan Anslow wrote:

Go to the webpage, wait while the page loads, "select" the data in the webpage by dragging my mouse over it, Copy to clipboard, call up my text editor, Paste the data into the editor, save the output, analyze the data using my C++/MFC Analyzer.

I think you can replace all but the last of these steps with URLDownloadToFile().

"One man's wage rise is another man's price increase." - Harold Wilson

"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons

"You can easily judge the character of a man by how he treats those who can do nothing for him." - James D. Miles

[Bryan Anslow]

Hey, that works well, thanks for pointing me at that function. My minor hurdle right now is that I need to login with my userid and password to access one of the pages. Don't suppose that there is some way of getting my credentials in there somehow? Thanks, Bryan.

[David Crow]

See if NetUseAdd() helps.

"One man's wage rise is another man's price increase." - Harold Wilson

"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons

"You can easily judge the character of a man by how he treats those who can do nothing for him." - James D. Miles