webspider/crawler extract Javascript related info

[michael_jhons]

Hello everybody, I'm trying to develop a WebSpider that retrieves data related to "Sports" from twitter, Facebook and other sites/blogs, to display it all in my page. I just need to retrieve the information displayed on that page, but the problem I’m facing is that when I’m reading the twitter page, there is only a JavaScript code that display these data and not the data itself Is there any possibility to extract this information so I can log it in a Database for example? Thank you for your usual help

[Ennis Ray Lynch Jr]

Ajax, the bane of content thieves everywhere. It is actually easier to get the data from these types of sites you do not have to filter the HTML to get to the data. What you need to do is find the underlying js method that returns the data and call it directly. If the given site allows you to do so. The presence of an RSS feed is a de-facto indication that aggregation is expected. In fact, you could even get the data from the RSS feed as well.

Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. I also do Android Programming as I find it a refreshing break from the MS. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost

[RaviBee]

I handle situations like this by opening the page in an off-screen WebBrowser and reading the control's document's OuterHtml property after the document has finished loading.  Ennis' solution will work (and is the preferred one) if an underlying AJAX call exists. /ravi

My new year resolution: 2048 x 1536 Home | Articles | My .NET bits | Freeware ravib(at)ravib(dot)com

[Pete OHanlon]

Twitter has a well defined API for retrieving information in a variety of formats. Don't waste time trying to screen scrape when you can get the data easily using the RESTful API.

I have CDO, it's OCD with the letters in the right order; just as they ruddy well should be

My blog | My articles | MoXAML PowerToys | Onyx