GWT-RPC call from c# .net (google keywords tool external) help

sodevrom

Hello guys, I have an app made that worked with the old Google Keywords Tool External. It worked perfectly until a few days ago when they dropped the new interface. Now I can't seem to crack this myself, and I need some help. Using opera dragonfly (helps debug http requests) I managed to see how everything looks "in the background" but I don't understand a thing now. With the old site,everything was crystal clear, but now it's not. The new website is located here:https://adwords.google.com/o/Targeting/Explorer?\_\_u=1000000000&\_\_c=1000000000&ideaRequestType=KEYWORD\_IDEAS#search.none When you want to get keyword results (enter a keyword and press enter), google makes a POST request here: https://adwords.google.com/o/Targeting/g?\_\_u=1000000000&\_\_c=1000000000 The request headers are:

Host adwords.google.com
User-Agent Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Content-Type text/x-gwt-rpc; charset=utf-8
X-GWT-Permutation BB8DF70B3A151B0F88DA836D4D8117AD
X-GWT-Module-Base https://adwords.google.com/o/Targeting/
Referer https://adwords.google.com/o/Targeting/Explorer?\_\_u=1000000000&\_\_c=1000000000&ideaRequestType=KEYWORD\_IDEAS
Content-Length 1401
Cookie PREF=ID=be4a951075eeeb11:U=58b39e637fa5af6e:TM=1278871506:LM=1278876883:S=-9tdR3vSj5IZPe5e; NID=36=j_lqfxFokemN-
,.,,,,,,,__utmc=229779660
Pragma no-cache
Cache-Control no-cache

Now I don't understand what X-GWT-Permutation is (anx all the other x-gwt). I know it's google web toolkit ... but how the hell can I make a "call"to this website from c# .net (using httprequest) ? The post content sent is:

6|1|69|https://adwords.google.com/o/Targeting/|807E86DA5783D73090609BC8C75E16F6|\_|invoke|3|147|18o|142|143|g|TiAction (Search.SearchInput.KEYWORD_IDEAS.RelatedToKeywordSearchParameter)|stQmQu4bHA07VgYuJlYqcKD9eRU:1287707728771|18i|14o|14a|14c|c|h|i|161|18g|ADVANCED_OPTIONS|CountryTargetSearchParameter|TiImpression|ParameterInQuery|LanguageTargetSearchParameter|DeviceTypeSearchParameter|yi|y8|178|s8|en_US|t4|zh|yr|16z|ys|IDEA_IN_ADGROUP|NEGATIVE_KEYWORDS|KEYWORD|COMPETITION|GLOBAL_MONTHLY_SEARCHES|AVERAGE_TARGETED_MONTHLY_SEARCHES|TARGETED_MONTHLY_SEARCHES|IDEA_TYPE|AD_SHARE|EXTRACTED_FROM_WEBPAGE|SEARCH_SHARE|KEYWORD_CATEGORY|NGRAM_GROUP|10b|175|b5|zw|18p|115|United

Luc Pattyn

Hi, I'm not familiar with the Google interfaces, however I managed to do some other POSTs a few weeks ago, and basically this page[^] was all I needed.

sodevrom wrote:

It's not a normal http post, because it has no variables, it's a simple text that it's sent.

It doesn't matter what the post data is, just write it to the stream, then proceed with accepting the result using HttpWebRequest.GetResponse() and HttpWebResponse.GetResponseStream(). So if it worked for you earlier, there should not be a major problem (unless you want to understand what it is you are posting). :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

sodevrom

Hello, I know how to use the httpwebrequest/response :) I am not really a "noob" in programming. Now I decided to take a totally different approach, and get the text that I need by using a WebBrowser control. This is the best way to go I think, because it takes care of all the problems. I have only one issue. For example to complete a field I use this: HtmlElement username = this.wb.Document.All["username"]; username.SetAttribute("value", "aaaaa"); But how do I get the entire text of the webbrowser? It seems that if I use webBrowser1.DocumentText , it does not retrieve all the HTML code (it does not return everything that is visible). I think the problem is with dynamic code (javascript) How do I get everything? Thanks, Vlad

jobs2001

Vlad, you should receive the KeywordsTool data (keywords, CPC etc) using such code: CComQIPtr<IHTMLElement, &IID_IHTMLElement> pBody; CComQIPtr<IHTMLDocument2, &IID_IHTMLDocument2> pDoc2; CComPtr<IDispatch> pDisp = NULL; CHECKPTR(pDisp = objExplorer.get_Document()); CHECKPTR(pDoc2 = pDisp); HRESULT hr; hr = pDoc2->get_body( &pBody); BSTR BSTR_Page = NULL; if (pBody){ hr = pBody->get_innerHTML( &BSTR_Page ); } / if (BSTR_Page){ sPage= (LPWSTR) BSTR_Page; SysFreeString(BSTR_Page); }

jobs2001

to set certain HTML elements in the page, you could iterate through the page element collection and check for the class name / id etc you need. Note after DISPID_DOCUMENTCOMPLETE the KeywordTool's JavaScript starts to amend the document. So you would need to poll the page content every X milliseconds to check if the elements you require have been created. VariantInit(&index); V_VT(&index) = VT_I2; V_I2(&index) = 0; CComPtr<IHTMLElementCollection> pEltColl = NULL; CComPtr<IDispatch> pEltDisp = NULL; CComQIPtr<IHTMLDocument2, &IID_IHTMLDocument2> pDoc2; CComPtr<IDispatch> pDisp = NULL; CHECKPTR(pDisp = objExplorer.get_Document()); CHECKPTR(pDoc2 = pDisp); HRESULT hResult,hr; hResult = pDoc2->get_all(&pEltColl); if (SUCCEEDED(hResult) && pEltColl) { // Search by value (manual search with a loop on all HTML Elements in document - could be long!) long intElements = 0; pEltColl->get_length(&intElements); for (int intElement = 0; intElement < intElements; intElement++) { CComPtr<IDispatch> pCurrentDispatch = NULL; VARIANT vElementIndex; VariantInit(&vElementIndex); V_VT(&vElementIndex) = VT_I4; V_I4(&vElementIndex) = intElement; hResult = pEltColl->item(vElementIndex, index, &pCurrentDispatch); if (SUCCEEDED(hResult)) { CComQIPtr<IHTMLElement, &IID_IHTMLElement> pCur_Element = pCurrentDispatch;