I think you're missing the Register tag at the top in the master page where you specify the assembly, in your case, that would be the AJAXToolkit assembly that is to be registered in the program. I'm also new to Sharepoint and if you're trying to implement Ajax Toolkit in SP2010, its not going to be so straightforward mate, that's all I can tell you right now. You also have to get an older release of the AJAX Toolkit coz the latest one is compatible with .net 4.0 while Sharepoint stands at 3.5! :) But good luck to u though!
Rocky My Blog