To properly escape a string value that you put in a javascript string literal, you replace \ with \\ and ' with \'. Before that you can replace a line break with the escape code \n. And don't use the language attribute in the script tag. It's been deprecated for many years.
this.ClientScript.RegisterStartupScript(this.GetType(), "Message", "alert('" + ex.Message.Replace("\r\n", @"\n").Replace(@"\", @"\\").Replace("'", @"\'") + "');");
Despite everything, the person most likely to be fooling you next is yourself.