Go to the system advanced and in start up and recovery set the machine to produce a memory dump, you can chose full or mini. Mini is OK ish, in which case its written to system\minidumps as a dmp file with the date/time in the name. If you chose full dump its written to \system\memory.dmp and overwrites any existing one. To look at the contents use windbg, and open the file. Then run !analyze -v from the command line (bottom of windbg window) and it will do an autoanalyse. Be careful, these aren't always accurate and often blame the wrong component. Now you need to debug the issue. Commands like !poolused can hep you find drivers that are using excessive memory, !thread and .thread are very useful to switch to any thread in the system so you can look at each threads stack and wait times. In fact what you can do with windbg is immense, its got a massive and powerful command set. There is a doc that comes with windbg on basic debugging practices, it is very worth reading.
