I agree with everything OG has said on this, but I would offer two further points to consider: USB 2.0 can operate at 60 MBS (that's bytes not bits). Nyquist theory for A2D conversion -basically- says that sampling anything above 2 X Freq_max is good, suggesting that for audio, sampling above 40 KHz would serve the audio spectrum. You have lots of slack in 60 MBS. The second point is that windows is time non-deterministic in events. That is, you do not know if USB events are 15 ms between, or 50 ms between, you have no capability of forcing this. And for the software to do the turn-around takes a bunch of time. To do time deterministic events, you would need a Real Time Operating System or RTOS like some versions of Linux or QNX or NI Labview RT. That said, I don't think your problem is in the USB 2.0 area, but in some other area that is causing the delays.
Ken