I gave it a shot and came up with an answer using a few less calls:

int avg = inputs.OrderBy(i => Math.Abs(i - inputs.Average())).First();