We saw fairly eye to eye.. Here was my first go:

int closest = inputs.FirstOrDefault(num => Math.Abs((Math.Abs(num) - (int)inputs.Average())) == inputs.Min(diff => Math.Abs(Math.Abs(diff) - (int)(inputs.Average()))));

And with some visual optimization

int closest = inputs.FirstOrDefault(num => Math.Abs(num - inputs.Average()) == inputs.Min(diff => Math.Abs(diff - inputs.Average())));

[EDIT] And thinking a bit more about it

int closest = inputs.Select(input => new
{
input,
diff = Math.Abs(input - inputs.Average())
}).OrderBy(x => x.diff).FirstOrDefault().input;