No problem, the problem is with me, I am trying to get us lined up correctly so that we are working on the same thing.
Very likely, Mathematica or any CAS will be able to help you. I will see if I can come up with a method that gets your answer for this example you posted. Hopefully then, this method will be able to do others too.
He should leave out the f(x). It has no meaning here.
It should be
2) 15. Let f(a,b) be defined as a^2+b^2-2, which of the following CANNOT be the value of f(a,b)?
Is it 2?
What kind of numbers are a and b? Integers? What other constraints are put on them?