For one thing it looks like it can be done by inspection.
Big O Notation defines the behavior of a function as it approaches some value. Often, we say infinity, but generally "an arbitrarily large number" is sufficient.
It is clear that as n gets large the n^2 term is going to drown out the other terms. So we can say by inspection that O(f(n)) == O(n^2)
Or maybe you could try to show a bit more rigorously.
But this looks like overkill to me.
You might pick up a few hints on how engineers view this problem from here:
One of them uses a method similar to your professor.