I'm not sure what Error -10810 is (it isn't listed in MacErrors.h), but it's pretty clear what the problem is. You're trying to open a GUI application while the GUI belongs to a different user. (In your case, you're trying to run Messages.app as root. What exactly do you think that's going to do?)

If you run exactly the same command from a shell running as the GUI's user, it works fine. Run it from any other user, and it fails.