Chew on this for a minute...
What if your horse could speak? What do think he would say? Would it be something like "I luv you mommy!"? OR would it be like "Get away from me. I don't like you."? Watch your horse's body language while you're around him. When you go to catch him from the pasture, does he gallop off to the other end of the pasture to avoid being caught? When you enter his stall, he turn away from you and point his haunches at you? (I know my ponies catch me before I catch them - they come willing to me)
That could be a sign. Read the signs before something goes wrong.
Take the time it takes so it takes less time.