Let me explain the difference in simple terms:
Adverse: Used when describing something that works against something else.
The key is SOMETHING.
This word is never applied to people, but rather circumstances, conditions, reactions, effects, etc.
Example: The boy had an adverse reaction to the peaches. He started getting hives.
Explanation: The boy had a negative or hostile reaction to eating peaches. Hives, a reaction that causes red blotches on this skin and itches, was the negative reaction.
Averse: Used when describing someone's feeling or attitude against something.
The key is PERSON'S FEELING.
This word always applies to how people behave, and is always followed by the preposition 'to'.
Example: The Director was averse to approving the document.
Explanation: The Director did not want to approve the document.