Well, he probably was, honestly. A traitor, I mean.

As far as being a prick -- well.

If he was a traitor, maybe he left to try to keep the rest of them safe. Neville's parents left him and Evelyn with their Gran, you know? Even if they are exactly as awful in real life as they were in the play last year, I'm sure they left Neville and Evelyn behind to keep them safe, not because they hate children.