Am I in the wrong at all for this?

Ur not a bad person for that. He treated you both like crap so its his loss. Who cares what anyone else thinks? If u like eachother than go for it :)