My headcanon:
Madame knows Billy since childhood. Billy helped her and promised to always be with her. But unfortunately he betrayed Madame or fell in love with some other girl.
Madame never forgive that.