This is such a rough situation, and I think you made the right choice and have been continuing to make it. You do not deserve to e treated that way--no one does--and removing yourself from the opportunity may be the easiest way to be firm and avoid pushback.

As worrisome as it may be having your mother in Florida, she at least is in the position she earned for herself-- and so is your brother.