Can my daughter be affected by this and how can I not be affected as well?

I am not sure what advice you are looking for here. I can't imagine if your husband doesn't frighten you, why in the world do you not have custody? I can't imagine anything that would make me leave my child.