As Trustee of a Trust can I be sued?

I am the Trustee of my late mothers Trust. My brother is a beneficiary of the Trust, He is a homeless drunk that refuses to get help. He keeps calling me and asking for money, I have 10,000 that he is entitled to, but I know if I give him that money he will check into a cheap hotel and go on a 6 month drunk until the money is gone. I am tired of the phone calls and just want to give him the money so he leaves me alone. Can he sue me later claiming I gave him the money knowing he is not of sound mind?