Can they take the car away from me?

I took over the car payments of a car from my brother and his wife. we signed a notarized letter saying that the car was mine and i was giving the payments. i also have all the receipts to show the payments i made. Now that I made my brother angry at me they are trying to take the car away from me and are looking for me to serve me papers. This is the onlu vehicle i have and i can't just give up on all the money i paid. can they just take it away or would they have to sue me?