Will I most likely permenently have sole custody of my son?

I am a 24 yr old single mother with a 20 month old special needs son. I recently regained full physical & legel custody of him last month finally after 16 months involved in a CHIPS case. The reason being involved in the chips case was due to my son spending the first 5 months in the hospital for having serious medical needs and 4 different surgerys. Social services filed a petion with concerns of me being a single mother not being able to handle his special needs by myself and they believe I wasent spending enough time at the hospital with him which angered me. He was then placed in a temporary medical foster home. Now after 16 months its all over and my son is doing excellent and he is with me most of the time. His father wants to now take me to family court to fight for joint custody. He is 29, lives with his mom, is vindictive, condesending, like to power trip, refuses to pay child support and doesnt communicate with me that well. His visitation with our son is very reasonable for being the non custodial parent. His visitation schedule is every wednesday night and every other wknd fri sat and sun. He believes that if he takes me to family court since there was a previous chips case that this all could go to trial and i could lose my son which i believe is not possible since I am a good mother and already have sole custody. Does he really have a chance or should I just not worry?