My husband avoids touching me during fasting hours in Ramadan which makes me feel bad. Is he right?

This is permissible in the shari'ah to avoid any temptation which may vitiate fasting. Your husband may avoid physical contact with you fearing to commit anything which may vitiate his fast.
May Allah grant you ever love and tranquility.