When you leave prayer and Holy Mass to serve those in need, you are losing nothing, because serving those in need is going to God and you should see God in them.