A person who denies the obligation of prayer is certainly a non-believer, or 'kafir'. We have no hesitation in describing him as such because he denies an essential point of our religion which is commonly known to all. A person who omits to offer one prayer every now and then cannot be described as a non-believer or a 'kafir', because he acknowledges the duty of prayer and he practices it. Its omission is certainly a sin of which he should repent properly. Such a repentance cannot be described as proper unless it is accompanied by stopping the omission completely. Your friend is correct in being careful with regard to who can be described as non-believer. He is not right, however, in saying that we cannot call a non-Muslim a 'kafir', because the term 'kafir' means a non-Muslim.
(Source: Arab News) Classified in Kafir > Describing one as 'Kafir'