In order to remove a user from your list of frequent contacts, simply hold on the picture of the person for around a second and you will see a trash can appearing that looks like this:

Simply drop that picture of the specific contact in the trash can and it will be removed from your list. You can add that contact back again at anytime. Contacts don't get alerted when you remove them from your frequent contact list.

Did this answer your question?