Android- How to selectively query the Contacts Content Provider based on my own SQLite tables?

I have a Friends SQLite table and I am trying to create something similar to snapchat where you can add friends based on your contacts, so when the app pulls your contacts using the Content Provider it should only retrieve contacts who are not only already your friends. Ideally, I'd perform a JOIN with the Friends table and the Contacts database but as the discussion following this question explains it is not possible to perform a JOIN between tables in the Contacts database and my own SQLite tables.

Right now, I query all contacts and then iterate through each one querying my Friends table to see check that they are not already a friend and only then do I add them to the RecyclerView to be displayed. Is there a better way to create this behavior? Because this seems like a very inefficient way of doing things