SQL get rows which are not in

I got three tables in my sqlite database called member, group and group_member.

The member table looks like this:

member_id|first_name|...

The group table like this:

group_id|name|...

And the group_member table like this:

group_id|member_id

Now I want the user of my application to add/remove members from a group by having two lists of people besides each other. The first list should contain all people that are not in the group and the second list contains all members of the group.

not in group          in group
------------------------------
person1 |            | member5
person2 |    Add ->  | member6
person3 |  <- Remove |
person4 |            |

I know how I can get all members in the group, but how do I get the members that do not belong to the group to fill the first list?