Returning top 3 count values in sql queries

I have grouped my table based on a column field, and simultaneously displayed count value for each group. For example

user   |    count(user)
user 1 |     4
user 2 |     3
user 3 |     3
user 4 |     3
user 5 |     2

Now I need to display only top 3 users count in this case user 1, user2, user 3, user 4.

I have written the grouping logic but when I try to order values of count by desc and then restrict the result to 3(Limit 3), I only get user 1,2, and 3. Which is not correct because user 4 also comes in top 3 count.