Insert values into sqlite database in order

I have a high score table in SQLITE that supports three simple operations:

1. Adding a new user along with their high score.
2. Updating an existing users high score.
3. Getting the leaderboard (all users and scores in order of score):
   SELECT * FROM scores ORDER BY high_score DESC

Everything works, but I'm worried how this scales: with 10,000 users sorting the high scores takes ~60ms which is OK, but this time goes up roughly linearly such that if I had 100,000 users requesting the high scores would take ~600ms which is far too slow.

Is there a smart way to insert new users/update their scores in order to avoid having to do a complete sort whenever I retrieve the leaderboard? E.g. Something like a C++ priority_queue or a python heapq.

I suppose I could sort and replace the entire database on every insert (e.g. Sort an entire SQLite table) but that seems like overkill.