Rails order by a field in parent belongs_to association

I have three models in my Rails app:

class User < ActiveRecord::Base
  has_many :numbers
  has_many :messages, through: :numbers
end

class Number < ActiveRecord::Base
  belongs_to :user
  has_many :messages
end

... Number migration file ...
  t.string :digits, index: true # Example: '14051234567' (no + sign)
...

class Message < ActiveRecord::Base
  belongs_to :number
end

In my controller:

sort_mode = # asc or desc
@messages = current_user.messages.order(???)

The thing is that I want to sort those messages by their numbers' digits. How to do that dynamically (depending on sort_mode)?