How to make simple GROUP BY use index?

I want to get average temperatures hourly for given table with temperature reads of a thermometer, with row structure: thermometer_id, timestamp (float, julian days), value (float) plus ascending index on timestamp.

To get whole day 4 days ago, I'm using this query:

SELECT 
    ROUND(AVG(value), 2), -- average temperature
    COUNT(*)              -- count of readings
FROM reads
WHERE 
    timestamp >= (julianday(date('now')) - 5) -- between 5 days
    AND 
    timestamp < (julianday(date('now')) - 4)  -- ...and 4 days ago
GROUP BY CAST(timestamp * 24  as int)         -- make hours from floats, group by hours

It does it work well, yet it works very slowly, for a 9MB database, 355k rows, it takes more than half a second to finish, which is confusingly long, it shouldn't take more than few tens of ms. It does so on not quite fast hardware (not ssd though), yet I'm preparing it to use on raspberry pi, quite slow in comparison + it's going to get 80k more rows per day of work.

Explain explains the reason:

"USE TEMP B-TREE FOR GROUP BY"

I've tried adding day and hour columns with indexes just for the sake of quick access, but still, group by didn't use any of the indexes.

How can I tune this query or database to make this query faster?