I have a database representing metadata of a security camera NVR. There's a 26-byte recording row for every 1-minute segment of video. My design limits are 8 cameras, 1 year (~4 million rows, half a million per camera). I've faked up some data to test performance. This query is slower than I expected:
select
recording.start_time_90k,
recording.duration_90k,
recording.video_samples,
recording.sample_file_bytes,
recording.video_sample_entry_id
from
recording
where
camera_id = ?
order by
recording.start_time_90k;
That's just scanning all data for a camera, using an index for filtering out other cameras and ordering. Index looks like this:
create index recording_camera_start on recording (camera_id, start_time_90k);
explain query plan looks as expected:
0|0|0|SEARCH TABLE recording USING INDEX recording_camera_start (camera_id=?)
The rows are quite small.
$ sqlite3_analyzer duplicated.db
...
*** Table RECORDING w/o any indices *******************************************
Percentage of total database...................... 66.3%
Number of entries................................. 4225560
Bytes of storage consumed......................... 143418368
Bytes of payload.................................. 109333605 76.2%
B-tree depth...................................... 4
Average payload per entry......................... 25.87
Average unused bytes per entry.................... 0.99
Average fanout.................................... 94.00
Non-sequential pages.............................. 1 0.0%
Maximum payload per entry......................... 26
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1488
Primary pages used................................ 138569
Overflow pages used............................... 0
Total pages used.................................. 140057
Unused bytes on index pages....................... 188317 12.4%
Unused bytes on primary pages..................... 3987216 2.8%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 4175533 2.9%
*** Index RECORDING_CAMERA_START of table RECORDING ***************************
Percentage of total database...................... 33.7%
Number of entries................................. 4155718
Bytes of storage consumed......................... 73003008
Bytes of payload.................................. 58596767 80.3%
B-tree depth...................................... 4
Average payload per entry......................... 14.10
Average unused bytes per entry.................... 0.21
Average fanout.................................... 49.00
Non-sequential pages.............................. 1 0.001%
Maximum payload per entry......................... 14
Entries that use overflow......................... 0 0.0%
Index pages used.................................. 1449
Primary pages used................................ 69843
Overflow pages used............................... 0
Total pages used.................................. 71292
Unused bytes on index pages....................... 8463 0.57%
Unused bytes on primary pages..................... 865598 1.2%
Unused bytes on overflow pages.................... 0
Unused bytes on all pages......................... 874061 1.2%
...
I'd like something like this (maybe only a month at a time, rather than a full year) to be run every time a particular webpage is hit, so I want it to be quite fast. But on my laptop, it takes most of a second, and on the Raspberry Pi 2 I'd like to support, it's way too slow. Times (in seconds) below:
laptop$ ./bench-profiled
trial 0: time 0.534 sec
trial 1: time 0.558 sec
trial 2: time 0.571 sec
trial 3: time 0.561 sec
trial 4: time 0.597 sec
trial 5: time 0.552 sec
trial 6: time 0.607 sec
trial 7: time 0.641 sec
trial 8: time 0.603 sec
trial 9: time 0.659 sec
...
raspberrypi2$ ./bench-profiled
trial 0: time 6.536 sec
trial 1: time 6.246 sec
trial 2: time 6.235 sec
trial 3: time 6.090 sec
trial 4: time 6.077 sec
trial 5: time 6.249 sec
trial 6: time 6.239 sec
trial 7: time 6.230 sec
trial 8: time 6.075 sec
trial 9: time 6.077 sec
...
I'll likely end up doing some sort of denormalized data, but first I'd like to see if I can get this simple query to perform as well as it can. My benchmark's pretty simple; it prepares the statement in advance and then loops over this:
void Trial(sqlite3_stmt *stmt) {
int ret;
while ((ret = sqlite3_step(stmt)) == SQLITE_ROW) ;
if (ret != SQLITE_DONE) {
errx(1, "sqlite3_step: %d (%s)", ret, sqlite3_errstr(ret));
}
ret = sqlite3_reset(stmt);
if (ret != SQLITE_OK) {
errx(1, "sqlite3_reset: %d (%s)", ret, sqlite3_errstr(ret));
}
}
I made a CPU profile with gperftools. Image:
$ google-pprof bench-profiled timing.pprof
Using local file bench-profiled.
Using local file timing.pprof.
Welcome to pprof! For help, type 'help'.
(pprof) top 10
Total: 593 samples
154 26.0% 26.0% 377 63.6% sqlite3_randomness
134 22.6% 48.6% 557 93.9% sqlite3_reset
83 14.0% 62.6% 83 14.0% __read_nocancel
61 10.3% 72.8% 61 10.3% sqlite3_strnicmp
41 6.9% 79.8% 46 7.8% sqlite3_free_table
26 4.4% 84.1% 26 4.4% sqlite3_uri_parameter
25 4.2% 88.4% 25 4.2% llseek
13 2.2% 90.6% 121 20.4% sqlite3_db_config
12 2.0% 92.6% 12 2.0% __pthread_mutex_unlock_usercnt (inline)
10 1.7% 94.3% 10 1.7% __GI___pthread_mutex_lock
This looks strange enough to give me hope it can be improved. Maybe I'm doing something dumb. I'm particularly skeptical of the sqlite3_randomness and sqlite3_strnicmp operations:
- docs say
sqlite3_randomnessis used for inserting rowids in some circumstances, but I'm just doing a select query. Why would it be using it now? From skimming sqlite3 source code, I see it's used in select forsqlite3ColumnsFromExprListbut that seems to be something that'd happen when preparing the statement. I'm doing that once, not in the part being benchmarked. strnicmpis for case-insensitive string comparisons. But every field in this table is an integer. Why would it be using this function? What is it comparing?
Schema:
-- Each row represents a single recorded segment of video.
-- Segments are typically ~60 seconds; never more than 5 minutes.
-- Each row should have a matching recording_detail row.
create table recording (
id integer primary key,
camera_id integer references camera (id) not null,
sample_file_bytes integer not null check (sample_file_bytes > 0),
-- The starting time of the recording, in 90 kHz units since
-- 1970-01-01 00:00:00 UTC.
start_time_90k integer not null check (start_time_90k >= 0),
-- The duration of the recording, in 90 kHz units.
duration_90k integer not null
check (duration_90k >= 0 and duration_90k < 5*60*90000),
video_samples integer not null check (video_samples > 0),
video_sync_samples integer not null check (video_samples > 0),
video_sample_entry_id integer references video_sample_entry (id)
);
I've tarred up my test data + test program; you can download it here.
Aucun commentaire:
Enregistrer un commentaire