Trouble with Sqlite subquery

My CustomTags table may have a series of "temporary" records where Tag_ID is 0, and Tag_Number will have some five digit value.

Periodically, I want to clean up my Sqlite table to remove these temporary values.

For example, I might have:

Tag_ID      Tag_Number
0           12345
0           67890
0           45678
1           12345
2           67890

In this case, I want to remove the first two records because they are duplicated with actual Tag_ID 1 and 2. But I don't want to remove the third record yet because it hasn't been duplicated yet.

I have tried a number of different types of subqueries, but I just can't get it working. This is the last thing I tried, but my database client complains of an unknown syntax error. (I have tried with and without AS as an alias)

DELETE FROM CustomTags t1
WHERE t1.Tag_ID = 0
    AND (SELECT COUNT(*) FROM CustomTags t2 WHERE t1.Tag_Number = t2.Tag_Number) > 1

Can anyone offer some insight? Thank you