In my experience you’ll do much better with sqlite using lots of filters/features. However it does NOT seem to “track” changes to settings so you need to become disciplined in clearing the cache if you make changes to addon settings or other settings from the admin panel. It does NOT seem to affect product or category changes, but it misses many others (this is the PDO version of sqlite for 2.0.x). But for runtime, performance is much improved due to the indexing of the sqlite db versus using filenames in directories.
Before you clear your cache someitme, I’d be very interested to know the result of:
$ find var/cache -type f -print | wc -l
This will tell the number of files that are in your cache directory tree.
Since you’re on a dedicated server, you’re not competing with other sites in the file cache (OS level) so you probably don’t see the same issues as those on VPS or shared hosting environments.