I noticed everytime after I clear the cache(delete ./public_html/var/chche folder), my site's loading is 200-300ms faster, which looks a good thing but I suspect there's some issue behind it, I don't think the cache file needs to be deleted frequently. V3.03, using 'file' for backend cache. Any idea?
You are faster disabling the cache? That would imply there are some underlying issues on your server or you are using shared hosting. Maybe you're just looking at one page that uses only a few templates? Would guess that going to checkout will be very slow after clearing the cache but reasonably fast without. 2-300ms on a page load is nothing to consider. Barely human perceptible. I.e. it's the blink of an eye.
would also help if you'd give full details. I.e. what page, what tool(s) you're using for measurement, etc…