How do you knwo what cache type to use?

Not sure about others method, I’ve just switched to sqlite.

Before sqlite, I was using the default “files” method.



For the files method, upon some monitoring, I’ve realized that in 1 day time, if I do not clear the cache via the &cc method, the number of directories in the var/cache folders can be >2000 [my ftp truncate at 2000, so exact number I’m not sure]. Of which majority are pfilter_xx directoris with a single cache.csc file inside. Not sure how these kind of directories creation affect the file system on the server itself. But decided to try sqlite for now.

[quote name=‘nodame’]Not sure about others method, I’ve just switched to sqlite.

Before sqlite, I was using the default “files” method.



For the files method, upon some monitoring, I’ve realized that in 1 day time, if I do not clear the cache via the &cc method, the number of directories in the var/cache folders can be >2000 [my ftp truncate at 2000, so exact number I’m not sure]. Of which majority are pfilter_xx directoris with a single cache.csc file inside. Not sure how these kind of directories creation affect the file system on the server itself. But decided to try sqlite for now.[/quote]



Please update us on your speed improvements

Not sure about speed improvements as I am more concern in getting zero error.log files at this moment than speed from cscart.



So since I’ve selected sqlite as cache method for a few hours since the afternoon, my store running cscart script have generated an error.log with these lines within:



“[29-Oct-2010 18:23:29] PHP Warning: SQLiteDatabase::query() [sqlitedatabase.query]: too much data for one table row in /home/nodame/public_html/nodame/core/cache/class.cache_backend_sqlite.php on line 89”



The sqlite cache.db file is about 19mb now. And there is a cache.db-journal file in the cache directory too.



I would like to know if anyone running the sqlite cache version is having the same problems?

I had to switch back to “file” method when the sqlite cache method totally disabled my store with error msges after about 10 hours of running it without doing a cache refresh.

Upon doing a cache refresh at the admin area, the store went back to normal with the error msges gone.

But I couldn’t be monitoring the store whole day long to do cache refresh, and the main purpose of me switching to other cache method to try is to see if I can avoid having to do periodic cache refresh as with the 'file" method.

Since it fail in this purpose and failed more drastically than “file” method (it completely disable the store by displayed Fatal php error msges when going to the store URL, both backend and frontend), I’ve since reverted back to the “file” method for now.

I assume mysql method will be same if not more drastic (since mysql will directly affect server load)

May I know if anyone has any feedback on the “shmem” method?

[quote name=‘nodame’]I had to switch back to “file” method when the sqlite cache method totally disabled my store with error msges after about 10 hours of running it without doing a cache refresh.

Upon doing a cache refresh at the admin area, the store went back to normal with the error msges gone.

But I couldn’t be monitoring the store whole day long to do cache refresh, and the main purpose of me switching to other cache method to try is to see if I can avoid having to do periodic cache refresh as with the 'file" method.

Since it fail in this purpose and failed more drastically than “file” method (it completely disable the store by displayed Fatal php error msges when going to the store URL, both backend and frontend), I’ve since reverted back to the “file” method for now.

I assume mysql method will be same if not more drastic (since mysql will directly affect server load)

May I know if anyone has any feedback on the “shmem” method?[/QUOTE]



Something must be wrong with your server setup, I have been running the Sqlite caching method on my site along with smartoptimizer for a month now without having to clear the cache and my db file is less than 1MB. I found the Sqlite method the best to use not so much for speed because I did not see any noticeable difference but not having to clear the cache so often like with file cache so I agree with Struck on this one.

[quote name=‘gasngrills’]Something must be wrong with your server setup, I have been running the Sqlite caching method on my site along with smartoptimizer for a month now without having to clear the cache and my db file is less than 1MB. I found the Sqlite method the best to use not so much for speed because I did not see any noticeable difference but not having to clear the cache so often like with file cache so I agree with Struck on this one.[/QUOTE]



I do not run smartoptimizer, so I cannot compare my site with yours.

For sqlite caching, the issues arise from “too much data for one table row…” errors which accumulate to finally crash the whole caching mechanisms. I don’t think that this is a server issue.



Running file caching method for 2 days now since. Not having the same problems as sqlite method. And I have not cleared the cache since and the site is still alive and kicking with no cache related errors (I think). As opposed to when I HAD to clear cache at least twice daily when I was running v2.0.x version of Cscart, else it will result in cache_handlers.php errors.



But one thing that worries me for now is that the number of subdirectories (mostly cquery_xxx, pfilters_xxx subdirs) in the var/cache directory is growing v fast. As of this moment, there are 17589 subdirectories inside my var/cache directory! (Got this number via bash running this " find . -type d -maxdepth 1 | wc -l")

I don’t see the point with CS. Don’t get me wrong, I know what cache is “supposed” to do but I have tested with and without and I see no difference in speed. So I will stick with “without” since it does not create any problems.

Nodame,


[QUOTE]Something must be wrong with your server setup[/QUOTE]



I agree in that your errors are most likely due to a difference in your server settings/installed modules. I have been using the sqlite method for well over a month now (even weeks before 2.1.1) and have not had any problems.



I am not a linux server expert (operated on Windows IIS for around 10 years prior to CS-Cart) so I cannot give you much in the way of specifics, I can only say you should most definitely be able to get the sqlite method running without errors.

[quote name=‘The Tool’]I don’t see the point with CS. Don’t get me wrong, I know what cache is “supposed” to do but I have tested with and without and I see no difference in speed. So I will stick with “without” since it does not create any problems.[/QUOTE]



Tool,



Have you disabled the cache in 2.1.1 yet, I would like to give this latest version another try without the cache if you have time to share? Thanks

[quote name=‘Struck’]Tool,



Have you disabled the cache in 2.1.1 yet, I would like to give this latest version another try without the cache if you have time to share? Thanks[/QUOTE]



I haven’t even dabbled into 2.1 yet but if CS finally has their shit together with 2.1.2 and dimensional shipping, I will definitely be moving in that direction.

[quote name=‘Struck’]Nodame,







I agree in that your errors are most likely due to a difference in your server settings/installed modules. I have been using the sqlite method for well over a month now (even weeks before 2.1.1) and have not had any problems.



I am not a linux server expert (operated on Windows IIS for around 10 years prior to CS-Cart) so I cannot give you much in the way of specifics, I can only say you should most definitely be able to get the sqlite method running without errors.[/QUOTE]



I’ve logged a bug report via the bug tracker on the sqlite errors “too much data…” generated. Hopefully cscart team will be able to figure out what caused this error?



I’ve used quite a number of filters for my products. Which I realized generated a lot of pfilters_xxx type cache entries. Wonder if it is a factor~



Besides speed, does caching helps in reducing the number of mysql connections, thus sql queries, I wonder?



If not, based on Tool’s experience, maybe just disabling it might be all for the better?~

All this cache thing is a mystery to a lot of us and I think we have a lot of other things that are more important than becoming cache experts.



I read an article about a library that was developed to help speed the page load by not showing content that is not visible on the screen to the user . Here is the Quote from the article that was done By David Artz:

[QUOTE]

Recent research by Clicktales shows that a range of 15% to 20% of page views actually reach the very bottom of the page and 32% to 36% of page views are unlikely to reach the 1000 pixel line. Yet we still serve up this unseen content and spend bandwidth on functionality, images, and rich media that users may never even get to in their journey.



This library was developed to test objects to see if they are visible on screen so we can make decisions on whether or not we should load them in. Sonar can also be used in cases where you may want to only perform certain actions (such as an ad refresh) if the user is actually viewing the object.







Introducing Sonar Detection



At the core of Sonar is its ability to detect if an object is visible on the user’s screen. The detect() method is used to do this, and returns true or false if the element is within a certain threshold from the edge of the screen. For example:



var detected = sonar.detect(document.getElementById(“myobject”), 400);



In the above code, the detected variable will be set to true if the “myobject” node is within 400 pixels from the edge of the screen, or false if not.



This can be useful for only doing certain events when an object is visible on the screen, for example refreshing advertisements. You would not want to refresh advertisements invisible to the user, and Sonar detection can be used to do this.

Polling Objects with Sonar Detection



Sonar comes with a built in polling feature that polls the objects with sonar.detect() as the user scrolls. Once an object is detected, it executes a callback function that you can define. Think of it as a way to turn Sonar detection into an event you can hook on to any object.



To poll objects with Sonar, do the following:



sonar.add(

{

obj: document.getElementById(“myobject-div”),

call: function(object)

{

// example code to embed a flash player

swfobject.embedSWF(“player.swf”, “myobject-div”, “640”, “500”, “9.0.0”,

{}, {file: “my-flash-movie.flv”, fullscreen: true},

{allowfullscreen: true, allowscriptaccess: “always”});

},

px: 400

});



In the above example, a Flash player will be injected into the “myobject-div” when the user scrolls to it. Let’s examine the construct a little further.



* obj: A reference to an object you want to poll with Sonar, or the “id” of the object.

* call: An anonymous function called once the object is detected with Sonar.

* px: The distance in pixels above and below the screen to include in the detection range.



Worth noting is that the callback function is passed back the object in the argument, making it easy to reference the object that triggered the detection. An example of this is coming up next.



Once all objects have been detected, Sonar politely removes itself from the onScroll event, and no longer polls for them.

Extending the Sonar Library



By now your mind is probably racing with all the cool things you can do with this simple library. One of the things we have wanted to do for a while is figure out a way to only load images when they are needed. A case study follows.



Let’s pick on a popular blog that provides amazing live-blogging coverage of MacWorld and other conferences. A run of Pagetest shows us the extent of the damage:



Engadget MacWorld 2008 Pagetest Results



* Requests: 200

* Bytes In: 5.6 MB

* Load Time: 35.2 Seconds



Wouldn’t it be great if we could stop the 50 images below the fold from loading, and load them when Sonar detects them? What would that look like?



“Ionized” Engadget MacWorld 2008 Pagetest Results



* Requests: 150 (25% Improvement)

* Bytes In: 1.2 MB (79% Improvement)

* Load Time: 9.4 Seconds (73% Improvement)



Saving 50 requests and 4.4 MB of bytes shaved over 25 seconds off this live blog’s load time!

The Sonar Ion (Images on Need) Extension



Ion is a simple function that does the following:



* Grab all the images on the page

* Loop through images

and for each one…

o Use sonar.detect to see if it is on the screen

o If not, set the source to nothing and sonar.add it to be polled on scroll

* Once detected, set the source of the image back to the original value



All that is required to get this going is including the following line of code:



sonar.ion(1000);



That’s it! The argument is the distance off the edge of the screen we want to consider the object within range of detection.



If you are interested in the guts of how this works, take a look at the code below. Pay special attention to the call property function, which is triggered once the image is within range of sonar.detect().



sonar.ion = function(distance)

{

// Grab all the images on the page.

var default_distance = 0, // Default distance.

images = document.getElementsByTagName(‘img’),

transparent_pixel = “http://www.artzstudio.com/i/x.gif”, // Needed for Firefox.

i;



distance = distance || default_distance;



// Loop through all the images.

//

for (i in images)

{

// If the image is not on the screen within the distance…

if (!sonar.detect(images[i], distance))

{

// Temporarily store the true source as a temporary

// property (__src) of this object.

images[i].__src = images[i].src;



// Replace the source with a blank image (Firefox) or a

// blank string (IE). No solution for Safari, as it

// downloads all images regardless of what we do.

images[i].src = (document.all) ? “” : transparent_pixel;



// Add the image to the sonar object, where it will then

// be polled as the user scrolls and execute the callback

// function once it is within range.

sonar.add({

obj : images[i],

px : distance,

call : function(object)

{

// Set the blank image source to the true value.

object.src = object.__src;



// Set the temporary property to undefined so

// the sonar.detect() function is no longer run on it.

object.__src = undefined;

}

});

}

}

};



Ion works great in IE and Firefox, but in Safari no matter what we tried we could not stop the download of the images. We even tried ripping them out of the DOM - no dice. Don’t let this stop you from giving it a shot yourself, (and telling me how you did it)!

Conclusion & Final Thoughts



Sonar helps us bridge the gap in loading anything - content, JavaScript, rich media, images - only when needed by users.



Think of combining Sonar with ideas like loading script elements or stylesheets into the document as needed.





[/QUOTE]



Here is a link to that JS library: [url]http://www.artzstudio.com/files/sonar/sonar2.js[/url]



Now if we can get somebody to implement this with CS along with everything else we have been doing , smartoptimizer, etc… we will all be cooking with Gas , don’t you think.

The caching starts to break down when a site uses a lot of features and filters. CS’s choice to cache filters creates a separate cache entry (a file for ‘files’ type and a row for either ‘mysql’ or ‘sqlite’ - assume shared memory is similar to ‘file’).



The problem in the files type is the sheer number of files. For each page load, the cache is read to load the Registry. In a ‘files’ based caching system, there is no index of the ‘tag’ for the cache. Hence, to find the cache file ABC.php, one has to open the underlying directory (var/cache) and read the listing until ABC.php is found. In a directory, with 1000’s of files in it, this can obviously put quite a strain on the underlying server, especially if that server is trying to serve other processes (even for the same site).



In the sqlite, mysql type of schemes, the ‘tags’ are indexed. So the index file is read which essentially lets the system find the object with much less overhead.



The other issue with the ‘files’ method is that there is no locking of data. Each time a page is loaded, it actually goes through the whole load process about 3 times due to the AJAX requests being done. At the end of every page load, a write is done to some of the registry entires like “last_url” and other things. So you have 3 processes all trying to write to a single large directory at once. Multiply that times the number of simultaneous users you have and you’ll begin to see the problem. Note too that a block in a file can be writen with new data while a process is reading the same block with different data.



The mysql version certainly should not be an issue except that the cache also caches recent DB query results. So it would be caching what it’s doing… But if a few MB of table data causes a problem in mysql then there’s a bigger underlying problem. Mysql should be able to handle single rows of 16MB by default and can be configured to use much larger…



My recommendation would be not to cache filters/features. Simply build the appropriately indexed tables from the admin interface when the features/filters are updated and all will be well. Without feature/filter caching, almost all types of caching will work well without issue. Sqlite should be fasted of the 3. I do not know about shared memory implementation.

Very interesting post - although I did not understand exactly what I should be doing…



I use the default (files) and I do not have any problems that I know of.



Perhaps 4000 product photos but few add ons enabled and only about 300 customers per day.



Should I be pro-actively changing something?

[quote name=‘tbirnseth’]The caching starts to break down when a site uses a lot of features and filters. CS’s choice to cache filters creates a separate cache entry (a file for ‘files’ type and a row for either ‘mysql’ or ‘sqlite’ - assume shared memory is similar to ‘file’).



The problem in the files type is the sheer number of files. For each page load, the cache is read to load the Registry. In a ‘files’ based caching system, there is no index of the ‘tag’ for the cache. Hence, to find the cache file ABC.php, one has to open the underlying directory (var/cache) and read the listing until ABC.php is found. In a directory, with 1000’s of files in it, this can obviously put quite a strain on the underlying server, especially if that server is trying to serve other processes (even for the same site).



In the sqlite, mysql type of schemes, the ‘tags’ are indexed. So the index file is read which essentially lets the system find the object with much less overhead.



The other issue with the ‘files’ method is that there is no locking of data. Each time a page is loaded, it actually goes through the whole load process about 3 times due to the AJAX requests being done. At the end of every page load, a write is done to some of the registry entires like “last_url” and other things. So you have 3 processes all trying to write to a single large directory at once. Multiply that times the number of simultaneous users you have and you’ll begin to see the problem. Note too that a block in a file can be writen with new data while a process is reading the same block with different data.



The mysql version certainly should not be an issue except that the cache also caches recent DB query results. So it would be caching what it’s doing… But if a few MB of table data causes a problem in mysql then there’s a bigger underlying problem. Mysql should be able to handle single rows of 16MB by default and can be configured to use much larger…



My recommendation would be not to cache filters/features. Simply build the appropriately indexed tables from the admin interface when the features/filters are updated and all will be well. Without feature/filter caching, almost all types of caching will work well without issue. Sqlite should be fasted of the 3. I do not know about shared memory implementation.[/QUOTE]



Yes, tbirnseth got the problem down by stating that "The caching starts to break down when a site uses a lot of features and filters. "

In reply to my bug at the bug tracker, imac have asked me how many features/categories/filters am I using in my store and how many products. Obviously, they are very aware of this problem. I’ve replied him already and his reply is that they will be investigating more into the problem.



Just to state that for “files” method, if you are using a lot of features/fitlers/categories with a lot of products, like me for example, my record number of subdirectories within var/cache directories had hit to about 37000 directories, before I cache refresh it the next day.



Even cache refreshing the “file” method was problematic for this amount of subdirectories. I had to rename the cache directory to eg. cache2 and create a new “cache” dir and then delete the cache2 dir. As deleting a dir with >30000 subdirs recursively takes simply TOOOOO long~ (Even via SSH) And will interfere with the running of the store as it is at the same time actively writing more subdirs cache files into the directory.



And if going by tbirnseth’s understanding of the cache mechanism of cscart for file method

[quote]

In a ‘files’ based caching system, there is no index of the ‘tag’ for the cache. Hence, to find the cache file ABC.php, one has to open the underlying directory (var/cache) and read the listing until ABC.php is found. In a directory, with 1000’s of files in it, this can obviously put quite a strain on the underlying server,

[/quote]

Imagine it not being 1000’s , but >30000 subdirs, where within each subdir is 1 cache.csc file~I think the server will just simply go “bonkers”.



Now I am back to sqlite method, and monitoring the cache size, and deleting the cache.db file to refresh it whenever it hit a certain size, or an error_log is generated due to “too much data” or “data locked” or other sqlite related errors.



I am not sure how effectively is cscart lightening the load on sql connections by implementing their current method of caching features/filters. And I DO appreciate the effort if it is indeed lightening server /sql load via caching (as running cscart does put a certain strain on sql with its many complicated sql statements). But I think they should have taken into account that a cscart can have unlimited features and filters and categories as “marketed” by them, and code the cache mechansim and test them according to such scale-up situations.

@traveler

[quote]

Should I be pro-actively changing something?

[/quote]



I’d stick with the addage “If it ain’t broke, don’t fix it”. If you use more than a couple of features and have more than 1000 products, you might want to consider using sqlite method. Especially if you use filters.



@nodame

[quote]

Imagine it not being 1000’s , but >30000 subdirs, where within each subdir is 1 cache.csc file~I think the server will just simply go “bonkers”.

[/quote]

Actually, this is using the system’s directory structure as an index. But note that the parent directory still has to be read in order to find the child directory. So if there are 30K, then that’s 30K (inodes in Unix-speak) that have to be looked at. Note also that the Kernel caches the inodes in use. So if the system inode cache isn’t big enough then it’s fighting with itself to simply read/write the files.



In a former life, I would have built the caching architecture based on a hash table of the keys at the various cache levels. The current is a multi-level cache (good) but has no real index.



Not sure about sqlite, but mysql will lock a table when it’s in the process of updating the table and its indexes. So if you’re seeing errors related to “data locked” it’s probably a result of too many PHP processes hitting the cache at the same time.



Caching needs an architectural review from several perspectives:

  1. page load speed (single user, lightly loaded system - get rid of AJAX on loads)
  2. Cache accesses per page load (single user, lightly loaded system).
  3. page load speed 10 simultaneous users navigating different areas of a site (single site).
  4. Cache accesses (system) with 10 simultaneous users. I.e. try to cause “contention” and engineer to avoid.

I’d stick with the addage “If it ain’t broke, don’t fix it”. If you use more than a couple of features and have more than 1000 products, you might want to consider using sqlite method. Especially if you use filters.



tbirnseth,



Thank you.



I will stay with the files system for now. I only have a little more than 1000 products and I do not know what a filter is - so I probably am not using many…

with 2.1.4 now, what caching are you guys using?

The most advanced of us CS-Cart users strongly prefers the SQLITE cache method, set it & forget it. :wink:



However, there is one “Super Advanced User” which does not use any built-in Cache at all, this user is in a league of his own thus you should not attempt this! :stuck_out_tongue:

All websites with cs cart in all my servers have millions of dirs in cache files.

I never saw 14 millions of old files in a single site with feew products, what kind of software is this ?

All sites i'll be blocked until this type of cache work without make this mess.

1000 visitors open 10 products… great… more 100000 dirs in server.



Some one can think a little about it. I never saw nothing like this.



Thank You… for the time i need to wast only to correct this on server