We’ve got a site that is running pretty slow. Time to first byte (TTFB) can be up to 5 seconds or more. This is a lot of waiting for the customer who is supposed to be browsing the site. I’ve tried optimising PHP, Nginx, Litespeed, MySQL etc but have had no luck yet.



The server we are using has the following specs -



CPUs - 4 x 12 core @ 2.1 Ghz (48 cores in total)

Memory - 512 GB DDR3

Drive - SSD running at 500 MB/s

Running Centos with cPanel



The database is a decent size but nothing huge. It has the following records -



new_orders - 8,523,192

products_categories - 51,236

order_details - 28,839

images - 27,659

product_sales - 22,119

product_option_variants - 19,558

product_option_variants_descriptions - 19,558

orders - 17,744



MySQL slow logs the following slow queries -





/usr/sbin/mysqld, Version: 5.1.68-cll (MySQL Community Server (GPL)). started with:

Tcp port: 3306 Unix socket: /var/lib/mysql/mysql.sock

Time Id Command Argument

Time: 130412 16:12:32

User@Host: onlineshop_user[onlineshop_user] @ localhost []

Query_time: 2.836568 Lock_time: 0.000694 Rows_sent: 12 Rows_examined: 225052

use onlineshop_basedb;

SET timestamp=1365747152;

SELECT SQL_CALC_FOUND_ROWS products.*, descr1.product as product, MIN(IF(prices.percentage_discount = 0, prices.price, prices.price - (prices.price * prices.percentage_discount)/100)) as price, descr1.short_description, IF(descr1.short_description = ‘’, descr1.full_description, ‘’) as full_description, companies.company as company_name, GROUP_CONCAT(IF(products_categories.link_type = ‘M’, CONCAT(products_categories.category_id, ‘M’), products_categories.category_id)) as category_ids, products_categories.position, os_seo_names.name as seo_name FROM os_products as products LEFT JOIN os_product_descriptions as descr1 ON descr1.product_id = products.product_id AND descr1.lang_code = ‘EN’ LEFT JOIN os_product_prices as prices ON prices.product_id = products.product_id AND prices.lower_limit = 1 LEFT JOIN os_companies AS companies ON companies.company_id = products.company_id INNER JOIN os_products_categories as products_categories ON products_categories.product_id = products.product_id INNER JOIN os_categories ON os_categories.category_id = products_categories.category_id AND (os_categories.usergroup_ids = ‘’ OR FIND_IN_SET(0, os_categories.usergroup_ids) OR FIND_IN_SET(1, os_categories.usergroup_ids)) AND os_categories.status IN (‘A’, ‘H’) LEFT JOIN os_seo_names ON os_seo_names.object_id = products.product_id AND os_seo_names.type = ‘p’ AND os_seo_names.dispatch = ‘’ AND os_seo_names.lang_code = ‘EN’ WHERE 1 AND os_categories.category_id IN (1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 150, 155, 168, 169) AND (companies.status = ‘A’ OR products.company_id = 0) AND (products.usergroup_ids = ‘’ OR FIND_IN_SET(0, products.usergroup_ids) OR FIND_IN_SET(1, products.usergroup_ids)) AND products.status IN (‘A’) AND prices.usergroup_id IN (0, 0, 1) GROUP BY products.product_id ORDER BY descr1.product asc LIMIT 0, 12;

Time: 130412 16:12:40

User@Host: onlineshop_user[onlineshop_user] @ localhost []

Query_time: 2.685642 Lock_time: 0.000539 Rows_sent: 12 Rows_examined: 133744

SET timestamp=1365747160;

SELECT SQL_CALC_FOUND_ROWS products.*, descr1.product as product, MIN(IF(prices.percentage_discount = 0, prices.price, prices.price - (prices.price * prices.percentage_discount)/100)) as price, descr1.short_description, IF(descr1.short_description = ‘’, descr1.full_description, ‘’) as full_description, companies.company as company_name, GROUP_CONCAT(IF(products_categories.link_type = ‘M’, CONCAT(products_categories.category_id, ‘M’), products_categories.category_id)) as category_ids, products_categories.position, os_seo_names.name as seo_name FROM os_products as products LEFT JOIN os_product_descriptions as descr1 ON descr1.product_id = products.product_id AND descr1.lang_code = ‘EN’ LEFT JOIN os_product_prices as prices ON prices.product_id = products.product_id AND prices.lower_limit = 1 LEFT JOIN os_companies AS companies ON companies.company_id = products.company_id INNER JOIN os_products_categories as products_categories ON products_categories.product_id = products.product_id INNER JOIN os_categories ON os_categories.category_id = products_categories.category_id AND (os_categories.usergroup_ids = ‘’ OR FIND_IN_SET(0, os_categories.usergroup_ids) OR FIND_IN_SET(1, os_categories.usergroup_ids)) AND os_categories.status IN (‘A’, ‘H’) LEFT JOIN os_seo_names ON os_seo_names.object_id = products.product_id AND os_seo_names.type = ‘p’ AND os_seo_names.dispatch = ‘’ AND os_seo_names.lang_code = ‘EN’ WHERE 1 AND os_categories.category_id IN (2, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124) AND (companies.status = ‘A’ OR products.company_id = 0) AND (products.usergroup_ids = ‘’ OR FIND_IN_SET(0, products.usergroup_ids) OR FIND_IN_SET(1, products.usergroup_ids)) AND products.status IN (‘A’) AND prices.usergroup_id IN (0, 0, 1) GROUP BY products.product_id ORDER BY descr1.product asc LIMIT 0, 12;

Time: 130412 16:12:46

User@Host: onlineshop_user[onlineshop_user] @ localhost []

Query_time: 1.069359 Lock_time: 0.000524 Rows_sent: 12 Rows_examined: 51434

SET timestamp=1365747166;

SELECT SQL_CALC_FOUND_ROWS products.*, descr1.product as product, MIN(IF(prices.percentage_discount = 0, prices.price, prices.price - (prices.price * prices.percentage_discount)/100)) as price, descr1.short_description, IF(descr1.short_description = ‘’, descr1.full_description, ‘’) as full_description, companies.company as company_name, GROUP_CONCAT(IF(products_categories.link_type = ‘M’, CONCAT(products_categories.category_id, ‘M’), products_categories.category_id)) as category_ids, products_categories.position, os_seo_names.name as seo_name FROM os_products as products LEFT JOIN os_product_descriptions as descr1 ON descr1.product_id = products.product_id AND descr1.lang_code = ‘EN’ LEFT JOIN os_product_prices as prices ON prices.product_id = products.product_id AND prices.lower_limit = 1 LEFT JOIN os_companies AS companies ON companies.company_id = products.company_id INNER JOIN os_products_categories as products_categories ON products_categories.product_id = products.product_id INNER JOIN os_categories ON os_categories.category_id = products_categories.category_id AND (os_categories.usergroup_ids = ‘’ OR FIND_IN_SET(0, os_categories.usergroup_ids) OR FIND_IN_SET(1, os_categories.usergroup_ids)) AND os_categories.status IN (‘A’, ‘H’) LEFT JOIN os_seo_names ON os_seo_names.object_id = products.product_id AND os_seo_names.type = ‘p’ AND os_seo_names.dispatch = ‘’ AND os_seo_names.lang_code = ‘EN’ WHERE 1 AND os_categories.category_id IN (129, 130, 133, 149, 135, 141, 143, 144, 145, 146, 147, 148, 151, 152, 153, 154, 156, 167) AND (companies.status = ‘A’ OR products.company_id = 0) AND (products.usergroup_ids = ‘’ OR FIND_IN_SET(0, products.usergroup_ids) OR FIND_IN_SET(1, products.usergroup_ids)) AND products.status IN (‘A’) AND prices.usergroup_id IN (0, 0, 1) GROUP BY products.product_id ORDER BY descr1.product asc LIMIT 0, 12;

Time: 130412 16:12:47

User@Host: onlineshop_user[onlineshop_user] @ localhost []

Query_time: 0.228424 Lock_time: 0.000527 Rows_sent: 1 Rows_examined: 26155

SET timestamp=1365747167;

SELECT MIN(os_product_prices.price) min, MAX(os_product_prices.price) max FROM os_product_prices LEFT JOIN os_products ON os_products.product_id = os_product_prices.product_id LEFT JOIN os_product_prices as prices_2 ON os_product_prices.product_id = prices_2.product_id AND os_product_prices.price > prices_2.price AND prices_2.lower_limit = 1 AND prices_2.usergroup_id IN (0, 0, 1) INNER JOIN os_products_categories ON os_products_categories.product_id = os_products.product_id LEFT JOIN os_categories ON os_categories.category_id = os_products_categories.category_id LEFT JOIN os_companies as companies ON companies.company_id = os_products.company_id WHERE os_products.status IN (‘A’) AND os_products_categories.category_id IN (130, 133, 135, 141, 143, 144, 145, 146, 147, 148, 149, 151, 152, 153, 154, 156, 167, 129) AND (os_categories.usergroup_ids = ‘’ OR FIND_IN_SET(0, os_categories.usergroup_ids) OR FIND_IN_SET(1, os_categories.usergroup_ids)) AND (os_products.usergroup_ids = ‘’ OR FIND_IN_SET(0, os_products.usergroup_ids) OR FIND_IN_SET(1, os_products.usergroup_ids)) AND os_categories.status IN (‘A’, ‘H’) AND os_products.status IN (‘A’) AND (companies.status = ‘A’ OR os_products.company_id = 0) AND os_product_prices.lower_limit = 1 AND os_product_prices.usergroup_id IN (0, 0, 1) AND prices_2.price IS NULL;



I’ve tried running these commands into MySQL directly and it gives some slow queries -



Showing rows 0 - 11 ( 12 total, Query took 5.1814 sec)



Either MySQL is not optimised well or CS Cart database programming is totally inefficient.



I’ve gone through various versions of my.cnf but here is the current one -

Example MySQL config file for large systems.

#

This is for a large system with memory = 512M where the system runs mainly

MySQL.

#

MySQL programs look for option files in a set of

locations which depend on the deployment platform.

You can copy this option file to one of those

locations. For information about these locations, see:

http://dev.mysql.com/doc/mysql/en/option-files.html

#

In this file, you can use all long options that a program supports.

If you want to know which options a program supports, run the program

with the “–help” option.

# The following options will be passed to all MySQL clients
[client]
#password = your_password
port = 3306
socket = /var/lib/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-locking
safe-show-database
key_buffer_size = 256M
max_allowed_packet = 128M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 256
query_cache_size= 64M
query_cache_limit=2M
query_cache_type=1
table_cache=1024
join_buffer=2M
record_buffer=1M

# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin

# binary logging format - mixed recommended
binlog_format=mixed

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=, MASTER_PORT=,
# MASTER_USER=, MASTER_PASSWORD= ;
#
# where you replace , , by quoted strings and
# by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host =
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user =
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port =
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 256M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 64M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout



I have contacted CS Cart support and waiting for their response. Any suggestions would be appreciated!

Hi,



What version of CS-Cart?



How many records in categories table?





Can you also run following on your MySQL and let us know output:



SHOW STATUS LIKE '%Qcache%';



How is the load on the database server? Any IO wait?



Try briefly disabling blocks (mega menu for example) and refreshing page to see effect on page load time. Try and get it nailed down to a particular block. I've seen mega menu can be very slow with lots of categories. Maybe also check smarty debugging output for timing information.

Hi,



Running 3.0.4



There are 159 records in categories…



Here is the result from the query -



Qcache_free_blocks 4671 Qcache_free_memory 18250264 Qcache_hits 14502382 Qcache_inserts 4691373 Qcache_lowmem_prunes 2788791 Qcache_not_cached 36685 Qcache_queries_in_cache 23069 Qcache_total_blocks

51370



MySQL is running on the same server and they are the only user on the server so there shouldn't be any IO waits at all.

Can you just check in admin and confirm if store operates in LIVE or DEVELOPMENT mode?

It is currently in LIVE mode

Hi Speedy,



Sorry for not following up in more detail… been full of cold and off work.



Are you able to SSH in and take a look at output of “top” command?



This should give you an idea what's going on server wise.



I'm interested in the three figures in the top right which indicate average number of processes waiting on resources over the past 1, 5 and 15 minutes:



load average: 0.00, 0.00, 0.00



I'm also interested in the figures below like these:



Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st



In particular, give me an idea what the ones labelled as following read:



“us” (user-space CPU)

“id” (processor time idle)

“wa” (waiting on IO)



And finally looking at the process list, by default they'll be sorted by what's taking most CPU.



Give me an idea what's at the top, is it Apache or MySQL?



To it's left is a column headed S (status). what's it show for top processes?



Are there a few, or tens, or hundreds of stacked Apache processes waiting?

Does the site run slow in general, or just specific URLs?

Does the site run slow in general, or just specific URLs?

Your MySQL settings don't look at all unreasonable, but you have a lot of memory, and when it comes to MySQL you have to decide where to spend it. You might consider increasing your [color=#282828][font=arial, verdana, tahoma, sans-serif]query_cache_size to 128MB. [/font][/color]

Hi,



I'm very interested in this topic too, we've had major issues with MySQL overloading and throwing the server offline during busy periods, disaster with 2.012, seems better in 2.1.4 ( we havent upgraded yet to v3 ). But interested in this as we have several websites in v3.06.



Just one thing we noticed, having no error page defined in the blocks system caused huge usage in MySQL, it didn't seem to shut down MySQL threads, ( it left them open, until they built up and overloaded and mysql / server was restarted ).



Also the “Top 10” we had on our 1st page was sending huge queries looking up all sale records of all items over time to generate. Might be an idea to scrap this box if its on your page too.



We're currently looking into running an auto scaling system in the Amazon AWS platform. Currently working with notes from Wordpress and trying to replicate the setup, then stress test the scale up / down, until CS-CART acknowledge that mysql speed / optimisation needs to be a big feature for the future of the cart.



Stephen

In early version 2 sites the cache system was still very much under development if I remember right.



But In general I've found later version 2 sites (2.4/2.5) to perform well.



That having been said I don't doubt there are some slow queries in there.



But nothing that can't be mitigated by combination of CS-Cart's native cache and MySQL query cache.



You just have to debug these things to get a feel for what's going on.



For example are we dealing with a busy site, where a few bad queries are hogging MySQL threads for a long period of time, causing otherwise simple queries to wait on an available MySQL connection?



There's a command called mysqltop (or is it mytop, I forget) which will give you a good insight in to what's going on with MySQL.



Bear in mind that it's not unusual to have a few slow queries logged, because even if subsequently cached any distinct heavy query will naturally still be heavy the first time it's run, before it's cached.



But if the same heavy query remains heavy all the time then perhaps it's too big to fit in the MySQL Query cache.

And so we mitigate that by adjusting cache settings.

Of course if your app doesn't close connections then that will cause problems, only mitigation is very low mysql timeout, but that's a hack, better fixed within the application.

Here’s the print out from top -





top - 14:33:26 up 7 days, 17:48, 2 users, load average: 0.01, 0.04, 0.01

Tasks: 872 total, 1 running, 871 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 529242500k total, 6854112k used, 522388388k free, 392220k buffers

Swap: 4194296k total, 0k used, 4194296k free, 2802696k cached



I installed mytop -





MySQL on localhost (5.1.68-cll) up 4+13:59:51 [14:53:02]

Queries: 9.0 qps: 0 Slow: 0.0 Se/In/Up/De(%): 215156/00/00/00

qps now: 0 Slow qps: 0.0 Threads: 1 ( 1/ 4) 700/00/00/00

Key Efficiency: 99.6% Bps in/out: 0.0/ 0.1 Now in/out: 10.5/ 1.9k



Id User Host/IP DB Time Cmd Query or State

– ---- ------- – ---- — ----------

5325 root localhost onlineav_b 0 Query show full processlist







This server is sitting in our office waiting to be commissioned for some of our hosting infrastructure. I put the site onto this server to show CS Cart that they need to make their cart faster, even when the server specs are ridiculously high - there’s enough memory, enough cpu, the server is on a 1gb connection to this desktop, there is no one else on this machine and it’s a fresh install.



It eliminates any server issues that they can come up with. I’ve got more of these servers which I can setup if we need to do more testing