Emergency! Memory is getting crushed!

Hi All,
We are having a pretty severe issue right now with CS-Cart and can’t figure it out. Quick background:

  • We are using CS-Cart Multivendor and have almost 5,000 vendors on our site
  • We are running in AWS with 6 EC2 instances

Earlier this week memory usage spiked to like 98% on all the servers. It’s causing slowness on the site and complaints from buyers and vendors of things timing out.

We doubled the size of our memory tonight on the EC2 instances thinking it would solve the issue…and it’s starting to crawl even higher. The instances are now over 70% and growing…even though the memory is doubled from what it was.

Is there anyone here that has ideas on how to solve this issue? Is there something in the PHP Code causing a leak or something?

We are happy to hire someone that truly had experience with Cs-Cart PHP code to dig into things to make them more efficient. Our current developers are currently at a loss with this issue.

I appreciate anyone’s thoughts or if they can help.

Jon

Hi!

Do you have any idea how many concurrent requests per minute your installation is processing? Are they all payloads, or are there a significant number of bots crawling your site? Since you will need to check the access.log to verify this information, please also take a look if there are any requests that contain the items_per_page= parameter with a huge number passed.

Along with what @CS-Cart_team suggests, also look for excessive searches. I have run into issues where I would get no name bots from all over the world running excessive searches at the same time.

Thank you for this suggestion. It’s certainly something we will look into!

@CS-Cart_team

Thanks for replying. Currently we show 24 products per page. That said, the biggest issue right now isn’t just browsing the site, but our vendors using the vendor admin panel. They are having trouble creating new products, managing orders, etc.

Since the memory in our EC2 instances is at 90% it’s bogging down the site. As mentioned, even after doubling the memory of the servers the memory continued to increase. So there seems to be a memory leak we need help solving.

Does CS-Cart have any professional services to help us research and diagnose the issue?

Thanks

Jon

What @CS-Cart_team is referring to is that items_per_page= can have any number appended so be looking for high numbers (suggested to @CS-Cart_team in another thread that this needs to be changed to limit the number and they agreed). I haven’t seen any bots go over 96 on my site but scrolling through pages at a fast rate with a high number will put an excessive load on server.

Do you have something like cloudflare or another firewall in between? Maybe you can have a look at the traffic and look for unusual things and stop these. I know that as soon as I stop my special rules my website will crash within days because of whatever hits hit.

1 Like

How is your server setup?

One problem could be that you have set up php as mod_php.
That would mean php is loaded by apache even when it serves image files. Also you don’t get php object caching

If you setup php-fpm, then you wouldn’t have the above problem.

EDIT: Could also be any other problem. best to get someone who knows what they are doing to look at the whole thing