My site is modded up to the neck “No problems with speed” try it.
I’ve had no problems with speed, although I don’t have any loading time figures or anything and only very few products at the moment.
You can always disable modules you are not using, strip out the templates etc.
It depends on the mods I suppose. JS is a big bottleneck to speed as are images. If both of these are optimized you shouldn’t have any problems.