Speed was not the major issue, however now is better, cannot remember exactly how much worse it was exactly. Main issue was with caching, even I had disabled cdn caching for some parts of web I was not getting IPN messages from paypal, sometimes do sometimes not, so If I cannot get a reliable responds via cdn I do not need it then..
How much worse where you off with compared to without ?
I had a mobile / desktop rating with CDN of 67 / 79 and without 79 / 93