Varnish & Edge Side Includes

Hey Everyone -

Looking for some help with Varnish configs in front of a couple AWS ec2 instances that run our CS-Cart MV servers.

We're using Varnish in front of our CS Cart MV setup (NGINX terminated ssl proxy to Varnish then load balanced to 2 CS Cart servers running NGINX). I've noticed if I leave this in my vcl_backend_response for Varnish that for any user signed in CS Cart MV gets REALLY slow:

if (beresp.http.X-Do-ESI) {
set beresp.do_esi = true;
Basically if ESI is on, the backend gets super slow. I see waterfall response times of 12s+ which is absolutely crazy. If I comment the set beresp.do_esi = true; out, then I get something int he 3s - 4s range, which is still slow, but not 12s+ slow.
When properly cached, we'll get a page back in 30ms or so, which is what we're after in most cases.
Does anyone have any experience trouble shooting Varnish response times in Amazon EC2 environments? I'm also seeing some really inconsistent hit rates on most pages (home page, category pages, etc.). I'm looking for some help with configs, etc. so I can learn and start managing this on my own.
Thanks in advance!