Declined Orders Processing Anyway

We have PayPal Pro setup (and have had very little problems for many, many years), and today we are seeing a bunch of customer payments getting this message: "Your order has been declined by the payment processor. Please review your information and contact store administration." But then the payment is processed anyway and the order is created.

But the customer is stuck on the billing page to re-enter the payment information.

It seems like our cart is receiving (or just displaying) the "decline" notice before the "payment all clear" notice is given and causing issues. We contacted PayPal and they sent this back: "We have reviewed the logs and unfortunately, we are not sending any decline response. We have only sent a "Success" API response to your server for the payments."

Any information would be greatly appreciated or which log files would point us in the right direction. The admin logs showed this:

Orders (status change)
Order: # 11111
Status: Incomplete -> Failed

but no other information was with that notice. Then it went to PayPal, Response: VERIFIED,

then went to this:

Orders (status change)
Order: # 11111
Status: Failed -> Incomplete

then to this:

Orders (status change)

Order: # 11111
Status: Incomplete -> Processing

I attached a screenshot of the cart checkout screen message that some customers were getting after entering payment information. I tried to remove any identifying information...

Thanks!

More info: It is sending out this message code: text_order_placed_error

And for some reason the orders are going from "Incomplete" -> "Failed" and skipping "Open" and "Processing" statuses before the payment processor sends back the official signal.

This seemed to be the normal flow of orders when everything was working ok just two days ago:

1) Orders (create)

2) Requests (http/https request) - URL: https://api-3t.paypal.com:443/2.0/

3) Status: Incomplete -> Open

4) Requests (http/https request) - URL: https://www.paypal.com/cgi-bin/webscr

5) Status: Open -> Incomplete

6) Status: Incomplete -> Processing --- Order Successfully in the system

Now we seem to have this:

1) Orders (create)

2) Requests (http/https request) - URL: https://api-3t.paypal.com:443/2.0/

3) Status: Incomplete -> Failed

4) Requests (http/https request) - URL: https://www.paypal.com/cgi-bin/webscr

5) Status: Failed -> Incomplete

6) Status: Incomplete -> Processing --- The Order is in the system, but the customer is still left hanging on the payment checkout screen with the "Your order has been declined by the payment processor. Please review your information and contact store administration." error message. I modified the message to help customers until it is fixed.

Any thoughts?

(I also tried to re-attach the image file from the first post - maybe some duct-tape will help this time around - lol)

[attachment=14421:Payments-Declined-6-5-20.gif])

Payments-Declined-6-5-20.gif

More Info - Part 2:

It seems to be almost slightly related to this post:

https://forum.cs-cart.com/topic/55460-checkout-card-payment-problem/?hl=paypal+response#entry317101

But our "Order statuses" for "Open" has inventory set on: Decrease (so that should not be the problem here)

Today I noticed in the initial "Requests (http/https request) - URL: https://api-3t.paypal.com:443/2.0/ Request: " log the orders that have Status: Incomplete -> Failed right away (seems like all of them now) do not seem to have what looks to be an instant "Response: ..." pingback within the same log.

The "Response:" part and below it now seems to be completely missing from the requests:

Requests (http/https request)
URL: https://api-3t.paypal.com:443/2.0/
Request: '
... '

Response: xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:cc="urn:ebay:apis:CoreComponentTypes" xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility" xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext" xmlns:ed="urn:ebay:apis:EnhancedDataTypes" xmlns:ebl="urn:ebay:apis:eBLBaseComponents" xmlns:ns="urn:ebay:api:PayPalAPI">xmlns="http://schemas.xmlsoap.org/ws/2002/12/secext" xsi:type="wsse:SecurityType">2020-06-04T20:06:57ZSuccessee59b375729ce59.0054492453187.00NM****************

I also just sent this information to PayPal to see if it is on there end of truncating anything in the process(?)

Here is my experience.

I'm on Godaddy dedicated server for 5 years.
I had the same exact issue as you starting April 17, 2020. Out of noway, I started getting failed order notification emails and then followed by processed notification emails. Soon I noticed my customers are getting failed orders when they are checking out. Because of this, they process the payment again and again which resulted in duplicate payments. It is a nightmare when they see multiple charges on their credit card statement and we have to go through each transaction to process the refund.
I have reached out to the Paypal merchant support team and cscart support (used 10 support credit). After a week of troubleshooting and testing, there were no issues found.
Understand the payment process.
1) When the customer clicked the Submit my order button the script sends POST request to PayPal server with payment information and expects to receive the result.
2) The script sends valid payment information and no response is given from PayPal, so the store script transmits the Failed status to the order.
3) After Paypal processed the payment it sends the notifications (IPN) back to the store and that's when the order status becomes Processed.
cscart has no issue, Paypal also has no issue, now we have narrowed down the issue to the hosting network. To investigate further, you will have to get a PCAP file from your hosting provider. We believe there is data packet loss somewhere in the route. I have no time to deal with it and I migrated to a new server after dealing with this issue for 3 weeks. (Yes, everything is back to normal once I have migrated to a new server)
As of today, I'm still in touch with Paypal support team and wait for their updates. They were other customers from Godaddy hosting having the same issue as me.
I hope my experience can help you decide your next course of action.

Thank you so much for all the information kickzz! This is very helpful. And yes, we too are on Godaddy and bought a big 5 year VPS about three years ago - pretty much been great the whole time until now. I'm assuming this is most likely the issue at this point. I will reach out to Godaddy and see if they have made any headway into finding a fix, since they obviously had issues already with it. So far, changing the "text_order_placed_error" error message in Administration->Languages->Translations has really helped with our customers - we did get the multiple order situation before we saw what was going on though - not fun at all... I will update as I know more to help others - I do really appreaciate the post!

Just a quick, possibly sloppy short-term patchwork job quesion until we can find the true cause and fix between Godaddy and/or PayPal (Pro setup) - Could this PayPal Pro file be modified temporary to produce an "Open" status instead of "Failed" until the 2nd (http/https request) comes back with the "all clear" payment processing that switches the orders from what used to be Open -> Incomplete then from Incomplete -> Processing?

CS-Cart 4.7.3

\public_html\app\addons\paypal\payments\paypal_pro.php

(lines 239-241 in our version)

if (empty($paypal_response['order_status'])) {

$paypal_response['order_status'] = 'F';
}
Change the 'F' to a 'O' - seems like this area is checking for an empty paypal response, which kindof seems like what we are getting with the "Response:" part being completely missing now from the 1st (http/https request). I hopefully have the right spot and file here for what I was thinking of doing...
Thanks!
Obviously this is not an ideal setting long term, but maybe in theory be ok for a little while since they really should not be coming back empty ever anyways(??) Or is this just completely flawed logic here when dealing with payment processor IPNs... ;)

Update 6-13-20: Changing the variable above to "O" (Open) seemed to do the trick, at least with a few test runs (I also cleared the Cache before testing just in case: Administration -> Storage -> Clear cache). I also tested mis-typing in the credit card number on an order and it seemed to bounce back as "Failed" like it is supposed to do and kept me on the checkout screen to correct the credit card information. I was then able to type in the correct numbers and it processed the order successfully. So, everything seems to be working as it should now (at least with the very limited number of tests done so far).

Something also interesting was noticed: on the test run of an order with the intentionally mis-typed credit card information, the "Response:" information WAS INCLUDED in the 1st (http/https request) pingback (that order was supposed to fail, which thankfully it did). I'm not sure what exactly the difference is, but somehow all of the information came back on that 1st request. However, the "Response:" information was still NOT included with the valid credit card tests, even with the second payment attempt to enter the correct credit card information after it failed back to enter in the right credit card information.

I hope this information is helpful to others. I'm still wanting to find the real cause and solution from our payment processor and/or host provider. For now at least, life goes on...