Disclosing the Cause of an Issue to a Customer
-
I have a couple of hard drives on my desk that I ventillated with an AR-15. :)
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass." - Dale Earnhardt, 1997I should do that this weekend with my hard drive that my last company sent away to get fixed. They couldn't even recover anything, and were willing to spend a few $k's on it.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
-
Can you elaborate on what you mean by a 'non-transactional' financial system? I'm not sure I get what you meant.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
Ennis will likely provide a better explanation than I, but if messages are lost somehow, then you have a non-transactional system. Consider an accounting system that books two ledger entries for every accounting action. If one entry fails while the second one succeeds, your ledger system will never balance. Both entries are deemed a logical transaction and both must succeed for the accounting action to be correct. In your specific case, the loss of an acknowledgement message, should have resulted in a time-out error that would have rolled back the acceptance of the trade. That would have made it more transactional. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
-
It sounds like you work for a brokerage firm or trading house. Such a question is a matter for the compliance department: I would strongly recommend referring the question to them. If you say the wrong thing to an angry client, your firm could be in a lot of hot soup. Provide an explanation to Compliance, and let them pass it on to the client. Complaints of any kind should never, ever be handled by anyone else.
Yes, the company itself is a brokerage. I should have pointed out that I am not responsible for the error, the fix, or providing the explanation. I have been on the long nasty e-mail chain and it got me thinking about what this Level 1 support guy might send out if under enough pressure from the client.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
-
I'm looking to get some input from you guys on this matter. Last night we had a message get lost. Meaning, we never sent a trade acknowledgement to a customer, but booked the order in our system and executed it. The customer has been furious all morning about it. I did not investigate or work on the fix, but the problem is now fixed we are saying. However, the customer is demanding a full explanation of why it happened and how we fixed it. They won't start trading and it seems they might pull the plug if we don't supply it. I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
Do what I do when I forget birthdays. Set back your PC clock, compose lovely email. Send it. Correct PC clock again. Tell the person on the other side that their email must be slow or broken and to check the sent date, if they wonder why it only reached them now :)
-
Ennis will likely provide a better explanation than I, but if messages are lost somehow, then you have a non-transactional system. Consider an accounting system that books two ledger entries for every accounting action. If one entry fails while the second one succeeds, your ledger system will never balance. Both entries are deemed a logical transaction and both must succeed for the accounting action to be correct. In your specific case, the loss of an acknowledgement message, should have resulted in a time-out error that would have rolled back the acceptance of the trade. That would have made it more transactional. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
The actual issue was that when the production account for this customer was set up, it was mapped to the demo book keeping system, which sends the acknowledgement. The failure happened there, and the actual trade engine thought everything was fine.
Chris Meech wrote:
In your specific case, the loss of an acknowledgement message, should have resulted in a time-out error that would have rolled back the acceptance of the trade.
But now you got me thinking. How is one to know that our ack message was lost, without waiting to see. Our messages are shot out to our banks as soon as we get them, indifferent to whether the customer gets acknowledged, if we don't, our performance is unacceptable. You can't exactly rollback an accepted trade after the fact.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
-
The actual issue was that when the production account for this customer was set up, it was mapped to the demo book keeping system, which sends the acknowledgement. The failure happened there, and the actual trade engine thought everything was fine.
Chris Meech wrote:
In your specific case, the loss of an acknowledgement message, should have resulted in a time-out error that would have rolled back the acceptance of the trade.
But now you got me thinking. How is one to know that our ack message was lost, without waiting to see. Our messages are shot out to our banks as soon as we get them, indifferent to whether the customer gets acknowledged, if we don't, our performance is unacceptable. You can't exactly rollback an accepted trade after the fact.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
It also sounds like there is procesing going on at different locations such as the banks, clients and your own. All of which makes it even more difficult to establish logical transactions around business events. Some times the best that you can do is perform batch or end of day reconciliation processing that makes you aware of the breaks. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
-
It also sounds like there is procesing going on at different locations such as the banks, clients and your own. All of which makes it even more difficult to establish logical transactions around business events. Some times the best that you can do is perform batch or end of day reconciliation processing that makes you aware of the breaks. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
That is definitely the case and we have to do a daily report that is sent to a 3rd party agency for compliance, or we can be fined. The customer caught the issue rather quickly, but with a multiparty system like this the blame gets passed all over. Since support was confidence it was a setup error, and was able to track down the lost message, they disclosed it to the customer, even though it was an embarrassing error, at least it wasn't the code.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
-
That is definitely the case and we have to do a daily report that is sent to a 3rd party agency for compliance, or we can be fined. The customer caught the issue rather quickly, but with a multiparty system like this the blame gets passed all over. Since support was confidence it was a setup error, and was able to track down the lost message, they disclosed it to the customer, even though it was an embarrassing error, at least it wasn't the code.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
Was this trade the first one booked for the account? That's when these types of data configuration errors are usually caught. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
-
Was this trade the first one booked for the account? That's when these types of data configuration errors are usually caught. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
I can't verify, but it seems like it. Nothing like going live at 2am local time.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
-
Yes, the company itself is a brokerage. I should have pointed out that I am not responsible for the error, the fix, or providing the explanation. I have been on the long nasty e-mail chain and it got me thinking about what this Level 1 support guy might send out if under enough pressure from the client.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
That's what I figured, and situations like this are one of the reasons my own firm had me get the Series 7 and Series 66 license: to make damned sure that I knew the rules and had some legal/regulatory protection if I messed up. Drawing on my own training, I would suggest: * Create a special folder and copy Every. Last. One. of those emails into it, so they do not accidentally get deleted. Don't forget to also get the emails in your Sent folder. If anything has already been permanently deleted, try to get them from your server backups. * Make hard copies of every email, even when they overlap because of Reply To. What you want to do is create an unbroken chain of correspondence. * If you spoke to the client on the telephone, go to your recording device manager and get recordings of those calls. * Write up your own summary of events, including the resolution. Be as technical and detailed as you can, but also provide a non-technical summary suitable for the bosses. * If you know of anyone else who has been involved, ask them to do the same. * Bring all of this to your compliance department. If it will take a while to assemble everything, notify them now and get everything put together post haste. * Do not contact the client again. If he contacts you, your only response is, "You will need to speak to N. N." Notify the compliance department of this contact, and make sure you provide emails, print outs, recordings and/or transcripts as appropriate. These are the procedures that I had hammered into me, presumably for a reason. Think of this as kicking the problem upstairs and making it someone else's headache :rolleyes:
-
That's what I figured, and situations like this are one of the reasons my own firm had me get the Series 7 and Series 66 license: to make damned sure that I knew the rules and had some legal/regulatory protection if I messed up. Drawing on my own training, I would suggest: * Create a special folder and copy Every. Last. One. of those emails into it, so they do not accidentally get deleted. Don't forget to also get the emails in your Sent folder. If anything has already been permanently deleted, try to get them from your server backups. * Make hard copies of every email, even when they overlap because of Reply To. What you want to do is create an unbroken chain of correspondence. * If you spoke to the client on the telephone, go to your recording device manager and get recordings of those calls. * Write up your own summary of events, including the resolution. Be as technical and detailed as you can, but also provide a non-technical summary suitable for the bosses. * If you know of anyone else who has been involved, ask them to do the same. * Bring all of this to your compliance department. If it will take a while to assemble everything, notify them now and get everything put together post haste. * Do not contact the client again. If he contacts you, your only response is, "You will need to speak to N. N." Notify the compliance department of this contact, and make sure you provide emails, print outs, recordings and/or transcripts as appropriate. These are the procedures that I had hammered into me, presumably for a reason. Think of this as kicking the problem upstairs and making it someone else's headache :rolleyes:
That's good policy, and our broker and support people do that. They also have both licenses. I never come into contact with customers, so never needed to take the license.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
-
That's good policy, and our broker and support people do that. They also have both licenses. I never come into contact with customers, so never needed to take the license.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
I never deal with clients; I rarely even deal with our reps beyond "I've locked myself out of the website" and "What anti-virus software do you recommend?" Even so, my company has written policies that all personnel must read and answer questions on every six months.
-
I never deal with clients; I rarely even deal with our reps beyond "I've locked myself out of the website" and "What anti-virus software do you recommend?" Even so, my company has written policies that all personnel must read and answer questions on every six months.
Oh bugger - you just reminded me I need to go to my HR folder and check what mandatory courses are outstanding, may your day be better than mine is now going to be!
Never underestimate the power of human stupidity RAH
-
I'm looking to get some input from you guys on this matter. Last night we had a message get lost. Meaning, we never sent a trade acknowledgement to a customer, but booked the order in our system and executed it. The customer has been furious all morning about it. I did not investigate or work on the fix, but the problem is now fixed we are saying. However, the customer is demanding a full explanation of why it happened and how we fixed it. They won't start trading and it seems they might pull the plug if we don't supply it. I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
Get one of the staff who will NEVER speak to customers, to call the customer, and explain that it was their, human, error (they powered down the Swichinator 2000 when they were reclamming the brogal banch), offer their appologies then pass the call onto their account manager, who should explain that the person is now being sacked for their incompetence
MVVM# - See how I did MVVM my way ___________________________________________ Man, you're a god. - walterhevedeich 26/05/2011 .\\axxx (That's an 'M')
-
Get one of the staff who will NEVER speak to customers, to call the customer, and explain that it was their, human, error (they powered down the Swichinator 2000 when they were reclamming the brogal banch), offer their appologies then pass the call onto their account manager, who should explain that the person is now being sacked for their incompetence
MVVM# - See how I did MVVM my way ___________________________________________ Man, you're a god. - walterhevedeich 26/05/2011 .\\axxx (That's an 'M')
That reminds me of something, long ago: We apologise for the fault in the subtitles. Those responsible have been sacked. ... We apologise again for the fault in the subtitles. Those responsible for sacking the people who have just been sacked, have been sacked. ... The directors of the firm hired to continue the credits after the other people had been sacked, wish it to be known that they have just been sacked.
"Dark the dark side is. Very dark..." - Yoda ---
"Shut up, Yoda, and just make yourself another toast." - Obi Wan Kenobi -
I'm looking to get some input from you guys on this matter. Last night we had a message get lost. Meaning, we never sent a trade acknowledgement to a customer, but booked the order in our system and executed it. The customer has been furious all morning about it. I did not investigate or work on the fix, but the problem is now fixed we are saying. However, the customer is demanding a full explanation of why it happened and how we fixed it. They won't start trading and it seems they might pull the plug if we don't supply it. I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
Have you checked out the The Bastard Operator From Hell-Style Excuse Server[^]? ;) Or just pick one from the full list of BOFH excuses.
-
I'm looking to get some input from you guys on this matter. Last night we had a message get lost. Meaning, we never sent a trade acknowledgement to a customer, but booked the order in our system and executed it. The customer has been furious all morning about it. I did not investigate or work on the fix, but the problem is now fixed we are saying. However, the customer is demanding a full explanation of why it happened and how we fixed it. They won't start trading and it seems they might pull the plug if we don't supply it. I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
- You can refuse to tell the customer what happened. 2) You can tell the customer what happened. 3) You can escalate. You admit you don't want to do 2 (I concur). You admit that if you do 1, the customer might leave (I would, and I would tell everyone else to leave). That means you must escalate. In the customer's terms, they want to know what happened, why it happened, how it got fixed. It all boils down to one question - HOW DO I KNOW THIS WON'T HAPPEN AGAIN? Simply saying it won't happen again won't work. You (the company) promised him that WHEN HE OPENED THE ACCOUNT. So promising it again (after the screw up) just makes you look bad. Frankly speaking, given the cause of the error, changes need to be made so it never happens again -- changes in policy that are far above your pay grade. Again, it needs to escalated.
-
I'm looking to get some input from you guys on this matter. Last night we had a message get lost. Meaning, we never sent a trade acknowledgement to a customer, but booked the order in our system and executed it. The customer has been furious all morning about it. I did not investigate or work on the fix, but the problem is now fixed we are saying. However, the customer is demanding a full explanation of why it happened and how we fixed it. They won't start trading and it seems they might pull the plug if we don't supply it. I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
wizardzz wrote:
I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
You already had a bug and admitted fault, so they have all the trash they need. Sounds like they're wanting their trust in your system to be restored -- not just that this particular bug won't repeat, but that they won't experience other kinds of (potentially worse) bugs with your system -- and giving you an opportunity to do that. Its a no-brainer to me, take it, but make sure to address both of those concerns or they'll probably "pull the plug" anyway (I would).
patbob
-
I'm looking to get some input from you guys on this matter. Last night we had a message get lost. Meaning, we never sent a trade acknowledgement to a customer, but booked the order in our system and executed it. The customer has been furious all morning about it. I did not investigate or work on the fix, but the problem is now fixed we are saying. However, the customer is demanding a full explanation of why it happened and how we fixed it. They won't start trading and it seems they might pull the plug if we don't supply it. I for one, strongly feel that we shouldn't disclose how an issue was resolved after we have accepted responsibility. In my industry, someone who is a customer, is often times a competitor, too. It isn't worth it to give them any trash they can use against us to woo potential customers.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
You're dealing with millions of dollars, on behalf of your customers. And if you're handling transactions for multiple brokers, that can easily be hundreds of millions of dollars. It's common in the financial processing sector that customers: * will not accept the latest version of your software without THEIR testing. * will demand full accountability when errors or problems occur - with detailed tracking & auditting * will switch away to another processing firm if you're not cost-competitive or reliable * your customers are often your competitors It's been this way for years. As someone else said, you don't have to give them the full internal description of the problem. But you should be honest and state the basic problem and that it was fixed. Unless you have an ongoing reliability or performance problem, your reputation won't be ruined. All of the financial processing (firms) fuck up once in awhile.
-
I should do that this weekend with my hard drive that my last company sent away to get fixed. They couldn't even recover anything, and were willing to spend a few $k's on it.
"I have a theory that the truth is never told during the nine-to-five hours. " — Hunter S. Thompson My comedy.
We had that happen to our production server a few years ago - a RAID-1 system and the people in the area where the server was located neglected to mention the noise that was coming out of the machine until it was too late. And they weren't doing the nightly backups. Cost the company $6K to recover most of the data and the system was down for two weeks (good thing a few still knew how to do things manually) :sigh:
Steve _________________ I C(++) therefore I am
