Actinic Payments server down

[cbarling]

You're right Paul, what can I say?

Believe me, you aren't the only one not satisfied with this.

Chris Barling
CEO, Actinic

[starleisure]

Yes I agree Paul someone somewhere needs a kick up the *rse.

I would say the server status page is a load of rubbish and doesn’t accurately reflect the status of the service we are being provided with.

I called Credit Call this morning myself and I wasn’t happy with the answers I was getting from the Actinic staff on the phone (the Actinic staff seemed very blasé about the whole situation)

Credit call said there was nothing wrong and that as far as they were aware nothing had been wrong in the last 24hours!!!

I was then called back by Actinic as credit call phoned them on my behalf as I told credit call I could not get any sense out of the Actinic staff and Actinic told me they woiuld email all customers once the system was running correctly again.

Guess what I am still waiting!!!

Can I get a refund on my support contract if I cancel it today on the unused bit ?

Simon

[Ben]

I'm sorry to hear that the Support team didn't come across as professional/informed or seemed to be blasé this morning. There certainly wasn't a blasé atmosphere in the office and the team were handling an unusually high call queue, understandably. There were two factors that contributed to not being as informed as we'd like:

1. The Actinic Payments services team usually receive alerts when there are problems and can inform the Support team (and hopefully fix the issue) before customers report it. This didn't happen today for technical reasons, so there was no awareness until we took your calls this morning.
2. Purely coincidentally, we were also experiencing issues with our hosted services, including our own website and Community. The Support team were also fielding calls reporting these issues and this probably caused some confusion.

Once again, I'm sorry if the communication from Support was not helpful. Hopefully, the above explains some of what was less than ideal circumstances.

[Darren B]

Perhaps AP should be seperated from general support

seperate phone numbers would be a good start, along with key personal identified within actinic that are informed directly of issues so they can relay information to there respective teams.

I dont know the inner workings of Actinic but unfotunately when something goes wrong no one really seems to have any idea that something is wrong. As bad as it sounds a vision of headless chickens seems to spring to mind. Obviously this is a perspective from my side of the fence and i just say it how it is, which normally gets me in trouble.

[TraceyHand]

Perhaps AP should be seperated from general support

I couldn't agree more.
Separate phonelines, separate staff.

[cbarling]

We will be reviewing this incident carefully. We do have different staff supporting the escalation of incidents for payments, although the same front line team support incoming calls for all products currently.

Obviously it's hugely disappointing that having had no downtime since November 2005, we've had two incidents within a few weeks of each other.

The issue on this occasion was due to the way things were monitored. There will obviously be hardware failures from time to time. You can't prevent that. And these should be flagged by monitoring - which actually in this case they were.

However, both ourselves and Creditcall have concentrated on monitoring the authorisation process, which in this case (and you could argue unfortunately), was working fine throughout. The problem was that one of the callback servers had stopped, and for ecommerce transactions, that is vital. We (Actinic) don't monitor those servers directly - hence our status page saying all was OK.

Anyway, I've already had a long discussion with the CEO of Creditcall about the incident. Obviously we are reviewing things and don't want this happening again. Both of us really, really want a highly reliable service.

It won't be any consolation, but this was totally different from the previous 43 minute outage, and caused by a completely separate scenario.

Chris

[Darren B]

It won't be any consolation, but this was totally different from the previous 43 minute outage, and caused by a completely separate scenario.

Chris

I dont think i am alone in thinking it was definately more than 43 minutes, it stopped then started then stopped last time messing around for over 2 hours

Maybe the front line support team should be the first to know of any issues as they seem to be the last.

[Darren]

Chris,

Would the problems effect the Fraud score at all. A client has an order, 3 declined payments, one processed, but Actinic is showing a red "Held" status and a score of +40.

Could this be attributed to the previous failed attempts?

Hi Rich

As I communicated with Mick yesterday, it looks like that card was entered a couple of times without the correct CVV2 number, that could affect the score.

Hope that helps

[cbarling]

Here's our monitoring records:

14/05/2009 11:30:00 Down
14/05/2009 11:31:00 Down
14/05/2009 12:07:00 Up (Down for 37 minutes)
14/05/2009 12:15:00 Down
14/05/2009 12:22:00 Up (Down for 7 minutes)

Creditcall's more detailed and frequent monitoring said 43 minutes total down time.

Chris

[wesleythorne]

we had 3 or 4 orders over night on monday that did not have payments allocated although payment was made when checking in the admin panel.
this started at 6.31pm 22.06.2009 and last one was 06.13am 23.06.2009

kind regards
wesley

[Rich Brady]

Hi Rich

As I communicated with Mick yesterday, it looks like that card was entered a couple of times without the correct CVV2 number, that could affect the score.

Hope that helps

Darren,

Thanks for the lightening fast reaction from you and 3rd man regarding this problem.

We had assumed that "held" status was due to the previous failed attempts, but wanted reassuring.

Cheers,

Rich

[cbarling]

Thanks Wesley, that's very useful.

Hopefully it was clear that the figures I put up were for the previous incident (14 May 2009).

The incident yesterday (22 and 23 June, 2009) was that for some customers, call backs didn't occur, even though payments were taken. I don't have exact times yet, but believe that it lasted from around 18.00 on 22 June to around 9.30 on 23 June, which fits in with what you said.

Chris

[Darren B]

Here's our monitoring records:

14/05/2009 11:30:00 Down
14/05/2009 11:31:00 Down
14/05/2009 12:07:00 Up (Down for 37 minutes)
14/05/2009 12:15:00 Down
14/05/2009 12:22:00 Up (Down for 7 minutes)

Creditcall's more detailed and frequent monitoring said 43 minutes total down time.

Chris

Hi Chris

I dont dispute these figures but i had contact from before this time by a customer telling me they had problems. Ben has contacted me and i have explained to the best of my fading memory what happened that day.

But in brief a customer contacted me saying they had problems, i ran two checks from different websites one works one did not. Perhaps this was a sympton of the forth comming failure. As we all know these intermitant faults might never be picked up, so credit calls records are correct. I dont expect monitoring to pick up every problem, i was merely making the point from this side of the fence (again) it alwasy looks different and sometimes is.

Yesterdays problems dont seem to have any effect on me and all in all i am happy with AP, simple communication to your 1st line support even to the point of - We are investigating an issue with AP at this moment intime would surely sound better than staff who dont seem to know. Just my observation but previous post show one got told you were aware and one got a confused response.

Just to clarify - i have not used support for a long time so cant comment on the current situation my observations are that the people making comment here are a new group of people but seem to be saying the same / similar thing

Darren

[Fairyglass]

I received an e-mail from Actinic a couple of days ago detailing what to do to check whether we had orders affected by the outage. The e-mail states:

"Check for orders which are missing the “User Reference” - these are the orders affected by the outage. First check to see if the transaction was successful*"

We've found three orders today in "pending payments" which have in fact been paid. These orders all HAVE a user reference, and were therefore ignored when we first checked. So it looks like, for us at least, that whether orders have a user reference or not is irrelevant- this means we have to check all orders in pending. Plus our customers should have received their goods by now, and of course have not.

[pmsupport]

"Check for orders which are missing the “User Reference” - these are the orders affected by the outage. First check to see if the transaction was successful*" We've found three orders today in "pending payments" which have in fact been paid. These orders all HAVE a user reference,

Ben, you are quite right, it appears the majority of our customers are reporting that the "user reference" is present and correct, so user reference being missing is not a good indicator of the order being effected.

The best way to check if the order was affected is to check orders which have downloaded into "Pending Payment Service provider" in Actinic during the time out of the outage, and check those payments were received successfully in the Actinic payments account administration area, as per the rest of the email you received.

We apologize if the email caused confusion.

Paul