Further to Market Communication 2023/008 dated 8 January 2023, we are now able to report that the Messaging Service (XAG) performance issues are resolved.
Our teams have worked with external partners to deliver several performance tuning changes, along with firmware upgrades, which in turn have resolved issues resulting in system slowness and messaging delays. We are no longer seeing any delays in our monitoring metrics, and wait times on the queues have reduced to enable good message throughput.
We have also sought feedback from several customers who had experienced delays, with feedback confirming the service is now running to requirements.
Data Centre Migration To confirm, the planned Data Centre migration, currently scheduled for 17 - 22 February 2023, has not been affected by this long-running performance problem in any way. The Messaging Service (XAG) is already hosted in the new Data Centre and, post Data Centre migration, all systems will be centralised together enabling further performance improvements.
Posted Feb 06, 2023 - 10:46 GMT
Monitoring
Service Update 17th January
We have been working with our partners to deliver some planned performance changes to XAG, this has resolved several of the contributing factors to the system slowness. As a result the service is running faster, without the errors seen previously which required message replays from the support team.
We have also successfully migrated the service back to the Primary Data Centre this weekend. Following the migration, we are still investigating some minor performance delays which were present yesterday but today the service is running as expected with no delays.
We are still monitoring the service closely and continue to work on further improvements and stabilisation actions.
Posted Jan 17, 2023 - 12:57 GMT
Update
Market Communication Update 6 January 2023 Following our previous communication on the 16 December 2022 Ref: 2022/105, we have been working hard over the holiday period to complete the data purging and other actions to improve system performance. Whilst we have seen some improvement in performance throughput, we are still experiencing some delays with responses.
Next Steps We have identified specific workstreams of investigation and will continue working with our partners on root cause resolution. We expect to implement the next round of performance improvement actions over the next 7 days.
Summary of the impact: • Document Uploads, Downloads and Work Orders being submitted to the IMR using DRI messaging are currently subject to delays of up to 8 hours, and there are occasional failures • Submissions are reaching the IMR as normal and in good time, but confirmation of success/failure is being delayed. • Failed messages are being identified daily by DXC support teams and reprocessed after 7pm. • Failed Premium Work Order submissions cannot be replayed by DXC so customers will be contacted the following day to resubmit where necessary. • Writeback Claim Notify messages are subject to up to a 1.5 hour delay.
Support required from Customers: • Repeated resubmission of DRI messages (particularly Document Downloads) is compounding current performance issues. Customers are requested to avoid resubmission for 36 hours after initial submission to allow for support teams to replay failed submissions. Successful submissions will be responded to following a delay, but resubmissions will only extend that delay further. In more urgent cases, please raise an incident with the DXC Service Centre (service.centre@dxc.com). • Customers can check the IMR directly for the presence of documents which have not yet had a response. ECF2 Claims Awaiting Action can be used to view claim details directly for respective organisations. • In the event that you have a Premium Payment Warranty, Prompt Payment Discount, High Value Premium (Cash) or Notice of Cancellation that you require to be reviewed urgently, please contact either Lisa McEvoy (lmcevoy2@dxc.com) on 07867 140929 or Jean Tait (jtait4@dxc.com) on 07824 302583.
Should you have any questions regarding this communication, please email the DXC Service Centre service.centre@dxc.com or call 0870 380 0830
Posted Jan 09, 2023 - 11:13 GMT
Update
We are continuing to work through the performance improvement plan with maintenance activity which will require system downtime, this is scheduled over the coming weekend and bank holidays. We have communicated this on the Service Status page, maintenance updates.
This plan should start to show gradual service improvements until we are back to normal operating levels. However, during this week we have seen a larger than normal incoming volume which is compounding the issue and causing a message backlog, there is currently around a 1-hour delay on Writeback notify responses and a 24-30 hour delay in message processing. We are expecting to clear this backlog and data purging over the holiday period.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 23, 2022 - 11:03 GMT
Update
We are continuing to work through the performance improvement plan with maintenance activity which will require system downtime, this is scheduled over the coming weekends and bank holidays. We will communicate this on the Service Status page, maintenance updates.
This plan should start to show gradual service improvements until we are back to normal operating levels.
We are also drafting a Market Communication to provide further information.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 16, 2022 - 09:59 GMT
Identified
The system has continued to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently approximately a 1-hour delay with Writeback notify messages.
We are working on a recovery plan with the database which we believe is the major contributing factor to the performance issue, we are looking to verify this with maintenance activity planned over the coming days which should start to show gradual service improvements.
This maintenance will require small periods of downtime over night as well as over the weekend for the next couple of weeks as we work through this. We will communicate this on the Service Status page, maintenance updates.
We will provide further updates, but on a less frequent basis than twice daily as we work through this plan. We are also drafting a Market Communication to provide further information.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 08, 2022 - 17:43 GMT
Update
The system has continued to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently a 25-minute delay with Writeback notify messages.
We are still working on the previously identified issue with the database which may be contributing to the performance issue, we are still verifying this and will provide an update on a resolution plan asap.
We will provide a further update by Midday tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 08, 2022 - 13:45 GMT
Update
The system has continued to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently no delay with Writeback notify messages.
Today we have identified a specific issue with the database which may be contributing to the performance issue, we are testing further this evening and will be working on a forward fix plan over the coming days.
The work planned this evening requires a service outage from 23:00 - 04:00, we will also communicate this via the Maintenance update.
We will provide a further update by midday tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 07, 2022 - 17:29 GMT
Update
The system has continued to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently no delay with Writeback notify messages.
Today we have identified a specific issue with the database which may be contributing to the performance issue, we are testing further and will be working on a forward fix plan over the coming days.
We will provide a further update by midday tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 06, 2022 - 19:30 GMT
Update
The system has continued to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently a 30-minute delay with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required. We are also continuing testing some changes in our test environment which may help to resolve the issue.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 06, 2022 - 14:18 GMT
Update
The system has continued to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently no delay with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required. We are also continuing testing some changes in our test environment which may help to resolve the issue.
We will provide a further update by midday tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 05, 2022 - 18:22 GMT
Update
The system continues to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently a 5-minute delay with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required. We are also continuing testing some changes in our test environment which may help to resolve the issue.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 05, 2022 - 14:40 GMT
Update
The system continues to process incoming messages, however as the volume has increased during business hours there are delays to L3 Responses. The system will catch up with the backlog this evening. There is currently no delay with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required. We are testing some changes in our test environment over the weekend.
We will provide a further update at Midday on Monday.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 02, 2022 - 17:14 GMT
Update
The system continues to process incoming messages today, however as the volume has increased during business hours there are delays to L3 Responses. There is currently a small delay with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 02, 2022 - 12:35 GMT
Update
The system continues to process incoming messages today, however as the volume has increased during business hours there are delays to L3 Responses, the backlog will process overnight. There are currently no delays with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required.
We will provide a further update by Midday tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 01, 2022 - 17:10 GMT
Update
The system continues to process incoming messages today, however as the volume has increased during business hours there are delays to L3 Responses. There is also currently a 30-minute delay with Writeback notify messages as we are in peak period.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Dec 01, 2022 - 13:14 GMT
Update
The system has continued to process incoming messages today, however as the volume increases during the business day there have been delays to L3 Responses. There has also been delays of up to an hour with Writeback notify messages during peak periods.
Our support vendors have identified specific areas of focus within the systems which are being progressed in an effort to identify the cause and corrective actions required.
We will provide a further update at midday tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 30, 2022 - 19:33 GMT
Update
The system backlog has cleared overnight, and it continues to process ok today with the current volumes, although we do expect some delay to L3 Responses as the volumes increase. There are currently no delays with Writeback notify messages.
Our support vendors have identified specific areas of focus within the systems which are being progressed today in an effort to identify the cause and corrective actions required.
We will provide a further update at COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 30, 2022 - 09:13 GMT
Update
Although the system managed to catch up with the backlog overnight it continues to process today with some delay to L3 Responses, there is also delays with Writeback notify messages.
Our support vendors are still engaged, and we have investigative actions planned through this week to identify the cause and corrective actions required to fully resolve the running performance issue.
We will provide a further update at COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 29, 2022 - 14:12 GMT
Update
The application has continued to process incoming messages today but there is still some delay with L3 Responses. We are still assessing the system performance and further tuning is in progress.
Our support vendors are still engaged, and we have investigative actions planned through this week to identify the cause and corrective actions required to fully resolve the running performance issue.
We will provide a further update by 9:00 today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 28, 2022 - 21:33 GMT
Update
The application is functioning normally today, however there is still some delay with L3 Responses.
We are still assessing the system performance, and further tuning is in progress.
Our support vendors are still engaged and working to review and identify the cause and corrective actions required to fully resolve the running performance issue.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 28, 2022 - 12:35 GMT
Update
We have successfully moved the XAG application along with XDH and DFV across into our secondary Data Centre over the weekend. The system is fully operational and processing new requests.
We are assessing the system performance as the volumes increase into the business day.
Our support vendors are still engaged and working review and identify the cause and corrective actions required to fix the running performance issue.
We will provide a further update by Midday today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 28, 2022 - 09:33 GMT
Update
The system continues to process incoming messages at a good pace, however there may still be some delay to L3 responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue.
All set to move this service to the newly upgraded platform in our secondary Data Centre tomorrow as part of the ongoing remediation.
We will provide a further update by 09:00 HRS Monday.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 25, 2022 - 18:46 GMT
Update
The system continues to process incoming messages at a good rate this morning, however there may still be some delay to L3 responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue.
We also plan to move this service to the newly upgraded platform in our secondary Data Centre tomorrow as part of the ongoing remediation effort and system upgrade plan.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 25, 2022 - 08:28 GMT
Update
The system continues to process incoming messages at a good rate this morning, however there may still be some delay to L3 responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue.
We also plan to move this service to the newly upgraded platform in our secondary Data Centre at the weekend as part of the ongoing remediation effort and system upgrade plan.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 24, 2022 - 09:23 GMT
Update
The system continues to process incoming messages at a good rate, generally with good turnaround times for most requests however there may still be some delays in L3 responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue. We also have system upgrades in progress as part of the ongoing remediation effort.
We will provide a further update at 09:00 tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 23, 2022 - 16:35 GMT
Update
The system continues to process incoming messages at a good rate this morning, however there may still be some delay to L3 responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue. We also have system upgrades in progress as part of the ongoing remediation effort.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 23, 2022 - 10:57 GMT
Update
The system has processed the majority of yesterday's high volume overnight following the Data Centre Dress Rehearsal and is processing incoming messages ok this morning, however there will still be some delay to L3 responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue. We also have system upgrades in progress as part of the ongoing remediation effort.
We will provide a further update by midday today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 23, 2022 - 08:59 GMT
Update
The system has continued to accept and process new requests, although the volumes are around 35% higher than normal due to the two-day business outage as part of the Data Centre Dress rehearsal, therefore a backlog has built up today which has delayed message responses. We are monitoring and expect this backlog to clear overnight.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue. We also have system upgrades planned and agreed to start tonight as part of the ongoing remediation effort.
We will provide a further update by 09:00 tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 22, 2022 - 18:09 GMT
Update
The system is processing new requests, although the volumes are higher than expected due to the two-day business outage as part of the Data Centre Dress rehearsal, therefore we do see a backlog building which will delay message responses.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue. We also have system upgrades planned and agreed to start tonight as part of the ongoing remediation effort.
We will provide a further update by COB today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 22, 2022 - 12:25 GMT
Update
All the processing backlog was cleared overnight before services were re-enabled for the Market at 06:00 this morning. The system is processing new requests, but we are closely monitoring the system volumes as they increase into the business day.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the running performance issue. We also have system upgrades planned and agreed to start tonight as part of the ongoing remediation effort.
We will provide a further update by Midday
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 22, 2022 - 09:04 GMT
Update
We have been working to clear the processing backlog today while the system is still unavailable externally as part of the Data Centre Migration Dress Rehearsal, we expect this to complete in the next 2 hours, please do not resubmit messages.
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the performance issue. We also have system upgrades planned and agreed to start this week as part of the ongoing remediation effort.
We will provide a further update by 9:00 tomorrow morning.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 21, 2022 - 17:49 GMT
Update
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the performance issue.
We have completed Database optimisation & housekeeping tasks over the weekend in an effort to improve the performance for start of business tomorrow. We have also planned and agreed upgrade activity to start this week after the Data Centre Migration Dress Rehearsal.
We do still have some backlog that is processing today while the system is still unavailable externally as part of the Data Centre Migration Dress Rehearsal, please do not resubmit messages.
We will provide a further update by close of business today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 21, 2022 - 12:26 GMT
Update
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the performance issue. We do still have some backlog that will be processed on Monday when the system comes back online after the Data Centre Migration Dress Rehearsal, please do not resubmit messages.
We will provide a further update by close of business today.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 18, 2022 - 09:12 GMT
Update
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the performance issue. The service will be unavailable from 18:00 this evening due to the Data Centre Migration Dress Rehearsal, however we do expect to clear the message backlog overnight.
We will provide a further update by 09:00 tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 17, 2022 - 18:08 GMT
Update
We are still continuing investigations with our support vendors to identify the cause and corrective actions required to fix the performance issue. The service remains operational and is accepting new requests, but the L3 message responses will continue to be delayed until we resolve this.
We will provide a further update by 18:00
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 17, 2022 - 15:01 GMT
Update
We are continuing investigations with our support vendors to identify the cause and corrective actions required to fix the performance issue. The service remains operational and is accepting new requests, but the L3 message responses will continue to be delayed until we resolve this.
We will provide a further update by 15:00
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 17, 2022 - 12:01 GMT
Update
We have continued to work with our support vendors overnight and have identified where the issue is, we are now doing further analysis to identify the cause and corrective actions required.
We will provide a further update by midday
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 17, 2022 - 08:53 GMT
Update
We are continuing our investigations tonight as a priority with this issue. Please be assured that although there are currently message delays through the system, messages will not be lost. Please do not resubmit messages as this will further contribute to the performance problem.
We will provide the next update by 9:00 tomorrow morning.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 16, 2022 - 19:40 GMT
Investigating
We are continuing to investigate and are working on priority actions to resolve this performance issue as soon as possible.
We will provide the next update by 20:00
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 16, 2022 - 18:05 GMT
Update
We are continuing to investigate and are working on priority actions to resolve this performance issue as soon as possible.
We will provide the next update by 18:00.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 16, 2022 - 16:44 GMT
Update
Following a period of close monitoring of the XAG service we have started to see performance degradation again. This is impacting the DRI (Upload/Download, Notify & Search) service. There are also delays with L3 responses due to a larger than normal message queue as a result of the slowness.
Our Support Teams are taking priority action to investigate and resolve this performance issue.
Our next update will be issued no later than 16:00 hrs.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 16, 2022 - 14:26 GMT
Monitoring
The XAG message backlog has cleared overnight, and the system is performing well as expected this morning. We are continuing to closely monitor the service today.
DXC apologises for the inconvenience and the impact caused to our customers operations.
Posted Nov 08, 2022 - 08:41 GMT
Update
We are continuing to work on a fix for this issue.
Posted Nov 08, 2022 - 08:37 GMT
Identified
We have identified some stuck processes which we have cleared, and the backlog is now processing ok, we will be closely monitoring this overnight and will provide further updates before the start of business tomorrow.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 07, 2022 - 17:39 GMT
Investigating
We are experiencing slowness with XAG services which is impacting the DRI (Upload/Download, Notify & Search) service. There are also delays with L3 responses due to a larger than normal message queue as a result of the slowness.
Our Support Teams are taking priority action to investigate and fix - we will provide an estimate for full-service restoration as soon possible.
Our next update will be issued no later than 16:00 hrs.
DXC apologises for the inconvenience and the impact being caused to our customers operations.
Posted Nov 07, 2022 - 14:16 GMT
This incident affected: Repository (DRI Upload, DRI Download, DRI Search, DRI Notify) and Claims (Writeback - Claim Notify).