Opening an Incident

(1) Incident

Once a case has been through the AIC process and is Identified as an incident, the steps below should be followed.

• (2) Open an Escalation Incident Ticket (Jira)
  • The naming convention used for raising Jira escalations should follow the format below.
    • [Severity] –[Client] - [product] – [date] – [Short Description]
  • All investigation details must be added to the Jira, including a detailed description of the issue.
  • The scope of the issue must be shared with the team. The items listed below need to be added to the Jira description. If we have no information for some of the fields in the list below add “NA.”
    • Impact on the client:
    • Impact to end client:
    • Examples and screenshots:
    • Logs:
    • Temporary fix criteria:
    • When the issue started to happen:
    • Have any released been recently deployed to Production:
    • Have we provided a defect?
  • Product managers and relevant engineering and operations team members should be tagged in the escalation. Tagging can be done by using “@” and then the names of the users you would like to tag.
  • SLA details should be shared on the ticket.
  • Link the Support case to the Escalation, link the Jira to the Zendesk case.
  • Irrespective of all the Clients and Products for all Jira’s we create for Engineering Team. We need to tag Tech Consultancy Team in CC.

• (3) Support opens an Incident channel
  • A channel is opened on Microsoft teams, the “Teams” functionality should be used.
  • Name the channel using the following naming convention.
    • ZD Ticket number – Severity – Short description.
  • Add all team members tagged on the Escalation and support management.
  • Add an opener as the first comment on the incident channel.
    • This should include Client name, product, time, and date the incident began and the scope of the issue. Also add a brief description of the issue and a link to the Jira and the Zendesk case.
  • Get a link to the channel you have just created and share it on the Jira and the Zendesk case that have been created for the Incident.
  • Add a comment saying support has received a new incident and post the Jira for it in the Application Support teams Channel

• (4) Open an Incident bridge.
  • Request that a call be opened on the incident channel created in step (3)
  • Should the team not wish to open a call, make sure you drive the incident on the Incident channel created in step 3

• (5) Client Communications
  • After the initial acknowledgement in the AIC step and steps 2, 3 and 4 are completed, the team should contact the client via Zendesk with an update.
    • Updates should include the actions that you have taken so far. Let the client know you have raised this as a critical escalation and that we and our teams are working to resolve the incident with absolute urgency. Add a well worded apology. “We apologies for any disruption this may be causing you and the team”. Let the client know that you will keep them up to date with any added information as the team works to resolve the case. Give an update every hour
  • CC’ the account manager into the conversation on Zendesk. Internal account manager details can be found HERE by clinking on the client in the Clients section.
  • If unsure of any wording or detail, ask for a communication to be double checked by a pier.

• (6) Incident resolved
  • Confirmation in writing on the incident channel must be given for an incident to be declared as resolved.
  • Once declared as resolved the exact time of the resolution needs to be recorded.
  • Ask the team involved to update the Jira with any items that may be outstanding. This could be a patch or an item for development that relates to the incident. Anything related to the incident should be linked to this Jira. Make sure you add yourself as a watcher to any of the Jira cases that may have been linked.

• (7) Client Communications
  • Once an Incident is resolved, we will need to tell the client, please include the detail below.
    • The incident has now been resolved.
    • A short very high-level explanation of what the issue was.
    • A note to let them know we are monitoring the system to ensure stability.
    •  A comment to let the client know the team will conduct root cause analysis and provide an incident report asap.
    • Let the client know the support case will now be closed as the incident has now been resolved. Be sure to mention that the escalation remains open, even though the support case is closed and that the team will reach out about any pending actions.
  • You have linked the Zendesk case to the Jira and you have linked the Jira to any items that may be outstanding. You have set yourself as a watcher in every Jira related to the incident. Please be sure to pay attention to comments and work done on those Jira’s. You must reach out to the reporter of an incident with outstanding details.

• (8) RCA, Incident Report, Account management.
  • Root Cause Analysis to be completed by Development or Infrastructure.
    • Request for an RCA should be made by the support team after an incident is declared resolved on the incident channel.
  • Once the root cause analysis is completed an incident report should be completed by the support team. This is extremely sensitive and account managers and relevant stakeholders will need to review the report and approve it before it is sent out.
    • Account management to reach out to the client and deliver the incident report.
  • Root cause analysis and incident reports should be added to the Jira that was created for the Incident.
  • Any preventative measures related to this incident should be followed up by the support team. These should be marked as completed on the Jira for the incident.
  • All incidents should be added to the Incident Tracker which can be found at the bottom of the Client Support Page

• (9) Escalation into L4 (Emergency escalation)
  • Should a case not be getting the required attention, support will escalate to the Engineering Manager.
  • Incidents requiring escalation should be raised to the head of support. This will then be presented to the head of delivery for approval of emergency escalation.
  • Once approved for escalation, the engineering manager will be informed of the situation.
  • A comment on the Jira case should be added. This will state the case has been escalated to L4, a comment on the incident channel should be added for visibility.
  • The Engineering Manager will then assign appropriate resources and monitor the situation.