Incident Details:

Ticket ID: 51060 - Merit Error – Invoice in Hold Status - Zendesk Ticket
Created at: 4th July 2023 (02:55 PM IST)
Jira ID: 14061 - https://cognizantproducts.atlassian.net/browse/MER-14061
Priority: Normal
Issue Description: Users were checking the Hold status of the invoice when this error occurred.
Current Status: Pending with User Information

Whenever support team received Merit application issue, we need to get the below information before we reach out to L3 engineers.

Screenshot off the error – Collect the error screenshot from the user issue report email. If it is not provided. Please collect if from affected user

Who was the user that reported it? – Check the issue reported email and try to find out the username – For this issue Nakshatra is the affected user who has reported the issue
What was that user doing when it happened – Check the initial email which was reported by user and get the info from Description – In this issue user was just checking the Invoice hold status, Flagged the Invoice and clicked on Edit. Then user got this error and Merit got logged out.

Did it slow down or freeze? – As per users update - Its slowdown and kicking out

Did it affect just one user or all of them? - As per update from affected user – As of now 3 users in the team got same issue and error

The Latest log files - Collect the web logs from and SQL logs from both Production servers 18.19 and 18.43 and update them over the Jira case - 14061

What are the steps that must be taken care by support team after collecting the above details

Login into production server through Jump server and go to the Weblogs and check in which server user login by finding his/her name in the logs.

For example, if we find the affected users connected to Merit application through 18.19 server. Then login into the SQL and run the below query

Select * from MRT_SNCLAIMS_USR_FSTOTR where MRTUSERNAME = ‘Nakshatra Aradhye’

Now check the user ID in the below extract after running the above query in SQL.

User ID – U0071

Then go to the D:/Meritweb/TEMP folder in that we can see the username in the folder.

Example: U071/U71

After finding the user number in the D:/Meritweb/TEMP folder open it, if you find the file value having Zero KB ‘0KB’ then try to rename it in the end by giving _01 or _02
If you can edit and rename the file name, then there is no issue with that respective user. Please revert the file name as it was there before renaming
When trying to rename the file name, if you are not allowed to edit the file name and it’s giving you error, then there is an issue with the user’s task which he was trying to perform

Now go to the SQL Logs open the recent and with maximum KB high number as shown in below screenshot

Now open the SQL log file and try to find out the word like Invoice Hold or if user is facing audit issue, check with the word audit. If you find it in the log file, then we can follow the below checks

Example Invoice Hold - search for Invoice

Open the Merit not working checks excel sheet and collect all the data from both 18.19 and 18.43 servers as shown below:
CPU
Memory
Drive Space
Check if any service is erroring out, if yes collect the logs
Check the web logs, check the affected users connected to which server. Collect the web logs and web API trace logs from both servers (logs from 1 hour back issue reported)
Collect the CPU and Memory percentage from task manager as shown below screenshot

Now check the Drive Space in both servers, if you find any drive is full and red in colour. Try to delete the unwanted files from that drive.

Check the logs of all the automatic services in both servers and check if you have any errors. If you find any errors collected them in a Zip folder

Now check the Web logs from both servers and try to find out the server that affected users and their team is connected

Collect the web logs and web API trace logs from both the production servers (Collect from past one hour of issue reported time)

Collect the w3wp.exe thread count from task manager -> Details tab as shown in below screenshot

Now enter all the details collected from both the servers in the Merit not responding check list excel

Open IIS application from your search menu from both the servers and recycle only MERIT API in both servers.

Now open the Details tab from task manager again and collect the w3wp.exe thread count as shown in below

Note: Once we do Merit API recycle from IIS. We can see 3 w3wp.exe thread counts in details. But after some time, one thread will be disappeared

Now reply to user in Zendesk ticket that we have did IIS Reset in both servers and request them to access Merit application and check if it is working and all the affected users are able to do their day-to-day activities.

Open Jira parent ticket https://cognizantproducts.atlassian.net/browse/MER-14061 and update all the details, logs and screenshots collected in both servers and update in the Merit not working teams chat channel that all the logs and screenshots are updated in Jira – 14061

Now open performance monitor in both prod servers, Performance Monitor->Data collector set->user defined ->May 22 – 18.19 -> right click and stop it.

Go to C-Drive, Users, Mohammed Dadapeer, Desktop and copy the Performance monitor logs data

Now paste the performance monitor data file in Jump server C-Drive Temp folder by naming the file with today’s date and server number as shown in below figure.

Update in Jira 14061 and Merit not working chat channel that we have pasted the performance monitor data logs in Jump server and paste the path.

Now again go to C-Drive, Users, Mohammed Dadapeer, Desktop and delete the performance monitor data file

Open performance monitor, Data collector sets, User defined and right click on May 22 – 18.19 and click on start option as shown in below figure.

Now check with user via Zendesk ticket and get confirmation if the Merit application is working fine and close the case.

Related to