As the administrator of a VMware Horizon View virtual desktop infrastructure (VDI) for a county hospital, I’m frequently looking for applications that abuse infrastructure resources by consuming excessive CPU, RAM, or Disk.
Allscripts Pro EHR and Allscripts Pro PM are two applications that often create unnecessary challenges in our environment.
- Allscripts Pro PM requires Microsoft Data Execution Protection (DEP) to be disabled, which creates an unnecessary security risk
- Allscripts Pro EHR has a way of launching several times, with each hidden process consuming an ever-increasing quantity of CPU resources
While the DEP issue bothers me the most, today I’m going to address how runaway CPU processes of Allscripts Pro EHR on enough virtual desktops can bring your entire VDI infrastructure to its knees. I’ll use Windows PowerShell to identify the problem desktops and stop those runaway processes.
Allscripts does not support VDI
To be fair, Allscripts does not support VDI. They only support deploying their client to end users via a Citrix XenApp farm or by traditional physical PCs. Absurd, I know.
For four years I have pleaded for Allscripts to support VDI, VMware Horizon View, and VMware ThinApp application virtualization solutions. Sadly, they refuse. I think I now understand why.
What happens to one, happens to many
Our VDI infrastructure consists of 10 physical ESXi hosts that each run 80 to 100 virtual desktops.
If a single application on a single virtual desktop consumes an excessive amount of CPU resources, it’s no big deal.
But when a couple hundred virtual desktops run the same poorly designed application that is consuming an excessive amount of CPU, the physical hosts begin to run out of available CPU resources and all desktops begin to suffer performance issues.
Yikes! Too many concurrent desktops with high CPU.
Using PowerShell to discover the evil application
All of our virtual desktops are linked-cloned and have a consistent naming convention. If I use the VMware vSphere Client to view all virtual machines in the cluster and sort by CPU, I can quickly identify the virtual desktops that are consuming the most CPU resources.
Hmmm. Most of my problem VMs are in the two Allscripts desktop pools.
Looks like most of my CPU-consuming desktops are in our two Allscripts pools named VDI-ASPMLO and VDI-ASPMMO. I wonder what application is unique to those pools that might be causing the problem. I think I’ll use some PowerShell to figure it out.
This first PowerShell line creates a variable that includes the 200 virtual desktops that begin with the computer name “VDI-ASPM”.
PS C:> $TargetComputers = Get-ADComputer -Filter 'Name -Like "VDI-ASPM*"' | ForEach {$_.Name}
Next I want to retrieve all processes running on those computers that are consuming at least 1000 CPU resources, then sort them by amount of CPU consumed.
PS C:> Invoke-Command -ComputerName $TargetComputers -ScriptBlock { Get-Process | Where CPU -gt 1000 } -ErrorAction SilentlyContinue | Sort CPU -Descending | Format-Table -AutoSize
The result looks something like this:
CPU(s) Id ProcessName PSComputerName ------ -- ----------- -------------- 17,528.25 2292 EMR VDI-ASPMMO-027 16,352.27 4740 EMR VDI-ASPMMO-004 15,268.02 304 EMR VDI-ASPMMO-047 14,148.84 420 EMR VDI-ASPMLO-032 11,604.27 4180 EMR VDI-ASPMLO-058 10,463.36 5844 EMR VDI-ASPMMO-027 8,339.34 3148 EMR VDI-ASPMLO-061 8,243.88 3084 EMR VDI-ASPMLO-061 8,065.75 5352 EMR VDI-ASPMLO-104 7,948.77 4300 EMR VDI-ASPMLO-032 7,924.20 2032 EMR VDI-ASPMLO-061 7,293.23 2368 LibreOffice Writer VDI-ASPMLO-027 7,104.48 2636 EMR VDI-ASPMMO-055 7,085.19 5772 EMR VDI-ASPMMO-042 7,081.38 3728 EMR VDI-ASPMLO-100 7,052.19 1564 EMR VDI-ASPMMO-024 6,559.42 1360 EMR VDI-ASPMLO-029 6,391.80 5596 EMR VDI-ASPMLO-029 6,242.86 616 EMR VDI-ASPMMO-106 6,192.31 3604 EMR VDI-ASPMLO-032 6,062.73 332 EMR VDI-ASPMLO-052 5,875.50 2340 EMR VDI-ASPMLO-052 5,732.13 3924 EMR VDI-ASPMLO-029 5,659.23 2792 EMR VDI-ASPMLO-052 5,400.81 980 svchost VDI-ASPMMO-002 4,971.89 2504 EMR VDI-ASPMLO-052 4,946.58 3288 EMR VDI-ASPMMO-106 4,893.67 848 svchost VDI-ASPMMO-002 ...
I think I see a pattern. The EMR process belongs to the application Allscripts Pro EHR. It looks like there are many EMR processes that are consuming more than the normal amount of CPU. The average CPU process for the EMR application in our environment should be about 350 — not 1,000 to 17,000!
Why are there multiple EMR processes?
Is the user just doing something within EMR that is CPU intensive, or is something else going on?
To investigate, I first established a remote PowerShell session into one of the desktops that is consuming the most CPU resources: VDI-ASPMMO-027.
PS C:> Enter-PSSession VDI-ASPMMO-027
Now that I have a remote session, I wanted to take a closer look at the high-CPU consuming EMR process.
[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU -Descending | Format-Table -AutoSize
Here was the surprising result.
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
484 30 116928 47140 413 18,256.72 2292 EMR
617 36 151188 67428 489 11,197.84 5844 EMR
805 77 236652 61420 735 506.00 4272 EMR
694 70 206676 61904 623 117.70 2336 EMR
For an unknown and unnecessary reason, there appear to be four EMR processes running on desktop VDI-ASPMMO-027.
If I use VMware vSphere, I can open a remote virtual console to view the user’s desktop. Here’s what I find.
It’s frustrating that Allscripts would use the same icon for Pro PM and Pro EHR.
From left to right, we find several Internet Explorer tabs are open, one Misys Vision application open, one Allscripts Pro PM application open, one Allscripts Pro EHR application open, and one Allscripts Pro EHR Input Manager open. All of these processes can be accounted for, except for three of the CPU-intensive EMR processes.
It appears that the user is using only one EMR process and that three other EMR processes are consuming CPU resources but are not being used by the end user.
I’d like to kill EMR
Typically, the process that is consuming the most CPU resources is the process the end user is actively using. But I don’t think that is the case. I think the user is using the process (ID 2336) that is consuming the least amount of CPU resources.
To test, I’m going to kill all EMR processes except for the one that is consuming the least amount of resources.
If this PowerShell line returns all EMR processes sorted from least to first…
[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU | Format-Table -AutoSize
…then adding the “-Skip 1” command will filter out the EMR process that is consuming the least amount of CPU resources (e.g. ID 2336).
[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU | Select -Skip 1
Here is the result. Only three of the four EMR processes were returned.
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
746 77 234000 52444 731 567.00 4272 EMR
617 36 151188 67432 489 12,213.92 5844 EMR
478 30 116932 47136 413 19,265.73 2292 EMR
All that’s left to do is bet that the user is using the EMR process that is consuming the least amount of CPU resources (e.g. ID 2336), and kill the others. I do this by piping the above result to the Stop-Process commmandlet.
[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU | Select -Skip 1 | Stop-Process -Force
In each case, when I run this command while remotely viewing the user’s desktop, the end-user experiences no disruption in service. After killing the rogue Allscripts Pro EHR applications, performance for the end user dramatically improved.
Automatically kill rogue EMR processes
If you are not interested in manually finding and killing rogue EMR processes, here’s how to combine Group Policy, Scheduled Tasks, and PowerShell to kill Allscripts Pro EHR automatically.
Open Group Policy Management Editor and create a new policy named “PowerShell Stop-Process EMR if more than one is running” (or whatever you like). Navigate to Computer Configuration > Preferences > Control Panel Settings > Scheduled Tasks and create a new scheduled task named “PS Stop-Process EMR if multiple are running” that runs under the “NT AUTHORITY\System” account.
Create a new Trigger that begins early and runs every five minutes. I suggest you should “Delay task for up to (random delay): 1 minute” to avoid all desktops from running this task at the same time.
Under the Actions tab, I created two actions. Both are “Start a program” and both have Program/Script as “PowerShell”. My two Add Arguments are:
- Get-Process | Where ProcessName -eq “EMR” | Sort CPU | Select -Skip 1 | Stop-Process -Force
- Get-Process | Where ProcessName -eq “EMR” | Where CPU -gt 2000 | Stop-Process -Force
The first kills all but the least-consuming-CPU process. The second kills any EMR process that exceeds 2000 CPUs (which is way above average).
That’s it. All that is left to do is link your newly created Group Policy to the Active Directory Organizational Unit (OU) that contains the desktops you want to receive this policy.
In about 90 minutes, all desktops in the OUs you selected will have a new Scheduled Task that will run every 5 to 6 minutes that will kill all but the one EMR process that is consuming the least amount of CPU resources.
In our case, our ESXi hosts and end users were much happier to no longer have CPU resources needlessly consumed by a poorly designed application.