Sluggish performance within Windows 7/10 VDI session (Non network related!)

I’ve come across this now with multiple customers. Most of the time it’s showing up when people are moving to windows 10, after experiencing poor performance with W7 desktop VMs and as part of the migration process, they buy all new tin.

They experience sluggish responses when opening various apps and for example with IE, it’s painfully obvious that something is wrong. It will spike the CPU to a high percentage, take quite a while to open, then the CPU will drop and it’ll be OK for browsing until more tabs are opened and similar behaviour will occur.

When this occurs, I always ask if there’s any resource pools set, any QoS or similar on the storage and so on, but generally, I know there isn’t. vSphere monitoring will show that everything is OK, so will anywhere you look within Horizon and vROPs. I’ll ask if the physical hardware has power management/saving on within the BIOS and I usually get the following answers:

“Oh, I’m sure it’s been turned off, but I didn’t build the servers”

“Yep, no power management, vSphere says so.”

“I don’t know. How do you even do that?”

“I’ve asked and they say it has been.”

Every time. Every. Time. 

It can be difficult to definitely show that this is the source of the problem when everything you have access to says it all should be OK, plus people also have many different levels of experience with these things. I came across it about 7 years ago when the company I worked for had a mix of Dell and HP servers and being a PC gaming enthusiast, I was always trying to eke out as much performance of a PC as I could, so I tried to do the same with servers – no overclocking though! So power saving settings would be the first thing to get turned off!

So if you’re experience similar problems, with apps spiking the CPU, being sluggish, then the CPU dropping, check your physical hardware to make sure power management isn’t set to power saving, it’ll save you a whole load of heart ache! This also applies to Citrix as well.

If you need proof of this, or want to check it, there’s various tools, but I always check using Systrack – we have a tool as part of the suite called Resolve, which allows for in-depth analysis of specific machines (as well as the ability to compare to other machines/groups) and this will show straight away if a machine is being throttled, or has memory ballooning. Throttling can also show if a CPU is overheating and the BIOS throttles it back to avoid shutting down – many a place have thought they need new machines, or new CPUs, but no, they simply need to get those dusty fans cleaned out!

throttle

May show up a little bit too small for some screens, but what you’d see, is the CPU is throttled to 66%. The CPU usage is low, but due to the throttling, the Thread count and interrupt per seconds are high. The CPU should be 100% or even higher with some modern CPUs, but unless you’re really trying to save some power… You don’t want it lower than 100%

What’s also interesting, is when people go back to the older servers with Windows 7 on and realise that the poor performance throughout, was also due to the power management not being turned off… as a fair few manufacturers ship hardware with this as default…

Volatile Environment/Client information

I recently had cause to revisit viewing/editing the values in the volatile environment key, within the view desktop session, for a customer who needed to identify various pieces of information from the connecting endpoint. One of our Devs needed to be able to find this information and pass it through to Systrack to create some bespoke dashboards, using what is produced within the volatile environment key. There’s some really useful information in here if you’ve never had cause to go through it!

HKCU\Volatile Environment\x

Client System Information

Horizon View Events DB viewer

I had a customer today who needed to see what was going on in the Events DB, as they were having issues with disconnects, without having to pull the information out manually using SQL. I came across the following fling that came in really useful:

https://labs.vmware.com/flings/horizon-view-events-database-export-utility#summary

In these circumstances, you’ll probably still need to check out the actual Horizon View and PCOIP logs

https://kb.vmware.com/s/article/1027744

IGEL and Lakeside Systrack

Obviously I’ve been in a new role at Lakeside so I’ve been living/breathing/travelling/living that, as well as more citrix environments *cough cough* – but that doesn’t really matter if I’m honest, as systrack works equally well with both. Actually, it works brilliantly with everything you can throw it at. I’m still amazed at the pain points that systrack could have fixed for me over the past 5 years and aside from new customers I’m dealing with, I’m also seeing some familiar faces from the past that I know can make use of it!

This post is to show that IGEL, as a partner, has jointly developed a solution to monitor their IGEL OS with systrack – now, as anyone in EUC who has used thin clients or a thin OS will know, they’re a little black box unless you start digging into logs, which may not actually make much sense! Now you can use the analytics to show what’s going on, in conjunction with whatever VDI solution you’re always monitoring.

I know of projects using various zero/thin clients that still complain about latency, performance issues etc and they STILL have no monitoring or any insight whatsoever and the technical skills of the people on the ground are low, to to say the least!

IGEL and Lakeside

Flings of the day!

OK, if you aren’t au fait with Vmware Flings, go here:

VMware Flings

They’re basically community built apps/tools developed and provided for free to enhance many of the features within vmware products.

Some of them you’ll never use and some of them will become invaluable.

Today, I’d like to share with you, a few of my favourites.

ESX Embedded host client

This is something that is really, really handy. I’ve been in situations where networking has been spotty and even physical connections via KVM haven’t been feasible, either due to being at the forefront of standing up new hardware, stood in a freezing cold DC in the middle of the night, secretly taking a bite out of a protein bar and a swig from the tesco finest diet energy drink before one of the engineers sees the Food/Drink alarm going off and comes to tackle you to the ground… Or just because everything has gone wappy and you can’t connect.

This installs something akin to the web based vSphere client, directly on the host, meaning no reliance on vcentre, or when the vSphere client is having issues, or your laptop doesn’t let you change the VLAN on your NIC etc – I just found it was another backup in case of emergency (and we all need those!)

My other favourite for Horizon View is…

Horizon Tool Box

It does Auditing…It does Power Policy…But most importantly…REMOTE SUPPORT AND CONSOLE ACCESS! In all fairness, the ability to power on machines at certain times etc is really useful… The remote access leverages MS remote support as well as allowing console access to shadow.

I’ve been involved with many View deployments where one of the first concerns of IT staff is “How do we remote on to users desktops?” – and this answers that question. It’s quick and easy. Just remember to actually read the instructions (I know, I know, that can be a difficult to motivate yourself to do!). One of the main problems comes when people don’t enable RDP on the desktops – just an FYI there 😉

View Client Resizer

I use a 2k monitor with a 4k laptop screen and as you can imagine, apart from the usual resolution/display scaling issues, it can be a right pain when working on multiple view sessions – especially when I forget the set the res to not full screen across my monitors, leading to a black screen and a broken connection – this little exe means you can manage all the sessions you have open and is a handy little tool!

 

Horizon 7 Multi-site reference paper

Shameless link alert! ALERRTTTT!

This is a really, really good reference architecture paper – I think most of us within the space can setup and configure components such as workspace/Identity manager and Appvolumes, but not everyone gets the opportunity to really go to town across multiple sites with a view to building it up from the bottom – so this is where this paper comes in really handy. There’s everything you need for those situations where you’d have go hunting for the further information and also helps for getting buy in as to what exactly is needed as defined by a whitepaper. There are other configurations as solutions for multi site architecture (I know, I’ve done a few myself), but with this, you know exactly what you put in and what you’ll get out.

Horizon 7 Multi-site architecture

Can’t connect to VDI – thanks again f5.

f5 have the marketing reputation as the de facto load balancer for Horizon View. In my own experience of using the virtual appliance with View 6.2.2 (and using it for other services) I have to say I wouldn’t want to touch one or recommend one again.

Aside from the many woes that I’ve experienced and at the expense of everyones precious time, I saw a new issue the other day that could help anyone else troubleshooting View/f5 issues.

After a host appliance went down and then everything was resolved, none of the View connections that were being brokered by the f5 would work. Traffic would flow through to Security Servers and UAGs, Connection servers were accessible internally, but any connections that were reliant on the f5 (via the view Iapp), were terminating with ‘network error’, then ‘authentication error’. TCP dumps showed traffic, everything was up and there was also the possibility of storage and/or networking issues due the host failure.

But…

Everything else seemed fine. The quik view/ihealth of the f5 looked fine and everything else worked. After engaging f5 support, they suggested that they’d seen this before and noticed that for a few seconds, the f5 had gone from Active to Standby and back again. He simply suggested restarting VDI APM daemon.

Voila! Connections working again!

What was most annoying that nowhere said the service hadn’t started, or got stuck, it was only because f5 had seen the issue before that they pointed us towards that.

So, emergency over and I’m sharing the various services/daemons in case you encounter a similar issue.

 

Daemon Description Impact if not running Relevant log file   
acctd The RADIUS accounting daemon used by BIG-IP APM to send RADIUS accounting start and stop messages to external RADIUS servers. RADIUS accounting messages are not sent to external RADIUS servers /var/log/apm
aced The aced process provides RSA SecurID authentication functionality for BIG-IP APM’s access policy engine. RSA SecurID authentication fails /var/log/apm
apmd The apmd process executes access policy for a user session; this includes Authentication, Authorization, hosting Accounting, and Audit. It also provides an MPI interface, as well as support for access control protocol. No access policy enforcement for user session or any MPI-reliant processes, such as rewrite and websso /var/log/apm
antserver The antserver process allows Secure Web Gateway (SWG) to dynamically filter web content. No dynamic web content filtering /var/log/apm
dnscached The dnscached process provides DNS cache functionality to BIG-IP APM subsystems. BIG-IP APM DNS performance is impaired /var/log/apm
eam The eam process provides external access management for 3rd party identity integration, such as Oracle Access Manager (OAM) single sign-on (SSO). OAM SSO authentication fails /var/log/apm
eca The eca process provides the client-side NT Lan Manager (NTLM) authentication mechanism. BIG-IP APM is unable to authenticate using NTLM /var/log/apm
mdmsyncmgr The mdmsyncmgr process fetch MDM-managed endpoint list from MDM servers and stores it in local MySQL database. BIG-IP APM is unable to fetch MDM-managed endpoint list. /var/log/apm
nlad The nlad process establishes communication channels to the Domain Controller (DC) for NTLM authentication. No NTLM communication to backend DC /var/log/apm
omapd The omapd process provides the IF-MAP server implementation for SWG and AFM user identification. No user identification for SWG /var/log/omapd
rba The rba process provides support for client-side Kerberos authentication. No Kerberos authentication /var/log/apm
rewrite The rewrite process rewrites links in web content for Portal Access. Portal Access web links are not rewritten /var/log/rewrite
samlidpd The samlidpd process interacts with the mcpd process to automate SAML IdP connector creation. SAML IdP connector creation fails /var/log/saml_automation.log
urldb The urldb process categorizes incoming URLs for SWG. No SWG URL categorization /var/log/apm, /var/log/urldb-trace.log
urldbmgrd The urldbmgrd process downloads and indexes the URL categorization database for use by the urldb process. URL categorization for SWG is impaired /var/log/apm, /var/log/urldbmgr-trace.log
vdi The vdi process handles communication for XML-based clients and back-end systems such as Citrix and VMware View. Citrix integration and RDP access fails /var/log/apm
websso The websso process provides Single Sign-On (SSO) functionality for the BIG-IP APM system. SSO fails /var/log/apm

You can manage the services by using the TMSH utility – get putty’ed in (or similar) and follow this through:

  1. Log in to the tmsh utility by typing the following command:tmsh
  2. To stop, start, or restart a BIG-IP APM process, use the following syntax:<action> /sys service <process>

    In this command syntax, note the following:

    • <action> is the action to be performed, such as stop, start, or restart
    • <process> is the name of the BIG-IP APM process

    For example, to restart the eam process, type the following command:

    restart /sys service eam

Horizon View 7 Network ports/diagram

I always find this document to be really helpful when deploying not only View, but when other components start getting involved:

vRealize Operations for Horizon
VMware Horizon Client
VMware Identity Manager
VMware Unified Access Gateway
VMware App Volumes
VMware User Environment Manager
VMware vCenter Server
VMware ESXi
VMware AirWatch

And you either need a checklist of ports or need to precisely and politely inform the Network team/Firewall admin that you need some ports opening up.

Me: Can you not just open all the ports I just told you from that IP?

FW Bloke: No, I need some documentation to prove those are the ports I need to change for my change request.

Me: But…I sent you it weeks ago?

FW Bloke: I know, but I didn’t make the change so need to raise a new CR.

Me: So… Just add the document I sent across…Please?

FW Bloke: I can’t, I need a new one.

Me:… I hate you.

FW Bloke: I don’t care.

Horizon 7 Network Ports

 

 

vRealize For Operations – Unable to pair the broker agent for Horizon

I actually wrote some documentation on this for my team, when I encountered the issue. Then I lost it. Hence setting up a website to store everything I come across, on!

This issue came about myself after I upgraded the version of VROPS I had installed and I couldn’t for the life of me understand why the agent simply refused to connect. So after much messing around, I found the following vmware KB:

VMware vRealize Operations Manager for Horizon 6.2 Broker Agent fails to pair with the Horizon adapter (2140844)

Yep, they decided to leave out the necessary ports allowed through the firewall on the appliance. Thanks vmware!

If you aren’t comfortable using VI – which for the uninitiated, can be a nightmare, you can take a copy of the file (download it/FTP it off etc) and make the changes, then re-upload it.

Don’t forget the restart the firewall service – also when I rebooted the VROPs server once, it lost the settings, so rookie mistake or something more…sinister… Not sure, don’t care, just had to do this again! 😉