Memory Control Script – Fine Tuning Process Memory Usage

In part 1 of this series I introduced a script which consists of over 900 lines of PowerShell, although over 20% of that is comments, that ultimately just calls a single Windows API, namely SetProcessWorkingSetSizeEx , in order to make more memory available on a Windows computer by reducing the working set sizes of targeted processes. This is known as memory trimming but I’ve always had issue with this term since the dictionary definition of trimming means to remove a small part of something whereas default memory trimming, if we use a hair cutting analogy, is akin to scalping the victim.

This “scalping” of working sets can be counter productive since although more memory becomes available for other processes/users, the scalped processes quickly require some of this memory which has potentially been paged out which can lead to excessive hard page faults on a system, when the trimmed memory is mapped back to the processes, and thus performance degradation despite there being more memory available.

So how do we address this such that we actually do trim excessive memory from processes but leave sufficient for it to continue operating without needing to retrieve that trimmed memory? Well unfortunately it is not an exact science but there are options to the script which can help prevent the negative effects of over trimming. This is in addition to the inactivity points mentioned in part 1 where the user’s processes are unlikely to be active so hopefully shouldn’t miss any of their memory – namely idle, disconnected or when the screen is locked.

Firstly, there is the parameter -above which will only trim processes whose working set exceeds the value given. The script has a default of 10MB for this value as my experience points to this being a sensible value below which there is no benefit to trimming. Feel free to play around with this figure although not on a production system.

Secondly, there is the -available parameter which will only trim processes when the available memory is below the given figure which can be an absolute value such as 500MB or a percentage such as 10%. The available memory figure is the ‘Available MBytes’ performance counter in the ‘Memory’ category. Depending on why you are trimming, this option can be used to only trim when available memory is relatively low although not so low that Windows itself indiscriminately trims processes. If I was trying to increase the user density on a Citrix XenApp or RDS server then I wouldn’t use this parameter.

Thirdly, there is a -background option which will only trim the processes for the current user, so can only be used in conjunction with the -thisSession argument, which are not the foreground window, as returned by the GetForeGroundWindow API, where the theory is that the non-foreground windows are hosting processes which are not actively being used so shouldn’t have a problem with their memory being trimmed.

Lastly, we can utilise the working set limit feature built into Windows and accessed via the same SetProcessWorkingSetSizeEx API. Two of the parameters passed to this function are the minimum and maximum working set sizes for the process being affected. When trimming, or scalping as I tend to refer to it as, both of these are passed as -1 which tells Windows to remove as many pages as possible from the working set. However, when they are positive integers, this sets a limit instead such that working sets are adjusted to meet those limits. These limits can be soft or hard – soft limits effectively just apply that limit when the API is called but the limits can then be exceeded whereas hard limits can never be breached. We therefore can use soft limits to reduce a working set to a given value without scalping it. Hard limits can be used to cap processes that leak memory which will be covered in the next article although there is a video here showing it for those who simply can’t wait.

Here is a an example of using soft working set limits for an instance of the PowerShell_ISE process. We start with the process consuming around 286MB of memory as I have been using it (in anger, as you do!):

powershell ise before trim

If we just use a regular trim, aka scalp, on it then the working set reduces to almost nothing:

powershell ise just after trimThe -above parameter is actually superfluous here but I thought I’d demonstrate its use although zero is not a sensible value to use in my opinion.

However, having trimmed it, if I return to the PowerShell_ISE window and have a look at one of my scripts in it, the working set rapidly increases by fetching memory from the page file (or the standby list if it hasn’t yet been written to the page file – see this informative article for more information):

powershell ise after trim and usage

If I then actually run and debug a script the working set goes yet higher again. However, I then switch to Microsoft Edge, to write this blog post, so PowerShell_ISE is still open but not being used. I therefore reckon that a working set of about 160MB is ample for it and thus I can set that via the following where the OS trims the working set, by removing enough least recently used pages, to reach the working set figure passed to the SetProcessWorkingSetSizeEx API that the script calls:

powershell ise soft max working set limit

However, because I have not also specified the -hardMax parameter then the limit is a soft one and therefore can be exceeded if required but I have still saved around 120MB from that one process working set trimming.

Useful but are you really going to watch to see what the “resting” working set is for every process? Well I know that I wouldn’t so use this last technique for your main apps/biggest consumers or just use one of the first three techniques. When I get some time, I may build this monitoring feature into the script so that it can trim even more intelligently but since the script is on GitHub here, please feel free to have a go yourself.

Next time in this series I promise that I’ll show how the script can be used to stop leaky processes from consuming more memory than you believe they should.

Advertisements

VMware integration added to Citrix PVS device detail viewer & actioner

You may be familiar with the script I wrote, previously covered here and available on GitHub here, that allows you to get a single pane view, either in csv or on-screen in a filterable and sortable grid view, of all your Provisioning Services devices together with information from Delivery controllers, such as machine catalogue and delivery group membership as well as registration and maintenance mode status. When using the grid view, you can select any number of devices to then get a GUI that allows operations like booting or shutting them down and removing from PVS and/or DDC.

When working at a customer recently I came across a number of VMs in VMware that were named using the XenApp worker naming scheme but weren’t being shown in the PVS or Studio consoles. Being the inherently lazy person that I am, I didn’t fancy deleting these individually in VMware and Active Directory, if they even existed in the latter, so I decided that it would be useful to add extra functionality to the script by getting it to add VMs that matched a specific naming pattern, so as not to pull in infrastructure VMs for example, that hadn’t already been pulled from Citrix PVS and DDC data. So I implemented this, utilising VMware  PowerCLI, and then also added a “Remove from Hypervisor” button to the action GUI so that these orphans can be removed in one go, including their hard drives.

To show VMs that don’t exist in either PVS or DDC in the grid view, simply add filters for where the DDC and PVS servers are empty.

show orphaned VMs

It will try to get AD account details too, such as the account creation and last logon dates and the description, in order to try and help you figure out what they are and if they have recently been used. They may not exist in AD, of course though, but that will be apparent in the data displayed, unless you don’t have domain connectivity/rights or the ActiveDirectory PowerShell module available.

This additional functionality is enabled by specifying the -hypervisors argument on the command line and passing it a comma separated list of your vCenter servers. If you do not have cached credentials (e.g. via New-VICredentialStoreItem) or pass through authentication working then it will prompt for credentials for each connection. You must have already installed the VMware PowerCLI package corresponding to the version of vSphere that you are using. There are examples of the command line usage in the help built into the script.

I then realised that in addition to the information already gathered that allows easy identification of devices booting off the wrong vDisk/vDisk version and devices that are overdue a reboot for example, that I could also pull in the following VMware details, again to help identify where VMs are incorrectly configured:

  • Number of CPUs
  • Memory
  • Hard drives (the size of each assigned)
  • NICs (the type of each assigned, e.g. “vmxnet3”)
  • Hypervisor

You can then sort or filter in the grid view or csv to uncover misconfigured VMs.

vmware info

The downside to all this extra information is that there are now up to 42 (a coincidence!) columns of information to be displayed in the grid view but, unfortunately, versions of PowerShell prior to 5.0 can only display a maximum of 30 columns. Csv exports aren’t affected by this limitation though. As I am often heard saying to my kids, it’s better to have something and not need it rather than need something and not have it – you can remove columns in the grid view, by right clicking on any column header, or in Excel, or whatever you use to view the csv. If this will impact you, consider upgrading as there are a whole load more PowerShell features that you’re missing.

To restrict what VMs are returned by the Get-VM cmdlet, you will probably need to use the -name argument together with a regular expression (aka regex) which will only match your XenApp/XenDesktop workers. For instance, if your VMs are called CTX1001 through CTX1234 and also CTX5001 onwards then use something like the following:

'^CTX[15]\d{3}$'

The -name parameter is also used to restrict what PVS devices are included so you can just include a subset if you have, say, a sub-naming convention to name development XenApp servers differently to production ones, e.g. CTXD1234 versus CTXP4567, which will make it quicker.

To check that a regular expression you build matches what you expect before you run the script, there are on-line regex checkers available but I just use PowerShell. For instance, typing the following in a PowerShell session will display “True”:

'CTX1042' -match '^CTX[15]\d{3}$'

I also decided to add a progress indicator since, with hundreds of devices, it can take several minutes to collect all of the relevant data although data is cached where possible to minimise the number of remote calls required. This can be disabled with -noProgress.

If you do have orphaned VMs and you want to remove them, highlight them in the grid view and then click “OK” down in the bottom right hand corner. Ctrl-A can be used to select all items in the grid view. This will then give you the action GUI (ok, not the prettiest user interface ever but it does work!):

pvs device actioner gui vm

where you can power off the VMs if they are on and then delete them from the hypervisor and from AD, all without having to go to any product consoles assuming that you are running the script under an account which has the necessary rights. When you quit this GUI, the devices that you originally selected in the grid view, will be placed into the clipboard in case you need to paste them into a document, etc.

Using -save, -registry and, optionally, -serverset will also save/retrieve  the server(s) specified by -hypervisors to the registry. This means that you don’t have remember server names every time you run the script – handy when you deal with lots of different customers like I do.

Be aware that it needs to be run where the PVS and DDC cmdlets are available so I would recommend installing on a dedicated management server which does not host the PVS or DDC roles so you can also use those consoles, and others you install, on there so that you don’t risk degrading performance of key infrastructure servers. Also, don’t forget VMware PowerCLI and the AD PowerShell module (part of the RSAT feature).

Whilst I have checked the operation of this script as much as one man in West Yorkshire can, if you use it then you do so entirely at your own risk and I cannot be held responsible for any unintentional, or intentional, undesired effects. Always double, and even triple, check before you delete anything!

Having said that, I hope it is as useful for you as it is for me – for a reporting and status tool, I use it daily (weekends included!).

Memory Control Script – Reclaiming Unused Memory

This is the first in a series of articles which describes the operation of a script I have written for controlling process memory use on Windows.

Here we will cover the use of the script to trim working sets of processes such that more memory becomes available in order to run more processes or, in the case of Citrix XenApp and Microsoft RDS, to run more user sessions without having them use potentially slower page file memory (not to be confused with “virtual” memory!). The working set of a process is defined here which defines it as “the set of pages in the virtual address space of the process that are currently resident in physical memory”. Great, but what relevance does that have here? Well, what it means is that processes can grab memory but not necessarily actually need to use it. I’m not referring to memory leaks, although this script can deal with them too as we’ll see in a later article, but buffers and other pieces of memory that the developer(s) of an application have requested but, for whatever reasons, aren’t currently using. That memory could be used by other processes, for other users on multi-session systems, but until the application returns it to the operating system, it can’t be-reused. Queue memory trimming.

Memory trimming is where the OS forces processes to empty their working sets. They don’t just discard this memory, since the processes may need it at a later juncture and it could already contain data, instead the OS writes it to the page file for them such that it can be retrieved at a later time if required. Windows will force memory trimming if available memory gets too low but at that point it may be too late and it is indiscriminate in how it trims.

Ok, so I reckon that it’s about time to introduce the memory control script that I’ve written, is available here and requires PowerShell version 3.0 or higher. So what does it do? Trims memory from processes. How? Using the Microsoft  SetProcessWorkingSetSizeEx  API. When? Well when do you think it should trim the memory? Probably not when the user is using the application because that may cause slow response times if the memory trimmed is actually required such that it has to now be retrieved from the page file via hard page faults. So how do we know when the user (probably) isn’t using the application. Well I’ve defined it as the following:

  1. No keyboard or mouse input for a certain time (the session is idle)
  2. The session is locked
  3. The session has become disconnected in the case of XenApp and RDS

As in these are supported/built-in but you are obviously at liberty to call the script whenever you want. They are achieved by calling the script via scheduled tasks but do not fret dear reader as the script itself will create, and delete these scheduled tasks for you. They are created per user since the triggers for these only apply to a single user’s session. The idea here is that on XenApp/RDS, a logon action of some type, e.g. via GPO, would invoke the script with the right parameters to create the scheduled task and automatically remove it at logoff. In it’s simplest form we would run it at logon thus:

.\Trimmer.ps1 -install 600 -logoff

Where the argument to -install is in seconds and is the idle period that when exceeded will cause memory trimming to occur for that session. The scheduled tasks created will look something like this:

trimmer scheduled tasks

Note that they actually call wscript.exe with a vbs script to invoke the PowerShell because I found that even invoking powershell.exe with the “-WindowStyle Hidden” argument still causes a window to very briefly popup when the task runs whereas this does not happen with the vbs approach as it uses the Run method of WScript.Shell and explicitly tells it not to show a window. The PowerShell script will create the vbs script in the same folder as it exists in.

The -logoff argument causes the script to stay running but all it is doing is waiting for the logoff to occur such that it can delete the scheduled tasks for this user.

By default it will only trim processes whose working sets are higher than 10MB since trimming memory from processes using less than this probably isn’t worthwhile although this can be changed by specifying a value with the -above argument.

So let’s see it working – here is a screenshot of task manager sorted on descreasing working set sizes when I have just been using Chrome.

processes before

I then lock the screen and pretty much immediately unlock it and task manager now shows these as the highest memory consumers:

processes after

If we look for the highest consuming process, pid 16320, we can see it is no longer at the top but is quite a way down the list as its working set is now 48MB, down from 385MB.

chrome was big

This may grow when it is used again but if it doesn’t grow to the same level as it was prior to the trimming then we have some extra memory available. Multiply that by the number of processes trimmed, which here will just be those for the one user session since it is on Windows 10, and we can start to realise some savings. With tens of users on XenApp/RDS, or more, the savings can really mount up.

If you want to see what is going on in greater detail, run the script with -verbose and for the scheduled tasks, also specify the -logfile parameter with the name of a log file so the verbose output, plus any warnings or errors, will get written to this file. Add -savings to get a summary of how much memory has been saved.

Running it as a scheduled task is just one way to run it – you can simply run it on the command line without any parameters at all and it will trim all processes that it has access to.

In the next article in the series, I’ll go through some of the other available command line options which gives more granularity/flexibility to the script and can cap leaky processes.

 

Finding Citrix PVS or Studio orphans

I recently released a script, which I use almost daily when working with PVS servers at version 7.7 or higher since that’s when a native PowerShell interface appeared, that cross references Citrix Provisioning Services device information to Delivery Controller and Active Directory. See here for the original post. This allows me to easily and quickly health check and potentially fix issues that would otherwise need a lot of manual work and jumping around in various consoles. Whilst the script could already easily identify devices that only existed in PVS, by filtering in the grid view or Excel where the DDC (Desktop Delivery Controller) field/column is empty, I realised I could extend the script to identify devices that exist on Delivery Controllers, so visible in Studio, but don’t exist in PVS. You may of course expect to find some devices in PVS but not present on a DDC, and hence Studio, such as devices used for updating vDisks via booting in maintenance mode since you won’t want to make those available via StoreFront or Receiver.

Once you have the on screen grid view or csv file open in Excel (or Google Sheets), show PVS devices not present on DDCs by simply filtering where the “DDC” column is empty, by clicking on the “Add Criteria” button. To show devices which are known to a DDC, so visible in Studio, but not in PVS, filter where the “PVS Server” column is empty.

pvs orphans

This of course assumes that you have specified the correct server names for your DDC and PVS servers via the -ddcs and -pvsservers options respectively. There’s no need to specify multiple servers for each if they share the same SQL database; only if they use different ones such as you might have for completely separate test and production environments. Comma separate them if you do specify multiple servers.

If you’ve got a mixture of PVS and MCS (or manual) machine catalogues then it will only display machines found on the DDCs you specify which are in PVS linked machine catalogues, unless you specify the -provisioningType parameter.

I’ve also added to the actions menu so that these potential orphans can then be removed from PVS or DDC if you select them in the grid view and then click “OK”.

remove orphans

I’ve also sneaked in a potentially handy feature where you can save the PVS and DDC servers to the registry so that you don’t have to specify them on the command line ever again (on that machine at least). This helps me, if nobody else, as I use the script at many different customers and I can’t always remember their specific server names, or sometimes specify the wrong ones. Save with -save and use these saved values with -registry, and an optional server set name via -serverSet so you can have different sets of servers, e.g. pre-production and production.

For example:

& '.\Get PVS device info.ps1' -ddcs ddc001 -pvsServers pvs001 -save

So next time you just need to run:

& '.\Get PVS device info.ps1' -registry

They are stored in HKCU so are per-user.

The script, amongst others, is available on GitHub here. It has to be run on a machine which has both the PVS and DDC PowerShell cmdlets available; such as one with PVS and Studio consoles installed. Also the ActiveDirectory PowerShell module, particularly if you want to include AD group membership information via the -ADGroups option.

Citrix StoreFront Log Viewer Tool

Have you ever had the need to debug StoreFront? I have on a couple of occasions and it wasn’t the easiest debugging exercise I’d ever undertaken unfortunately. Changing the logging level is easy enough with the PowerShell cmdlet Set-DSTraceLevel. For example, run the following on a StoreFront server to enable verbose logging (see https://support.citrix.com/article/CTX139592):

Add-PSSnapin Citrix.DeliveryServices.Framework.Commands

Set-DSTraceLevel –All –TraceLevel Verbose

Which will restart the Citrix services, having updated web.config files, and various log files will be created in “C:\Program Files\Citrix\Receiver StoreFront\Admin\trace”. It will also cause debug statements to be produced which can be picked up with tools like SysInternals dbgview.

The problem, in my experience, is that reading the log files, of which there are many, can be a bit of a chore. The log files are almost XML format but they are not fully compliant as they don’t have a top level node, since presumably adding this would have a performance hit. Even if you can get them into XML, working with them in XML isn’t particularly easy although that may depend on what XML tool you use (I would typically use Internet Explorer since that’s all I can rely on being on customer machines where I don’t want to start installing third party software).

Fortunately, PowerShell comes to our rescue (yet again) since it’s very easy in scripts to make this almost XML be properly formed so that this can then be quickly parsed and each log record output to a csv file or an on-screen grid view where filtering and/or searching can then take place.

The script is available here and can extract logs from multiple StoreFront servers, by accessing the logs via their C$ share, and splicing them together based on the time of each event. You can either specify a starting and ending date/time range via –start and –end respectively, specify –sinceBoot to include all entries since the last boot of each server or use –last with a number and a specifier such as ‘d’ for days, ‘m’ for minutes and ‘s’ for seconds so “-last 8h” means “in the last eight hours”. For example run the following to see all errors in the last two hours on the two specified StoreFront servers and display on-screen in a filterable grid view:

& '.\parse storefront log files.ps1' -computers storefront01,storefront02 -last 2h -subtypes error

If you have verbose logging enabled but only want to show warning and error entries then specify “-subtypes error,warning” since the default is to include all entries, including verbose ones.

Clicking “OK” at the bottom right of the grid view will copy any selected log lines into the clipboard, e.g. for web searches or logging with Citrix.

Specifying the -Verbose option gives information on what logs are being parsed, from which servers and for what time ranges.

Finally, don’t forget to change the StoreFront logging level back to something like “Error” rather than leaving it at “Verbose” as that is unlikely to help performance!

Citrix PVS device detail viewer with action pane

here

I find myself frequently using the script I wrote, see here, to check the status of PVS devices and then sometimes I need to perform power actions on them, turn maintenance mode on or off or maybe message users on them before performing power actions (I am a nice person after all). Whilst we can perform most of those actions in the PVS console, if you are dealing with devices across multiple collections, sites or PVS instances then that can involve a lot of jumping around in the PVS console. Plus if you want to change maintenance mode settings or message logged on users then you need to do this from Citrix Studio so you’ll need to launch that and go and find the PVS devices in there.

So I decided to put my WPF knowledge to work and built a very simple user interface in Visual Studio and then inserted it into the PVS device detail viewer script. Once you’ve run the script and got a list of the PVS devices, sorted and/or filtered as you desire, select those devices and then click on the “OK” button down in the bottom right hand side of the grid view. Ctrl-A will select all devices which can be useful if you’ve filtered on something like “Booted off latest” so you only have devices displayed which aren’t booting off the latest production vdisk. This will then fire up a user interface that looks like this, unless you’ve run the script with the -noMenu option or hit “Cancel” in the grid view.

pvs device actioner gui

All the devices you selected in the grid view will be selected automatically for you but you can deselect any before clicking on the button for an action. It will ask you to confirm the action before undertaking it.

pvs device viewer confirm

If you select the “Message Users” option then an additional dialog will be shown asking you for the text, caption and level of the message although you can pass these on the command line via -messageText and -MessageCaption options.

pvs device viewer message box

The “Boot” and “Power Off” options use PVS cmdlets rather than Delivery Controller ones since the devices may not be known to the DDC. “Shutdown” and “Restart” use the “Stop-Computer” and “Restart-Computer” cmdlets respectively and I have deliberately not used the -force parameter with them so if users are logged on, the commands will fail. Look in the window you invoked the script from for errors.

You can keep clicking the action buttons until you exit the user interface so, for instance, it can be used to enable maintenance mode, message users asking them to logoff, reboot the devices when they have logged off or you have had enough of waiting for them to do so and then turning off maintenance mode, if you want to put it back in to service after the reboot.

I hope you find it as useful as I do but note that you use the script entirely at your own risk. It is available here and requires version 7.7 or higher of PVS, XenApp 7.x and PowerShell 3.0 or later where those consoles are installed on the machine where you will run the script from (so that their PowerShell cmdlets are available too).

 

Update to Citrix PVS device detail viewer

In using the script, introduced here, at a customer this week, I found a few bugs, as you do, and also added a few new features to make my life easier.

In terms of new features, I’ve added a -name command line option which will only show information for devices that match the regular expression you specify. Now don’t run away screaming because I’ve mentioned regular expressions as, contrary to popular belief, they can be straightforward (yes, really!). For instance, if you’ve got devices CTXUAT01, CTXUAT02 and so on that you just want to report on then a regex that will match that is “CTXUAT” – we can forget about matching the numbers unless you specifically need to only match certain of those devices.

Another option I needed was to display Citrix tag information since I am providing a subset of servers, using the same naming convention as the rest of the servers, where there are tag restrictions so specific applications only run off specific servers. Using tags means I don’t have to create multiple delivery groups which makes maintenance and support easier. Specify a -tags option and a column will be added with  the list of tags for each device, if present.

However, adding the -tags option was “interesting” because the column didn’t get added. A bug in my code – surely not! What I then found, thanks to web searches, is that versions of PowerShell prior to 5 have a limit of 30 columns so any more than that and they silently get dropped. The solution? Upgrade to PowerShell version 5 or if that’s not possible and you want the tag information, remove one of the other columns by changing the $columns variable. Yes,  30 columns is a lot for the script to produce but I decided it was better to produce too much information, rather than too little, and then let columns be removed later in Excel or the grid view.

I also found a bug, yes really, where if the vDisk configured for a device had been changed since it was booted then it would not be identified as not booting off the latest. That’s fixed so remember you can quickly find all devices not booting off the latest production version of the vDisk or booting off the wrong vDisk by filtering on the “Booted off Latest” column:

booted off latest

The script is still available here (GitHub? never heard of it :-))