Update to Citrix PVS device detail viewer

In using the script, introduced here, at a customer this week, I found a few bugs, as you do, and also added a few new features to make my life easier.

In terms of new features, I’ve added a -name command line option which will only show information for devices that match the regular expression you specify. Now don’t run away screaming because I’ve mentioned regular expressions as, contrary to popular belief, they can be straightforward (yes, really!). For instance, if you’ve got devices CTXUAT01, CTXUAT02 and so on that you just want to report on then a regex that will match that is “CTXUAT” – we can forget about matching the numbers unless you specifically need to only match certain of those devices.

Another option I needed was to display Citrix tag information since I am providing a subset of servers, using the same naming convention as the rest of the servers, where there are tag restrictions so specific applications only run off specific servers. Using tags means I don’t have to create multiple delivery groups which makes maintenance and support easier. Specify a -tags option and a column will be added with  the list of tags for each device, if present.

However, adding the -tags option was “interesting” because the column didn’t get added. A bug in my code – surely not! What I then found, thanks to web searches, is that versions of PowerShell prior to 5 have a limit of 30 columns so any more than that and they silently get dropped. The solution? Upgrade to PowerShell version 5 or if that’s not possible and you want the tag information, remove one of the other columns by changing the $columns variable. Yes,  30 columns is a lot for the script to produce but I decided it was better to produce too much information, rather than too little, and then let columns be removed later in Excel or the grid view.

I also found a bug, yes really, where if the vDisk configured for a device had been changed since it was booted then it would not be identified as not booting off the latest. That’s fixed so remember you can quickly find all devices not booting off the latest production version of the vDisk or booting off the wrong vDisk by filtering on the “Booted off Latest” column:

booted off latest

The script is still available here (GitHub? never heard of it :-))

Advertisements

Citrix Provisioning Services device detail viewer

Whilst struggling to find some devices in the PVS console that I thought that I’d just added to a customer’s PVS server via the XenDesktop Setup wizard, I reckoned it should be relatively easy to knock up something that would quickly show me all the devices, their device collection, disk properties and then also cross reference to a Citrix Delivery Controller to show machine catalogue, delivery group, registration state and so on. Note that I’m not trying to reinvent that wheel thing here as I know there are already some great PVS documentation scripts such as those from Carl Webster (available here).

What I wanted was something that would let me quickly view and filter the information from multiple PVS servers, such as development and production instances. Whilst PowerShell can easily export to csv and you can then use Excel, or Google Sheets, to sort and filter, that is still a little bit of a faff so I use PowerShell’s great Out-GridView cmdlet which gives you an instant graphical user interface with zero effort (not that using WPF in PowerShell is particularly difficult!) which can be sorted and filtered plus columns you don’t want can be removed without having to modify the script.

The script takes two parameters which it will prompt for if not specified as they are mandatory:

-pvsServers

-ddcs

Both take comma separated lists of PVS servers and Desktop Delivery Controllers respectively although you can just specify a single server for each. If you’ve got multiple PVS servers using the same database then you only need to specify one of them. Ditto for the DDCs.

You can also specify a -csv argument with the name of a csv file if you do want output to got to a csv file but if you don’t then it will default to a filterable and sortable grid view.

Some hopefully useful extra information includes “Booted off latest” where devices with “false” in this column are those which have not been booted off the latest production version of their vDisk so may need rebooting. There’s also “Boot Time” which you can sort on in the grid view to find devices which are overdue a reboot, perhaps because they are not (yet) subject to a scheduled reboot. Plus you can quickly find those that aren’t in machine catalogues or delivery groups or where there is no account for them in Active Directory. You can also filter on devices which are booting off an override version of a vDisk which may be unintentional.

The script is available here (yes, I must start using Github!) and requires version 7.7 or higher of PVS since that is when the PowerShell cmdlets it uses were introduced. Run it from somewhere where you have installed the Citrix PVS and Studio consoles, like a dedicated management server – I’m a firm believer in not running these on their respective servers since that can starve those servers of resource  and thus adversely affect the environment. Ideally, also have the Active Directory PowerShell module (ActiveDirectory) installed too so that the device’s status in AD can be checked.

I’ve just picked out the fields from PVS, Delivery Controllers and AD that are of interest to me but you should be able to add others in the script if you need to.
Continue reading “Citrix Provisioning Services device detail viewer”

Taking and resizing a screen shot in an Outlook email directly

As an IT Consultant, I frequently have to send Outlook emails containing screenshots. In the old, old days, I’d press the PrintScreen key, paste into Paint (I will miss you old friend as you are seen to be retired), select what I needed and then paste into the email. Then along came the Snipping Tool which made things a little easier but it still means going away from your email and then coming back. Oh, and don’t go on about third-party tools – I stick with what’s built-in then I know I’ve got it wherever I go.

It seems to be a little known/used fact that you can add a screen capture button directly to the quick access toolbar (QAT) in all recent Microsoft Office products. So I added this to the QAT but what I then found myself doing for pretty much every screenshot was to shrink it which either involved dragging the resize points around or quite a few clicks and key presses in the email message which kind of cancelled out the ease of getting the image in there in the first place.

Taking VBA code I found here and here (I was going to write it from scratch but these were almost exactly what I needed so why reinvent this wheel thing?), I ended up with the following VBA code, placed in the ThisOutlookSession object (hit Alt F11 to bring up the Outlook VBA editor):

Public Sub Shrinker()
 Const wdInlineShapePicture = 3
 
 If TypeName(ActiveWindow) = "Inspector" Then
   If ActiveInspector.IsWordMail And ActiveInspector.EditorType = olEditorWord Then
     If ActiveInspector.WordEditor.Application.Selection.InlineShapes.Count > 0 Then
       For Each wrdShp In ActiveInspector.WordEditor.Application.Selection.InlineShapes
         If wrdShp.Type = wdInlineShapePicture Then
           wrdShp.ScaleHeight = wrdShp.ScaleHeight - 10
           wrdShp.ScaleWidth = wrdShp.ScaleWidth - 10
         End If
       Next
     End If
   End If
 End If
 
 Set wrdShp = Nothing

End Sub

You can then add a button to the QAT in an email compose window that calls this code:

outlook email macro commands QAT

Notice the “Screen Clipping” button which is added thus:

add screen clipping to qat

Which will then give you these toolbar buttons in your email compose window:

outlook email screen capturing

You can then click the screen capture icon when needed which will hide the email compose window, give you cross hairs to select the region you need and once released it will be pasted into your email, selected, which will come back into focus:

outlook email screen captured

If you need to resize then click on the QAT button you assigned to the Shrinker macro which will scale it down by 10% each time you click it. If you want to grow it again, just hit Ctrl Z to undo. If you need to enlarge rather than shrink, duplicate the macro and put +10 instead of -10 and assign this macro to the QAT too.

I hope this helps save you lots of time which it has for me. Don’t forget to save the macro – the easiest way is to close Outlook and click Yes when it asks whether to save ThisOutlookSession.

 

Scripted Reporting & Alerting of Citrix Provisioning Services Boot Times

Citrix PVS, formerly Ardence, is still one of my favourite software products. When it works, which is the vast majority of the time if it is well implemented, it’s great but how do you tell how well it is performing? If you’ve enabled event log generation for your PVS servers thus:

pvs event log server

then the Citrix Streaming Service will write boot times of your target devices to the application event log:

pvs boot event

So we can filter in the event log viewer or use the script I’ve written which searches the event log for these entries and finds the fastest, slowest, average, median and mode values from one or more PVS servers and optionally creates a single csv file with the results. A time range can also be specified, such as the last 7 days.

The script lends itself to being run via a scheduled task as it can either email the results to a specified list of recipients or it can send an email only when specific thresholds are exceeded, such as the average time being greater than say 2 minutes.

For instance, running the following:

& '.\Get PVS boot time stats.ps1' -last 7d -output c:\boot.times.csv -gridview

Will write the boot times to file, in seconds, for the last seven days on the PVS server where you are running the script. It will also display the results in a sortable and filterable gridview and output a summary like this:

Got 227 events from 1 machines : fastest 21 s slowest 30 s mean 25 s median 25 s mode 26 s (39 instances)

Or we could run the following to query more servers and send an email via an SMTP mail server if the slowest time exceeds 5 minutes in the last week:

& '.\Get PVS boot time stats.ps1' -last 7d -output c:\boot.times.csv -mailserver yourmailserver -recipients someone@somewhere.com -slowestAbove 300 -computers pvsserver1,pvsserver2

The script has integrated help, giving details on all the command line options available, and can be run standalone or via scheduled tasks.

The script can be downloaded from here. Full help is built in and can be accessed via F1 in PowerShell ISE or Get-Help.

 

Petya: disabling remote execution of psexec

This morning in response to reports of an outbreak of yet more malware I wrote a quick blog post on one way to stop the SysInternals psexec from being allowed to execute by using the Image File Execution Options registry key mechanism – see here for that post.

My technique was, and still is sound, but @RennJohnny correctly pointed out that if the psxec.exe executable was renamed then my approach would not work. I therefore set about finding another way to stop psexec from running for those who don’t (yet) have security software in place to stop the exploit. Note that you need to be running it on a system where you are an administrator and have the same rights on the remote system to be attacked. As the disposable virtual machine that I used for my testing is not on a domain, I passed explicit credentials to psexec for my testing – I don’t believe Petya operates this way but my solution will still work.

So when psexec is used to run something on a remote system, it works by creating a new service executable called psexesvc.exe which is embedded within the original psexec.exe file. This is copied to the Windows folder on the remote machine via the admin$ default share (hence why you need to be an admin to get psexec to work remotely). It then creates the PSEXESVC service with this, now local, executable, starts it and then runs the specified command.

What I found was that even when I copied psexec.exe to another file name, the file produced and copied to the remote system was still called psexesvc.exe. This is what happens when you run the copied psexec.exe and tell it to invoke a command on a remote machine:

psexesvc allowed

 

On that remote system we can then see this has been created in the services registry key:

psexesvc registry

How do we stop it? I reckon that the easiest way is to use good old Image File Execution Options (IFEO) mechanism again but this time we create the key “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\PSEXESVC.exe” and in there create the REG_SZ value called “Debugger” and set it to “svchost.exe”. Now when we try and run psexec to execute on the system where we just patched the registry, this happens instead:

psexesvc blocked

What happened? Well when the Service Control Manager (SCM) on the remote machine was asked to start the PSEXESVC service, it started the psexesvc.exe process but the IFEO entry we created caused it to run svchost.exe instead but as that can’t be used as a standalone service, it failed to start so SCM reported this to psexec which is the error we see above. You will also get this in the System event log of the remote system:

psexesvc eventlog.PNG

One way to roll it out to all your computers is to put the above registry value into a Group Policy Preference that applies to those computers.

You can also create a dummy psexesvc.exe file in your Windows folder, remove all permissions and change the owner to, say, TrustedInstaller, and that will also prevent it from running.

I hope this helps some of you and stay safe (and don’t run routinely with admin privileges!).

Petya: easily disabling access to psexec

So it seems there is yet another piece of ransomware in the wild which is more sophisticated than Wannacry as it uses multiple attack vectors. I have read that one of these is once a machine is infected that it uses the great SysInternals utility psexec.exe, and possibly Microsoft’s command line WMI utility wmic.exe, to spread further. Whilst there are products, like Ivanti Application Control, formerly AppSense Application Manager, that can be used to blacklist these, if you haven’t got those products today then you need a way of stopping these attack vectors. One way would obviously be to delete wmic.exe, or remove NTFS permissions to it, but you can’t do that for psexec since presumably the malware is either downloading it or has it embedded within its payload.

Here is where we can use the little known Image File Execution Options (IFEO) registry key to put a temporary, or permanent, block on these, or any other executables, so they cannot run. We can either get them to fail silently or run a script informing the users that their machine is infected.

IFEO has been around for 20+ years – in fact the very first version of the software that I wrote that became AppSense Application Manager used this feature. It is great for debugging but is also an attack vector itself for malware as it can be used to disable security programs with the same technique. One mitigating factor is that because the key is in HKLM, you need administrative rights to write to it and we don’t let any user run with administrative rights when they are running web browsers, email, Office products, etc. now do we?

So what do I do? In its simplest form, create the key “psexec.exe” in “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options” and then create a REG_SZ value called “Debugger” and set it to “svchost.exe” as below:

ifeo psexec

Job done! If you run psexec.exe before adding the above settings you’ll see something like this:

psexec before

But once the registry key and value are in place we get this instead:

psexec after

Attack vector thwarted! But aren’t we then running a service because svchost.exe is being run instead? No, service executables can’t just be run from the command line, they need to be invoked via the Service Control Manager (SCM), so this invocation of svchost.exe just fails silently.

Do the same with a wmic.exe key and that’s both supposed attack vectors blocked for now.

You can also set the value to run a script although you need to ensure that the script itself cannot be compromised using file system security. For instance, if I set the Debugger value to this:

"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -file c:\temp\infected.ps1

Note the use of the full path so we don’t accidentally run a malicious powershell.exe (unless your whole system is compromise!) and if using PowerShell rather than cmd.exe ensure that your PowerShell execution policy allows the script to run.

My c:\temp\infected.ps1 script contains just  this:

$null = [System.Reflection.Assembly]::LoadWithPartialName('Microsoft.VisualBasic')

$null = [Microsoft.VisualBasic.Interaction]::MsgBox( "Your system is infected - call IT now!" , 'OkOnly,SystemModal,Critical' , $MyInvocation.MyCommand.Name )

then when psexec is run a user will get this popup:

infected

How do I roll this out quickly in an enterprise? Unless you’ve already got something that can push out registry settings to multiple computers then I would suggest that Group Policy Preferences is one of the easiest ways of achieving this.

When your security software is up to date then you might want to delete the key/value unless you need to run psexec.exe for other reasons (it is a great tool and I hope it isn’t blocked by anti-virus software in the future).

It may also be worth changing the permissions on the sethc.exe (and wmic.exe) keys you create such that they are read-only to everyone, admins included, just in case further malware tries to target these keys. In fact, why not protect the whole IFEO key, particularly if you are letting people logon with administrative rights?

I  hope this helps and stay safe people!

A Technique to Run Scripts Asynchronously via AppSense Environment Manager

Whilst AppSense Environment Manager (EM) has a rich set of trigger points, such as logon and reconnect, where you can run built-in actions or scripts, until very recently there was no built-in way of running something say five minutes after logon or every thirty minutes.

I’ve seen techniques that create scheduled tasks, which you can do directly in a PowerShell script rather than having to modify XML task templates, but with these you still have to get the script you want to run onto the local file system. Whilst I came up with a method of embedding any arbitrary file into an AppSense EM configuration, such as image files (see here), that approach is a little cumbersome, particularly if you need to change the embedded script.

I find an easier way to achieve this goal is to write a script inside an EM logon action that takes a copy of itself to a local file and then invokes powershell.exe to run it so that it runs outside of EM thus allowing the logon to finish but the newly launched script is still running as it is hosted inside an asynchronous powershell.exe process. Inside this script you can then wait for any period of time and then perform an action and either exit or loop around ad infinitum, or at least until logoff.

It works by looking to see if the -async option has been specified when the script was run. When EM runs the script, it won’t specify any options so we detect this and make it fire off a copy of the same script, since EM will delete the original when the script it invoked finishes, via powershell.exe which will take a different path through the code because the -async parameter is specified this time.

There’s an example version 10 EM configuration available for download here although the technique will work with any EM version. You will need to put your code into the PowerShell custom action, after the comment line reading “we are now outside of EM so can trigger later”, to perform whatever you need it to do. You might just want it to do something every few minutes or hours or you could put a file system or registry watcher in to trigger when something specifically changes (now there’s an idea for a future post if ever there was one!).

If you need the script to run elevated then simply set the action to run as System in the configuration but remember that environment variables and HKCU will not be for the logged on user but for the System account.

If you are also using AppSense Application Manager to prohibit powershell.exe then as long as you have configured it to ignore restrictions during logon then  it should be ok but you could always add a Process rule for EMUser.exe to allow powershell.exe child processes.