Citrix StoreFront Log Viewer Tool

Have you ever had the need to debug StoreFront? I have on a couple of occasions and it wasn’t the easiest debugging exercise I’d ever undertaken unfortunately. Changing the logging level is easy enough with the PowerShell cmdlet Set-DSTraceLevel. For example, run the following on a StoreFront server to enable verbose logging (see https://support.citrix.com/article/CTX139592):

Add-PSSnapin Citrix.DeliveryServices.Framework.Commands

Set-DSTraceLevel –All –TraceLevel Verbose

Which will restart the Citrix services, having updated web.config files, and various log files will be created in “C:\Program Files\Citrix\Receiver StoreFront\Admin\trace”. It will also cause debug statements to be produced which can be picked up with tools like SysInternals dbgview.

The problem, in my experience, is that reading the log files, of which there are many, can be a bit of a chore. The log files are almost XML format but they are not fully compliant as they don’t have a top level node, since presumably adding this would have a performance hit. Even if you can get them into XML, working with them in XML isn’t particularly easy although that may depend on what XML tool you use (I would typically use Internet Explorer since that’s all I can rely on being on customer machines where I don’t want to start installing third party software).

Fortunately, PowerShell comes to our rescue (yet again) since it’s very easy in scripts to make this almost XML be properly formed so that this can then be quickly parsed and each log record output to a csv file or an on-screen grid view where filtering and/or searching can then take place.

The script is available here and can extract logs from multiple StoreFront servers, by accessing the logs via their C$ share, and splicing them together based on the time of each event. You can either specify a starting and ending date/time range via –start and –end respectively, specify –sinceBoot to include all entries since the last boot of each server or use –last with a number and a specifier such as ‘d’ for days, ‘m’ for minutes and ‘s’ for seconds so “-last 8h” means “in the last eight hours”. For example run the following to see all errors in the last two hours on the two specified StoreFront servers and display on-screen in a filterable grid view:

& '.\parse storefront log files.ps1' -computers storefront01,storefront02 -last 2h -subtypes error

If you have verbose logging enabled but only want to show warning and error entries then specify “-subtypes error,warning” since the default is to include all entries, including verbose ones.

Clicking “OK” at the bottom right of the grid view will copy any selected log lines into the clipboard, e.g. for web searches or logging with Citrix.

Specifying the -Verbose option gives information on what logs are being parsed, from which servers and for what time ranges.

Finally, don’t forget to change the StoreFront logging level back to something like “Error” rather than leaving it at “Verbose” as that is unlikely to help performance!

Advertisements

Citrix PVS device detail viewer with action pane

here

I find myself frequently using the script I wrote, see here, to check the status of PVS devices and then sometimes I need to perform power actions on them, turn maintenance mode on or off or maybe message users on them before performing power actions (I am a nice person after all). Whilst we can perform most of those actions in the PVS console, if you are dealing with devices across multiple collections, sites or PVS instances then that can involve a lot of jumping around in the PVS console. Plus if you want to change maintenance mode settings or message logged on users then you need to do this from Citrix Studio so you’ll need to launch that and go and find the PVS devices in there.

So I decided to put my WPF knowledge to work and built a very simple user interface in Visual Studio and then inserted it into the PVS device detail viewer script. Once you’ve run the script and got a list of the PVS devices, sorted and/or filtered as you desire, select those devices and then click on the “OK” button down in the bottom right hand side of the grid view. Ctrl-A will select all devices which can be useful if you’ve filtered on something like “Booted off latest” so you only have devices displayed which aren’t booting off the latest production vdisk. This will then fire up a user interface that looks like this, unless you’ve run the script with the -noMenu option or hit “Cancel” in the grid view.

pvs device actioner gui

All the devices you selected in the grid view will be selected automatically for you but you can deselect any before clicking on the button for an action. It will ask you to confirm the action before undertaking it.

pvs device viewer confirm

If you select the “Message Users” option then an additional dialog will be shown asking you for the text, caption and level of the message although you can pass these on the command line via -messageText and -MessageCaption options.

pvs device viewer message box

The “Boot” and “Power Off” options use PVS cmdlets rather than Delivery Controller ones since the devices may not be known to the DDC. “Shutdown” and “Restart” use the “Stop-Computer” and “Restart-Computer” cmdlets respectively and I have deliberately not used the -force parameter with them so if users are logged on, the commands will fail. Look in the window you invoked the script from for errors.

You can keep clicking the action buttons until you exit the user interface so, for instance, it can be used to enable maintenance mode, message users asking them to logoff, reboot the devices when they have logged off or you have had enough of waiting for them to do so and then turning off maintenance mode, if you want to put it back in to service after the reboot.

I hope you find it as useful as I do but note that you use the script entirely at your own risk. It is available here and requires version 7.7 or higher of PVS, XenApp 7.x and PowerShell 3.0 or later where those consoles are installed on the machine where you will run the script from (so that their PowerShell cmdlets are available too).

 

Update to Citrix PVS device detail viewer

In using the script, introduced here, at a customer this week, I found a few bugs, as you do, and also added a few new features to make my life easier.

In terms of new features, I’ve added a -name command line option which will only show information for devices that match the regular expression you specify. Now don’t run away screaming because I’ve mentioned regular expressions as, contrary to popular belief, they can be straightforward (yes, really!). For instance, if you’ve got devices CTXUAT01, CTXUAT02 and so on that you just want to report on then a regex that will match that is “CTXUAT” – we can forget about matching the numbers unless you specifically need to only match certain of those devices.

Another option I needed was to display Citrix tag information since I am providing a subset of servers, using the same naming convention as the rest of the servers, where there are tag restrictions so specific applications only run off specific servers. Using tags means I don’t have to create multiple delivery groups which makes maintenance and support easier. Specify a -tags option and a column will be added with  the list of tags for each device, if present.

However, adding the -tags option was “interesting” because the column didn’t get added. A bug in my code – surely not! What I then found, thanks to web searches, is that versions of PowerShell prior to 5 have a limit of 30 columns so any more than that and they silently get dropped. The solution? Upgrade to PowerShell version 5 or if that’s not possible and you want the tag information, remove one of the other columns by changing the $columns variable. Yes,  30 columns is a lot for the script to produce but I decided it was better to produce too much information, rather than too little, and then let columns be removed later in Excel or the grid view.

I also found a bug, yes really, where if the vDisk configured for a device had been changed since it was booted then it would not be identified as not booting off the latest. That’s fixed so remember you can quickly find all devices not booting off the latest production version of the vDisk or booting off the wrong vDisk by filtering on the “Booted off Latest” column:

booted off latest

The script is still available here (GitHub? never heard of it :-))

Citrix Provisioning Services device detail viewer

Whilst struggling to find some devices in the PVS console that I thought that I’d just added to a customer’s PVS server via the XenDesktop Setup wizard, I reckoned it should be relatively easy to knock up something that would quickly show me all the devices, their device collection, disk properties and then also cross reference to a Citrix Delivery Controller to show machine catalogue, delivery group, registration state and so on. Note that I’m not trying to reinvent that wheel thing here as I know there are already some great PVS documentation scripts such as those from Carl Webster (available here).

What I wanted was something that would let me quickly view and filter the information from multiple PVS servers, such as development and production instances. Whilst PowerShell can easily export to csv and you can then use Excel, or Google Sheets, to sort and filter, that is still a little bit of a faff so I use PowerShell’s great Out-GridView cmdlet which gives you an instant graphical user interface with zero effort (not that using WPF in PowerShell is particularly difficult!) which can be sorted and filtered plus columns you don’t want can be removed without having to modify the script.

The script takes two parameters which it will prompt for if not specified as they are mandatory:

-pvsServers

-ddcs

Both take comma separated lists of PVS servers and Desktop Delivery Controllers respectively although you can just specify a single server for each. If you’ve got multiple PVS servers using the same database then you only need to specify one of them. Ditto for the DDCs.

You can also specify a -csv argument with the name of a csv file if you do want output to got to a csv file but if you don’t then it will default to a filterable and sortable grid view.

Some hopefully useful extra information includes “Booted off latest” where devices with “false” in this column are those which have not been booted off the latest production version of their vDisk so may need rebooting. There’s also “Boot Time” which you can sort on in the grid view to find devices which are overdue a reboot, perhaps because they are not (yet) subject to a scheduled reboot. Plus you can quickly find those that aren’t in machine catalogues or delivery groups or where there is no account for them in Active Directory. You can also filter on devices which are booting off an override version of a vDisk which may be unintentional.

The script is available here and requires version 7.7 or higher of PVS since that is when the PowerShell cmdlets it uses were introduced. Run it from somewhere where you have installed the Citrix PVS and Studio consoles, like a dedicated management server – I’m a firm believer in not running these on their respective servers since that can starve those servers of resource  and thus adversely affect the environment. Ideally, also have the Active Directory PowerShell module (ActiveDirectory) installed too so that the device’s status in AD can be checked.

I’ve just picked out the fields from PVS, Delivery Controllers and AD that are of interest to me but you should be able to add others in the script if you need to.
Continue reading “Citrix Provisioning Services device detail viewer”

Taking and resizing a screen shot in an Outlook email directly

As an IT Consultant, I frequently have to send Outlook emails containing screenshots. In the old, old days, I’d press the PrintScreen key, paste into Paint (I will miss you old friend as you are seen to be retired), select what I needed and then paste into the email. Then along came the Snipping Tool which made things a little easier but it still means going away from your email and then coming back. Oh, and don’t go on about third-party tools – I stick with what’s built-in then I know I’ve got it wherever I go.

It seems to be a little known/used fact that you can add a screen capture button directly to the quick access toolbar (QAT) in all recent Microsoft Office products. So I added this to the QAT but what I then found myself doing for pretty much every screenshot was to shrink it which either involved dragging the resize points around or quite a few clicks and key presses in the email message which kind of cancelled out the ease of getting the image in there in the first place.

Taking VBA code I found here and here (I was going to write it from scratch but these were almost exactly what I needed so why reinvent this wheel thing?), I ended up with the following VBA code, placed in the ThisOutlookSession object (hit Alt F11 to bring up the Outlook VBA editor):

Public Sub Shrinker()
 Const wdInlineShapePicture = 3
 
 If TypeName(ActiveWindow) = "Inspector" Then
   If ActiveInspector.IsWordMail And ActiveInspector.EditorType = olEditorWord Then
     If ActiveInspector.WordEditor.Application.Selection.InlineShapes.Count > 0 Then
       For Each wrdShp In ActiveInspector.WordEditor.Application.Selection.InlineShapes
         If wrdShp.Type = wdInlineShapePicture Then
           wrdShp.ScaleHeight = wrdShp.ScaleHeight - 10
           wrdShp.ScaleWidth = wrdShp.ScaleWidth - 10
         End If
       Next
     End If
   End If
 End If
 
 Set wrdShp = Nothing

End Sub

You can then add a button to the QAT in an email compose window that calls this code:

outlook email macro commands QAT

Notice the “Screen Clipping” button which is added thus:

add screen clipping to qat

Which will then give you these toolbar buttons in your email compose window:

outlook email screen capturing

You can then click the screen capture icon when needed which will hide the email compose window, give you cross hairs to select the region you need and once released it will be pasted into your email, selected, which will come back into focus:

outlook email screen captured

If you need to resize then click on the QAT button you assigned to the Shrinker macro which will scale it down by 10% each time you click it. If you want to grow it again, just hit Ctrl Z to undo. If you need to enlarge rather than shrink, duplicate the macro and put +10 instead of -10 and assign this macro to the QAT too.

I hope this helps save you lots of time which it has for me. Don’t forget to save the macro – the easiest way is to close Outlook and click Yes when it asks whether to save ThisOutlookSession.

 

Scripted Reporting & Alerting of Citrix Provisioning Services Boot Times

Citrix PVS, formerly Ardence, is still one of my favourite software products. When it works, which is the vast majority of the time if it is well implemented, it’s great but how do you tell how well it is performing? If you’ve enabled event log generation for your PVS servers thus:

pvs event log server

then the Citrix Streaming Service will write boot times of your target devices to the application event log:

pvs boot event

So we can filter in the event log viewer or use the script I’ve written which searches the event log for these entries and finds the fastest, slowest, average, median and mode values from one or more PVS servers and optionally creates a single csv file with the results. A time range can also be specified, such as the last 7 days.

The script lends itself to being run via a scheduled task as it can either email the results to a specified list of recipients or it can send an email only when specific thresholds are exceeded, such as the average time being greater than say 2 minutes.

For instance, running the following:

& '.\Get PVS boot time stats.ps1' -last 7d -output c:\boot.times.csv -gridview

Will write the boot times to file, in seconds, for the last seven days on the PVS server where you are running the script. It will also display the results in a sortable and filterable gridview and output a summary like this:

Got 227 events from 1 machines : fastest 21 s slowest 30 s mean 25 s median 25 s mode 26 s (39 instances)

Or we could run the following to query more servers and send an email via an SMTP mail server if the slowest time exceeds 5 minutes in the last week:

& '.\Get PVS boot time stats.ps1' -last 7d -output c:\boot.times.csv -mailserver yourmailserver -recipients someone@somewhere.com -slowestAbove 300 -computers pvsserver1,pvsserver2

The script has integrated help, giving details on all the command line options available, and can be run standalone or via scheduled tasks.

The script can be downloaded from here. Full help is built in and can be accessed via F1 in PowerShell ISE or Get-Help.

Update 13/02/18

Now with -chartView and -gridView options to give an on screen  chart and grid view respectively.

 

Petya: disabling remote execution of psexec

This morning in response to reports of an outbreak of yet more malware I wrote a quick blog post on one way to stop the SysInternals psexec from being allowed to execute by using the Image File Execution Options registry key mechanism – see here for that post.

My technique was, and still is sound, but @RennJohnny correctly pointed out that if the psxec.exe executable was renamed then my approach would not work. I therefore set about finding another way to stop psexec from running for those who don’t (yet) have security software in place to stop the exploit. Note that you need to be running it on a system where you are an administrator and have the same rights on the remote system to be attacked. As the disposable virtual machine that I used for my testing is not on a domain, I passed explicit credentials to psexec for my testing – I don’t believe Petya operates this way but my solution will still work.

So when psexec is used to run something on a remote system, it works by creating a new service executable called psexesvc.exe which is embedded within the original psexec.exe file. This is copied to the Windows folder on the remote machine via the admin$ default share (hence why you need to be an admin to get psexec to work remotely). It then creates the PSEXESVC service with this, now local, executable, starts it and then runs the specified command.

What I found was that even when I copied psexec.exe to another file name, the file produced and copied to the remote system was still called psexesvc.exe. This is what happens when you run the copied psexec.exe and tell it to invoke a command on a remote machine:

psexesvc allowed

 

On that remote system we can then see this has been created in the services registry key:

psexesvc registry

How do we stop it? I reckon that the easiest way is to use good old Image File Execution Options (IFEO) mechanism again but this time we create the key “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\PSEXESVC.exe” and in there create the REG_SZ value called “Debugger” and set it to “svchost.exe”. Now when we try and run psexec to execute on the system where we just patched the registry, this happens instead:

psexesvc blocked

What happened? Well when the Service Control Manager (SCM) on the remote machine was asked to start the PSEXESVC service, it started the psexesvc.exe process but the IFEO entry we created caused it to run svchost.exe instead but as that can’t be used as a standalone service, it failed to start so SCM reported this to psexec which is the error we see above. You will also get this in the System event log of the remote system:

psexesvc eventlog.PNG

One way to roll it out to all your computers is to put the above registry value into a Group Policy Preference that applies to those computers.

You can also create a dummy psexesvc.exe file in your Windows folder, remove all permissions and change the owner to, say, TrustedInstaller, and that will also prevent it from running.

I hope this helps some of you and stay safe (and don’t run routinely with admin privileges!).