Advanced Procmon Part 2 – Filtering inclusions

In part 1 I showed some of the exclusions I would typically add to procmon’s filters to help reduce the volume of data you have to sift through. Having been working on a customer site last week where I had to try to find the cause of a disappearing “My Documents” folder, I think that it’s time to finally write part 2 so here goes.

What I had been asked to troubleshoot was why the hidden attribute on the user’s documents folder, which was being redirected, to a folder on the user’s H: drive (a DFS share, not that this is relevant other than it is not a local folder), was being set during a user’s XenApp session which made it rather difficult for them to work.

The first thing to check was what was doing the folder redirection which is where the first procmon filters were used. I spend a lot of time trying to trace where a Windows application or the Operating System itself stores a particular setting so that I can then automate its setting in someway. When it boils down to it, a setting is almost always going to be either stored in the registry or in a file which means we can use procmon to filter on the setting of the registry value or the writing of a file.

For example, let’s use procmon to see where Notepad (one of my favourite and most used apps) stores the font if you change it. I set my include filters as shown below – note that the excludes don’t really matter at this point:

procmon notepad filters

I then change the font to “Candara” in the Format->Font menu and click “OK”

notepad candara font

What we notice is that procmon doesn’t show anything after I click “OK” so have I got the filters wrong or is it saving it somewhere else (a network database for instance)? No, what we’re seeing is the result of a developer who has decided that changed settings will only get written back when the application exits normally rather than as soon as they are changed. It’s more efficient this way but does mean that changed settings won’t get written back if the application exits abnormally like if it crashes or it is terminated via task manager.

Note that the “WriteFile” operation isn’t always listed in the drop down list so if this is the case, select “IRP_MJ_WRITE” instead.

So after I exit notepad, procmon then shows me the following where I’ve highlighted the line where we can see the font being set to what I picked previously.

notepad font change

And there we have it – how to find where an application/process stores settings if it does persist them in a human readable form of course – it might encrypt them or store them as Unicode although the latter is relatively easy to spot, although not search for, as you’ll see zero byte padding for every character if you look at the data in regedit as shown below for a value in my mail profile.

unicode string data

What you might have to contend with is an application, like a Microsoft Office application for instance, writing a large number of settings in one go, rather than just the one you changed. What I do here, when the entry is a text entity like a folder name or some arbitrary string data, is to specify a string that is unlikely to be used anywhere else so I can either search for it in procmon or in the registry or file system directly. For instance last week I needed to know where Excel 2013 stored the folder that you specify in the “At startup, open all files in” setting in the Advanced->General options so I specified it as “H:\Elephant” and went on an elephant hunt (ok, search) …

So back to the tale of the hiding of the Documents folder. Using the above technique and because I know that redirected folder settings are written to “HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders” (and potentially “Shell Folders” in the same key but the former takes priority if the value exists in both keys), I set the procmon filters to the following in an admin session on my customer’s test XenApp server:

shell folders filter

At this point I didn’t know what the process was that was setting it so I didn’t filter on a specific process. Note that if multiple users had logged on during the procmon trace then it may have confused matters so you can filter on the session id too within procmon should you deem it necessary. You can right-click on the column headers in procmon, select “Select Columns” and then add columns like session id and user name although be careful if filtering on the latter as it may be a system process doing what you’re investigating rather than one running as that user.

procmon columns

This procmon trace told me that it was a process called EMUser.exe that was setting the “Personal” (Documents/My Documents) registry value which is part of the AppSense Environment Manager product. I checked the AppSense configuration and it was indeed setting the redirection but it wasn’t changing the attributes in any way that I could see.

The next part of the puzzle is to figure what I needed to filter on to be able to spot the changing of the attributes to hidden. This I did with the help of the good old attrib utility by creating a test folder and then running attrib to set and unset the hidden attribute on this folder so I could see what procmon reported.

attrib filter

Note my use of “echo %time%” – it can be a good idea to know what time you performed an operation in case there are many entries in a trace – I sometimes wait for the taskbar clock to hit an exact minute before I click something I want to trace the consequences of so I can search for that time in the log (and then exclude lines before that point if necessary by right clicking on that line and selecting “Exclude Events Before”).

So what we learn from the above is that when the hidden attribute is set, it will be the operation “SetBasicInformationFile” with “FileAttributes” in the “Detail” column containing the letter “H” since we see “HSDN” when I set hidden and system and we don’t see “H” when I subsequently remove the hidden attribute.

Back on the trail of the hidden documents folder, my filters became thus:

hidden attribute setting filter

If “SetBasicInformationFile” is not present in the drop down list then pick “IRP_MJ_SET_INFORMATION” instead.

I then logged on to the XenApp server with my test user account and very quickly noticed that the Documents folder was not visible so I stopped the trace at this point and found that it was Lync.exe (Communicator) that had set the hidden attribute. Next I reset the filter and just filtered on Lync.exe so I could try and figure out what Lync was doing just before it decided to set the hidden attribute. What I found was that is was querying the “Cache” value in the afore mentioned “User Shell Folders” key. Looking at this value I noticed that it too was set to “H:\Documents” which didn’t feel right given that it is normally set to the temporary internet files folder in the user’s local profile.

This is where you have to try and second guess developers or at least come up with a theory and then prove or disprove it. My theory was that Lync was querying what the Cache folder was set to, since it needed to use it, but had decided that it was a system folder that a user shouldn’t see so set it to hidden.

So what I did to try and both prove and fix this was to change the AppSense configuration to redirect the cache folder back to the temporary internet files folder in the user’s local profile (so “%USERPROFILE%\AppData\Local\Microsoft\Windows\Temporary Internet Files”). I did this and the documents folder no longer became hidden.

Thank you procmon – it would have been very difficult without you!


Advanced Procmon Part 1 – Filtering exclusions


For those of us who’ve been around the IT block a few times, we can remember life before Procmon, and filemon and regmon its honourable parents, and it was much, much harder to diagnose some issues although it still can’t tell you everything, unlike say an API monitor might. Anyway, this article is hopefully going to teach a few of you some extra filtering techniques that I’ve learned over many years of troubleshooting. The problem frequently being that you can’t see the wood for the trees in that there are so many events captured that you can’t find the ones you want amongst the many thousands of irrelevant ones.

Once you have configured filters, they can be exported to file via the File menu for subsequent importing at a later date or quickly configuring on another machine. I’d also select “Drop Filtered Events” from the Events menu since this will require less storage and resources although it does what it says on the tin so you won’t be able to see any of these dropped events if you later realise that you did actually want them.

Also, I always configure procmon to use a backing file rather than let it use the default of virtual memory as I believe that is usually less impactful of system resources, particularly when traces have to be run for a long time. This is on the File menu.

Results that can (usually) be safely ignored


You are almost certainly not seeing malicious code attempt a stack smashing hack; what is most likely happening is that the developer of the code that has had this result returned is trying to establish how big a buffer he (or she) needs to allocate in their code  in order to have the data returned that they require. This is because a developer doesn’t know in advance, for instance, how many registry values there will be in a key that he needs to enumerate so he’ll call the enumerate API with a zero length buffer which the API will interpret as a request to return the size of buffer needed to hold all of the data. The developer can then dynamically allocate a buffer of this size (and free it later when he’s finished with the data otherwise a memory leak will ensue) and then call the same API again with this buffer. You will usually see a procmon entry with all the same entries very soon after the “buffer overflow” one with a result of “success”. Many Microsoft APIs function this way.

buffer overflow


These will happen when APIs have been used that enumerate entries, such as the keys or values within a specific registry key. It is the API signalling to the developer that he should stop enumerating the item as there is no more data to be had.

In the example below we see that the Receiver process has been enumerating the per user installed products and after the fourth key (indexes start at zero not one) there are no more keys so the RegEnumKey API signals this by returning “no more entries”.

no more entries

Operations that can be ignored


Unless you are troubleshooting a handle leak or a failed access because an operation on a file or registry key fails since the handle has been closed, which is fairly pointless unless you have source code access for the suspect code and I find Process Explorer better for diagnosing handle leaks anyway since it will give you a list of the handles, then close operations can generally be ignored. We therefore can add the following to our filters:

procmon filter

There are probably more depending on exactly what it is you are trying to diagnose but these are the ones I typically start with to try and reduce the number of entries in a trace to make it easier to find what you are looking for. Remember also that you can right click on any item in a trace and select to exclude it which I do when I see an item appearing frequently in a trace that I reckon is nothing to do with what I am troubleshooting:

dynamic exclude 2

I tend to do this with writes to log files although sometimes it can be very useful to tie in a specific log file entry to other procmon activity – see here for a utility I wrote to quickly show you the text in a file at a given offset that you get from procmon.

Once you’ve found what you are looking for, or at least say the start of the area where the event of interest occurs, you can exclude all events before this point by right clicking on a specific line and selecting “Exclude Events Before” from the menu. This can make a large procmon trace quicker to search and filter further.

Processes that can be ignored

Unless investigating something that may be caused by your anti-virus product, I exclude all of it’s processes. Same for other file system filter or hook based products that your gut tells you aren’t relevant, such as the AMAgent.exe process which is part of AppSense’s Application Manager product.

Although the “System” process is ignored by default, I will sometimes unignore it if say I’m troubleshooting a service start issue at boot (where using the “Enable Boot Logging” option from the “Options” menu is handy and also for logon problems on desktop operating systems like Windows 7).

… and don’t forget that Ctrl-L is the shortcut to the filter dialog – use it and it will save you time if you are frequently tweaking filters as I tend to do.

Part 2 will cover filter includes for specific use cases such as trying to find where a particular application setting is stored in the file system or registry.

Teeing up your output

I was on customer site this week running some good old cmd scripts on a Windows Server that output to standard output, or stdout as we say in the Linux/Unix world, so showed in the cmd prompt that I’d run them from. But what I also needed as well as seeing the results as they happened was to record the output to a file in case I needed to refer back to any specific piece of it later given that it was running against nearly 6000 users.

Enter the Unix/Linux “tee” command that outputs to standard output as well as to a file. This is available through packages such as Cygwin but when on customer site you typically need access to a standalone executable that doesn’t need installing that you can just copy to the required system and run. To that end, here is my implementation of the “tee” utility that I wrote many years ago that will serve the purpose required.

At its simplest, it just takes an output file name that is overwritten with whatever it is piped to it so is invoked thus:

yourscript.cmd your parameters | tee outputfile.log

If you also want to catch errors then you need to do the following which is “stolen” directly from Unix shell scripting:

yourscript.cmd your parameters 2>&1 | tee outputfile.log

Since “2” is the file number for standard error (stderr) and “1” is the file number for standard output (stdout).

By default it overwrites the output file but specify a -a argument and it will append to an existing file and use -b if you have long line lengths as it reads a line at a time. Run with -? to see a usage message.

You can download the tool here but as ever it comes with absolutely no warranty and you use it entirely at your own risk (not that there’s any risk but I have to say that, don’t I?).

Yes, I know that tee, or tee-object, exists in PowerShell 3.0 and higher but sadly we still can’t guarantee that we have this on customer systems, much as I like PowerShell.

Transferring HP Recovery Media to Bootable USB Storage

Now that most desktops and laptops don’t ship with separate recovery media, like they did in the old days, and the cost of buying it afterwards is not insignificant, what happens if your hard drive completely fails thus taking with it the afore mentioned recovery media?

I kind of accidentally had this issue recently on a new laptop I was setting up so wondered if I could get the recovery media transferred from the hard disk to a bootable USB stick and then boot off this USB stick to perform the recovery to what was effectively a brand new hard drive. It was fortunately very easy to get this to work so here’s what you do:

  1. Get a blank USB stick/drive – for the recent HP laptop with Windows 8.1 I purchased, I used a 32GB stick although 16GB may just have worked.
  2. Format as NTFS – the main installation file is over 12GB but the maximum file size on FAT32 partitions is “only” 4GB so this is why FAT32 cannot be used.
  3. I’d taken an image of the laptop as it arrived, so before booting into Windows for the first time, so I mounted that on the system where I was preparing the bootable (not the destination laptop although you could use it). If your original recovery partition is still available you could use that instead.
  4. Copy all of the files/folders from the Recovery partition to the root of the USB stick. These are the folders you should see (note that they are hidden):hp recovery media
  5. On the USB stick, rename the file “\recovery\WindowsRE\winUCRD.wim” to “winre.wim” (this is the file that bcdedit shows as being the boot device in the \boot\BCD file)
  6. Make the USB stick bootable by running the following, obviously changing the drive letter as appropriate:
bootsect /nt60 e: /mbr

If it’s a Windows 8.x device then it may be configured for SecureBoot in which case you may need to enter the BIOS and disable this temporarily just whilst you are performing the recovery in order to get it to boot from USB. Don’t forget to change it back to the original settings once the restore is complete.

I’ll now keep this bootable around just in case the hard drive should fail or otherwise get hosed in such a way that the HP supplied recovery media will not work. At well under £10 currently for a USB 2.0 32GB USB stick, it’s a small price to pay.

Note that the recovery media is protected by a software mechanism that means that you cannot apply it to a different hardware model so this is not a means to clone illegal, activated, copies of Windows!

Vulnerability when using shared Windows base user profiles

In many Terminal Services/Remote Desktop and Citrix XenApp envrionments, a common practice is to use a single shared base profile for the users, typically a mandatory profile. Since there is effectively a single copy of this profile, the registry hive contained therein, either in an or ntuser.dat file, must have registry permissions such that the user whose profile gets built on top of this base has full access to it. However, since it is shared across many different users, it has to have a fairly lax set of permissions – typically allowing a group such as “Everyone” to have Full Control which is then propagated to each user’s individual profile as they logon.

What does this actually mean then when two or more users are logged on to the same server at the same time? Well one user can access, and change, another user’s HKEY_CURRENT_USER (HKCU) key via HKEY_USERS\<sid>. As a demonstration, see below where user gl-test02 has accessed the registry for user gl-test01 without even having to know any SIDs:

cross user profile

You might counter that registry editing tools are disabled through group policy but unless you are running something like AppSense Application Manager to give an extra layer of security via it’s Trusted Ownership feature, then a “policy proof” version is easy to produce with a binary editor and a little know how (a useful tool for troubleshooting in “locked down” environments). Even with an extra security layer in place, it is still relatively easy to access the registry via macros in any Microsoft Office product.

Being the responsible chap that I am, I wouldn’t write about a vulnerability without also offering a method of fixing it. All we need to do is to change the registry permissions at logon such that the group ACE (Access Control Entry) with Full Control is removed and replaced with one for the logging on user. I would normally use the Microsoft subinacl.exe utility but I couldn’t seem to get it to work on HKCU so I switched to setacl.exe from Helge Klein available here. I find the syntax a little perverse but it does the job so one shouldn’t grumble in what is a very flexible and granular free tool. Add the following line into a logon script, AppSense Environment Manager logon action, or similar, assuming you have the Everyone Full Control ACE in the base profile and your HKCU is secured:


setacl -on hkcu -ot reg -actn trustee 
-trst "n1:everyone;n2:%username%;ta:repltrst;w:d"

This only changes the top level key but as long as inheritance for subkeys is on then this should be enough and is very quick. This then means we get this after logon if we try the same method as above to access gl-test01’s registry hive from gl-test02’s session:

secured hkcu

Note that when running setacl you will probably see the following warnings, since it is hopefully being run as a non-administrative user who doesn’t have these privileges but also doesn’t need them for this particular use of setacl.

Privilege 'Restore files and directories' could not be enabled. 
SetACL's powers are restricted. Better run SetACL with admin rights.
Privilege 'Take ownership of files or other objects' could not be enabled. 
SetACL's powers are restricted. Better run SetACL with admin rights.

The mitigating circumstances here are that the information contained within the registry is probably not that sensitive, although there may be third party products that store passwords which could be used nefariously or even just lists of document names worked on which might interest someone.

Reasons for Reboots – Part 2

In part 1 we covered the relatively well known PendingFileRenameOperations registry value. In this post I will cover the much less well known mechanism that Windows Update uses to update files. I stumbled upon this mechanism by accident a while ago by trying to understand why, after applying Windows updates and it demanding a reboot, that there was nothing listed in PendingFileRenameOperations. It seems to be mostly for Side by Side assemblies (WinSxS). What I found was the following which seems to be largely undocumented:

  1. The “SetupExecute” value in “HKLM\SYSTEM\CurrentControlSet\Control\Session Manager” is populated with “C:\Windows\System32\poqexec.exe /display_progress \SystemRoot\WinSxS\pending.xml” where Poqexec.exe is the “Primitive Operations Queue Executor” according to the file’s (resource) properties.
  2. The pending.xml file contains the information about what files are to be replaced as well as registry entries to create/set/update.
  3. In “%systemroot%\winsxs\Temp\PendingRenames” you will find a number of *.cdf-ms which are referenced in the pending.xml file and are probably some kind of compiled manifest file which presumably also contain the actual binary content for the file updates.
  4. The SetupExecute value is actually processed at shutdown and the launched poqexec.exe process is presumably responsible for displaying the Windows updates messages. This can be captured with good old Process Monitor (procmon) although is tricky because to view the trace properly, procmon must be terminated cleanly which I achieved by having a cmd prompt running as system on the console logon screen via the sethc “hijack” method and using the /terminate parameter to procmon.
  5. The Windows Modules Installer service (TrustedInstaller.exe) has its startup type changed from “manual” to “automatic” so that it starts up at the next boot.
  6. By the time the system shuts down, the SetupExecute value’s contents are empty suggesting that poqexec has done all it needs to. Indeed, if you use Process Monitor to monitor the subsequent boot, poqexec does not feature at all.
  7. At the next boot, TrustedInstaller.exe then does more work to apply the updates, which can be seen if you enable boot logging in Process Monitor. This also writes to the CBS.log log file and seems to get its run order from “HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing” and %systemroot%\servicing.

Here is a very small extract from a pending.xml file prior to a reboot after applying some Windows updates:




Logs for the above can be found in “%systemroot%\Logs\CBS\CBS.log” where “CBS” stands for “Component Based Servicing”. Older log files are also found in this folder but are compressed into .cab archives. There is also a log file “%SystemRoot%\WinSxS\poqexec.log” but I have never found anything even vaguely useful in there (yet).

If you try to run poqexec.exe manually from an elevated command prompt, it will fail:


which is because it is a “native” application which needs to be executed in a specific way – see this article from Mark Russinovich for some details on native applications. Native applications are launched using the (undocumented) native API function “RtlCreateUserProcess” found in NTDLL.DLL. I wrote a utility using this API to see if I could manually run poqexec.exe to process the renames and deletes and potentially avoid a reboot but this does not seem to be the case unfortunately even if you can identify the components to be replaced and ensure that they are not in use. If you examine a system after Windows updates are applied and it is shutdown, e.g. if a VM then mount the virtual disk in another VM, typically the PendingRenames folder is empty and the SetupExecute value is also empty so some processing has definitely occurred.

Note that if you load the system registry hive of a machine that is not running, by using the Load Hive option in regedit in another windows machine of at least the same operating system version or higher, then you won’t see a “CurrentControlSet” key. This is because it is created at boot, and deleted at shutdown, from one of the ControlSet* keys in HKLM\System. Look at the “Current” value in HKLM\System\Select to tell you which key will become CurrentControlSet at the next boot but it is usually ControlSet001. This is also how booting into “Last Known Good Configuration” is implemented.

If you want to have a play around with poqexec.exe, entirely at your own risk obviously, then I’ve written a simple tool that will allow you to do so. It is available here and requires the 64 bit Visual C++ Redistributable Package for Visual Studio 2013 available here. Run it as follows:

NativeLauncher.exe \??\c:\Windows\System32\poqexec.exe “/display_progress \SystemRoot\WinSxS\pending.xml”

poqexec util

If anyone can shed any more light on any of this, please feel free to share! I did find some “Quite Interesting” information about Windows servicing here whilst researching for this post, such as how to make CBS logs even more verbose

Part 3 will cover how in use services and device drivers are flagged for removal at the next boot.


Reasons for Reboots – Part 1

So you’re quite happily working away having installed an update to an application that you’re not currently running only to find that the installer demands a reboot at the end of the installation anyway. “Why”, you ask yourself, “I wasn’t even using the program”. Over the next few posts, I will cover the main mechanisms that Windows uses to update in-use files and how you can sometimes safely make the required updates without rebooting.

In this article, we’ll look at the registry value “PendingFileRenameOperations” found in the key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager” although the value may not be present if there are no pending file renames to be performed at the next boot. This value consists of pairs of files – the first file of the pair is the source file and the second file is the destination. If the second file is the empty string then the source file is to be deleted.

pending file rename ops

In the example above, there are three Adobe Flash files that are to be deleted, presumably as they were in use when the Flash update was applied and the replacement files have different names (since the file names seem to include the version number in this example). Now whilst these files may have been in use when the update was applied, they may not be now so we can use SysInternals/Microsoft Process Explorer to see if these files are still in use or we can just try deleting the files anyway since if they are in use we will get an error.


In the Process Explorer handle search results above we can see that nothing currently has this file open so we could do the deletion now. If a process did have a handle open to the file that needed deleting then we would have to make a judgement call as to whether we could safely terminate the process holding the handle open, and restart it if required. Do not use the Process Explorer functionality that allows you to close a handle though as this may cause the process owning the handle to malfunction or crash since the application developer for that application believes they are in control of the handle’s lifespan and probably won’t expect, or cater for, external interference.

Next in the PendingFileRenameOperations value above, googledrivesync.exe is to be deleted and then the file “C:\Program Files\Google\Drive\TBM5CAD.tmp” is to be moved to googledrivesync.exe. Actually, the delete operation here is superfluous as the file move will overwrite googledrivesync.exe anyway. Here we could terminate googeldrivesync.exe, do the file move manually as an administrator and then restart (the updated) googledrivesync.exe.

You might think that if PendingFileRenameOperations just included file delete operations, rather than moves, then the reboot could be delayed until a convenient time. However, I have seen many times where the file(s) to be deleted is one that is still in use because the installer has failed to stop/restart whatever was using the old, to be deleted, file such that the system although claims to be running the updated software is in fact still running some of the old components which may cause problems. I always check PendingFileRenameOperations after an installation even if the installer hasn’t requested a reboot. These files are typically “rollback” files which are usually located in the %systemdrive%\Config.msi folder and have a “.rbf” extension. To find out what they are, file wise, since they will have been renamed, take a copy, add a .dll extension and then view the properties in Explorer.

rollback file

Note that regedit does not allow you to have empty strings when modifying a REG_MULTI_SZ value so if you want to edit the value, you must not use regedit – export to .reg, modify that in a text editor like notepad and re-import, otherwise the value will become corrupt. If there is no data (file names) left in the value then the value itself should be deleted. Don’t click “OK” when viewing the value data even if you haven’t changed it (my best practice is always to click “Cancel” on anything where you’ve not made a change anyway) otherwise you will get the warning below (which I personally feel is a (long standing) bug in regedit since REG_MULTI_SZ values can quite evidently contain empty strings):

regedit no multi sz

and the value will then look something like this:

bad pending file renames

which is unfortunately now rather broken and will potentially update/delete the wrong files!

You can add your own entries into the PendingFileRenameOperations using an old Microsoft command line tool called InUse.exe which is available here if you need it.

If you set Process Monitor to log from boot, you can see the PendingFileRenameOperations value being processed. It’s not very interesting though! You should see that it is processed in a top down manner – updates to the value when setting file moves/deletions are usually appended to the value so the most recent updates will feature at the end, not the start, of the value’s data.