## Teeing up your output

I was on customer site this week running some good old cmd scripts on a Windows Server that output to standard output, or stdout as we say in the Linux/Unix world, so showed in the cmd prompt that I’d run them from. But what I also needed as well as seeing the results as they happened was to record the output to a file in case I needed to refer back to any specific piece of it later given that it was running against nearly 6000 users.

Enter the Unix/Linux “tee” command that outputs to standard output as well as to a file. This is available through packages such as Cygwin but when on customer site you typically need access to a standalone executable that doesn’t need installing that you can just copy to the required system and run. To that end, here is my implementation of the “tee” utility that I wrote many years ago that will serve the purpose required.

At its simplest, it just takes an output file name that is overwritten with whatever it is piped to it so is invoked thus:

yourscript.cmd your parameters | tee outputfile.log

If you also want to catch errors then you need to do the following which is “stolen” directly from Unix shell scripting:

yourscript.cmd your parameters 2>&1 | tee outputfile.log

Since “2” is the file number for standard error (stderr) and “1” is the file number for standard output (stdout).

By default it overwrites the output file but specify a -a argument and it will append to an existing file and use -b if you have long line lengths as it reads a line at a time. Run with -? to see a usage message.

You can download the tool here but as ever it comes with absolutely no warranty and you use it entirely at your own risk (not that there’s any risk but I have to say that, don’t I?).

Yes, I know that tee, or tee-object, exists in PowerShell 3.0 and higher but sadly we still can’t guarantee that we have this on customer systems, much as I like PowerShell.

## Transferring HP Recovery Media to Bootable USB Storage

Now that most desktops and laptops don’t ship with separate recovery media, like they did in the old days, and the cost of buying it afterwards is not insignificant, what happens if your hard drive completely fails thus taking with it the afore mentioned recovery media?

I kind of accidentally had this issue recently on a new laptop I was setting up so wondered if I could get the recovery media transferred from the hard disk to a bootable USB stick and then boot off this USB stick to perform the recovery to what was effectively a brand new hard drive. It was fortunately very easy to get this to work so here’s what you do:

1. Get a blank USB stick/drive – for the recent HP laptop with Windows 8.1 I purchased, I used a 32GB stick although 16GB may just have worked.
2. Format as NTFS – the main installation file is over 12GB but the maximum file size on FAT32 partitions is “only” 4GB so this is why FAT32 cannot be used.
3. I’d taken an image of the laptop as it arrived, so before booting into Windows for the first time, so I mounted that on the system where I was preparing the bootable (not the destination laptop although you could use it). If your original recovery partition is still available you could use that instead.
4. Copy all of the files/folders from the Recovery partition to the root of the USB stick. These are the folders you should see (note that they are hidden):
5. On the USB stick, rename the file “\recovery\WindowsRE\winUCRD.wim” to “winre.wim” (this is the file that bcdedit shows as being the boot device in the \boot\BCD file)
6. Make the USB stick bootable by running the following, obviously changing the drive letter as appropriate:
bootsect /nt60 e: /mbr

If it’s a Windows 8.x device then it may be configured for SecureBoot in which case you may need to enter the BIOS and disable this temporarily just whilst you are performing the recovery in order to get it to boot from USB. Don’t forget to change it back to the original settings once the restore is complete.

I’ll now keep this bootable around just in case the hard drive should fail or otherwise get hosed in such a way that the HP supplied recovery media will not work. At well under £10 currently for a USB 2.0 32GB USB stick, it’s a small price to pay.

Note that the recovery media is protected by a software mechanism that means that you cannot apply it to a different hardware model so this is not a means to clone illegal, activated, copies of Windows!

## Vulnerability when using shared Windows base user profiles

In many Terminal Services/Remote Desktop and Citrix XenApp envrionments, a common practice is to use a single shared base profile for the users, typically a mandatory profile. Since there is effectively a single copy of this profile, the registry hive contained therein, either in an ntuser.man or ntuser.dat file, must have registry permissions such that the user whose profile gets built on top of this base has full access to it. However, since it is shared across many different users, it has to have a fairly lax set of permissions – typically allowing a group such as “Everyone” to have Full Control which is then propagated to each user’s individual profile as they logon.

What does this actually mean then when two or more users are logged on to the same server at the same time? Well one user can access, and change, another user’s HKEY_CURRENT_USER (HKCU) key via HKEY_USERS\<sid>. As a demonstration, see below where user gl-test02 has accessed the registry for user gl-test01 without even having to know any SIDs:

You might counter that registry editing tools are disabled through group policy but unless you are running something like AppSense Application Manager to give an extra layer of security via it’s Trusted Ownership feature, then a “policy proof” version is easy to produce with a binary editor and a little know how (a useful tool for troubleshooting in “locked down” environments). Even with an extra security layer in place, it is still relatively easy to access the registry via macros in any Microsoft Office product.

Being the responsible chap that I am, I wouldn’t write about a vulnerability without also offering a method of fixing it. All we need to do is to change the registry permissions at logon such that the group ACE (Access Control Entry) with Full Control is removed and replaced with one for the logging on user. I would normally use the Microsoft subinacl.exe utility but I couldn’t seem to get it to work on HKCU so I switched to setacl.exe from Helge Klein available here. I find the syntax a little perverse but it does the job so one shouldn’t grumble in what is a very flexible and granular free tool. Add the following line into a logon script, AppSense Environment Manager logon action, or similar, assuming you have the Everyone Full Control ACE in the base profile and your HKCU is secured:

setacl -on hkcu -ot reg -actn trustee



This only changes the top level key but as long as inheritance for subkeys is on then this should be enough and is very quick. This then means we get this after logon if we try the same method as above to access gl-test01’s registry hive from gl-test02’s session:

Note that when running setacl you will probably see the following warnings, since it is hopefully being run as a non-administrative user who doesn’t have these privileges but also doesn’t need them for this particular use of setacl.

Privilege 'Restore files and directories' could not be enabled.
SetACL's powers are restricted. Better run SetACL with admin rights.
Privilege 'Take ownership of files or other objects' could not be enabled.
SetACL's powers are restricted. Better run SetACL with admin rights.

The mitigating circumstances here are that the information contained within the registry is probably not that sensitive, although there may be third party products that store passwords which could be used nefariously or even just lists of document names worked on which might interest someone.

## Reasons for Reboots – Part 2

In part 1 we covered the relatively well known PendingFileRenameOperations registry value. In this post I will cover the much less well known mechanism that Windows Update uses to update files. I stumbled upon this mechanism by accident a while ago by trying to understand why, after applying Windows updates and it demanding a reboot, that there was nothing listed in PendingFileRenameOperations. It seems to be mostly for Side by Side assemblies (WinSxS). What I found was the following which seems to be largely undocumented:

1. The “SetupExecute” value in “HKLM\SYSTEM\CurrentControlSet\Control\Session Manager” is populated with “C:\Windows\System32\poqexec.exe /display_progress \SystemRoot\WinSxS\pending.xml” where Poqexec.exe is the “Primitive Operations Queue Executor” according to the file’s (resource) properties.
2. The pending.xml file contains the information about what files are to be replaced as well as registry entries to create/set/update.
3. In “%systemroot%\winsxs\Temp\PendingRenames” you will find a number of *.cdf-ms which are referenced in the pending.xml file and are probably some kind of compiled manifest file which presumably also contain the actual binary content for the file updates.
4. The SetupExecute value is actually processed at shutdown and the launched poqexec.exe process is presumably responsible for displaying the Windows updates messages. This can be captured with good old Process Monitor (procmon) although is tricky because to view the trace properly, procmon must be terminated cleanly which I achieved by having a cmd prompt running as system on the console logon screen via the sethc “hijack” method and using the /terminate parameter to procmon.
5. The Windows Modules Installer service (TrustedInstaller.exe) has its startup type changed from “manual” to “automatic” so that it starts up at the next boot.
6. By the time the system shuts down, the SetupExecute value’s contents are empty suggesting that poqexec has done all it needs to. Indeed, if you use Process Monitor to monitor the subsequent boot, poqexec does not feature at all.
7. At the next boot, TrustedInstaller.exe then does more work to apply the updates, which can be seen if you enable boot logging in Process Monitor. This also writes to the CBS.log log file and seems to get its run order from “HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing” and %systemroot%\servicing.

Here is a very small extract from a pending.xml file prior to a reboot after applying some Windows updates:

<DeleteFilepath=\??\C:\Windows\System32\wdigest.dll/>

Logs for the above can be found in “%systemroot%\Logs\CBS\CBS.log” where “CBS” stands for “Component Based Servicing”. Older log files are also found in this folder but are compressed into .cab archives. There is also a log file “%SystemRoot%\WinSxS\poqexec.log” but I have never found anything even vaguely useful in there (yet).

If you try to run poqexec.exe manually from an elevated command prompt, it will fail:

which is because it is a “native” application which needs to be executed in a specific way – see this article from Mark Russinovich for some details on native applications. Native applications are launched using the (undocumented) native API function “RtlCreateUserProcess” found in NTDLL.DLL. I wrote a utility using this API to see if I could manually run poqexec.exe to process the renames and deletes and potentially avoid a reboot but this does not seem to be the case unfortunately even if you can identify the components to be replaced and ensure that they are not in use. If you examine a system after Windows updates are applied and it is shutdown, e.g. if a VM then mount the virtual disk in another VM, typically the PendingRenames folder is empty and the SetupExecute value is also empty so some processing has definitely occurred.

Note that if you load the system registry hive of a machine that is not running, by using the Load Hive option in regedit in another windows machine of at least the same operating system version or higher, then you won’t see a “CurrentControlSet” key. This is because it is created at boot, and deleted at shutdown, from one of the ControlSet* keys in HKLM\System. Look at the “Current” value in HKLM\System\Select to tell you which key will become CurrentControlSet at the next boot but it is usually ControlSet001. This is also how booting into “Last Known Good Configuration” is implemented.

If you want to have a play around with poqexec.exe, entirely at your own risk obviously, then I’ve written a simple tool that will allow you to do so. It is available here and requires the 64 bit Visual C++ Redistributable Package for Visual Studio 2013 available here. Run it as follows:

NativeLauncher.exe \??\c:\Windows\System32\poqexec.exe “/display_progress \SystemRoot\WinSxS\pending.xml”

If anyone can shed any more light on any of this, please feel free to share! I did find some “Quite Interesting” information about Windows servicing here whilst researching for this post, such as how to make CBS logs even more verbose

Part 3 will cover how in use services and device drivers are flagged for removal at the next boot.

## Reasons for Reboots – Part 1

So you’re quite happily working away having installed an update to an application that you’re not currently running only to find that the installer demands a reboot at the end of the installation anyway. “Why”, you ask yourself, “I wasn’t even using the program”. Over the next few posts, I will cover the main mechanisms that Windows uses to update in-use files and how you can sometimes safely make the required updates without rebooting.

In this article, we’ll look at the registry value “PendingFileRenameOperations” found in the key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager” although the value may not be present if there are no pending file renames to be performed at the next boot. This value consists of pairs of files – the first file of the pair is the source file and the second file is the destination. If the second file is the empty string then the source file is to be deleted.

In the example above, there are three Adobe Flash files that are to be deleted, presumably as they were in use when the Flash update was applied and the replacement files have different names (since the file names seem to include the version number in this example). Now whilst these files may have been in use when the update was applied, they may not be now so we can use SysInternals/Microsoft Process Explorer to see if these files are still in use or we can just try deleting the files anyway since if they are in use we will get an error.

In the Process Explorer handle search results above we can see that nothing currently has this file open so we could do the deletion now. If a process did have a handle open to the file that needed deleting then we would have to make a judgement call as to whether we could safely terminate the process holding the handle open, and restart it if required. Do not use the Process Explorer functionality that allows you to close a handle though as this may cause the process owning the handle to malfunction or crash since the application developer for that application believes they are in control of the handle’s lifespan and probably won’t expect, or cater for, external interference.

Next in the PendingFileRenameOperations value above, googledrivesync.exe is to be deleted and then the file “C:\Program Files\Google\Drive\TBM5CAD.tmp” is to be moved to googledrivesync.exe. Actually, the delete operation here is superfluous as the file move will overwrite googledrivesync.exe anyway. Here we could terminate googeldrivesync.exe, do the file move manually as an administrator and then restart (the updated) googledrivesync.exe.

You might think that if PendingFileRenameOperations just included file delete operations, rather than moves, then the reboot could be delayed until a convenient time. However, I have seen many times where the file(s) to be deleted is one that is still in use because the installer has failed to stop/restart whatever was using the old, to be deleted, file such that the system although claims to be running the updated software is in fact still running some of the old components which may cause problems. I always check PendingFileRenameOperations after an installation even if the installer hasn’t requested a reboot. These files are typically “rollback” files which are usually located in the %systemdrive%\Config.msi folder and have a “.rbf” extension. To find out what they are, file wise, since they will have been renamed, take a copy, add a .dll extension and then view the properties in Explorer.

Note that regedit does not allow you to have empty strings when modifying a REG_MULTI_SZ value so if you want to edit the value, you must not use regedit – export to .reg, modify that in a text editor like notepad and re-import, otherwise the value will become corrupt. If there is no data (file names) left in the value then the value itself should be deleted. Don’t click “OK” when viewing the value data even if you haven’t changed it (my best practice is always to click “Cancel” on anything where you’ve not made a change anyway) otherwise you will get the warning below (which I personally feel is a (long standing) bug in regedit since REG_MULTI_SZ values can quite evidently contain empty strings):

and the value will then look something like this:

which is unfortunately now rather broken and will potentially update/delete the wrong files!

You can add your own entries into the PendingFileRenameOperations using an old Microsoft command line tool called InUse.exe which is available here if you need it.

If you set Process Monitor to log from boot, you can see the PendingFileRenameOperations value being processed. It’s not very interesting though! You should see that it is processed in a top down manner – updates to the value when setting file moves/deletions are usually appended to the value so the most recent updates will feature at the end, not the start, of the value’s data.

## How to automatically install selective Windows Updates

I know there is a way to get Windows Defender to specifically update its definitions via the MpCmdRun.exe command but I wanted to find a way that would allow me to schedule update installations for any arbitrary update, not just for Windows Defender.

To this end, I found the Microsoft documentation for the Windows Update Agent API and adapted the VBS script found at http://msdn.microsoft.com/en-gb/library/windows/desktop/aa387102(v=vs.85).aspx so that it could be passed a string to match on the command line and then go off and find, download and install any updates that matched this string in their title. Couple this with some rudimentary auditing to the event log and a scheduled task to run the script a few times per day and the problem/annoyance is solved.

The script can be found here and comes with absolutely no warranty (although has been working fine for me for months). You can test it via an administrative cmd prompt using cscript.exe to ensure it works correctly before creating a scheduled task for it, when debugging becomes slightly trickier. See an example command line in the screenshot below but it will also take /Verbose:y to give detailed output as to what it is doing at each stage (don’t specify this option when setting up the scheduled task).

Once you have the script saved in a suitable location, run the task scheduler (taskschd.msc) as an administrator and create a new task to look something like this:

Notice the use of the //B option to wscript.exe to suppress any message boxes the script may produce since that would cause the script to stall. Also, I’ve created multiple Daily schedules to run it in the morning, afternoon and evening although the task is also set to run it as soon as possible if the task is missed, e.g. the machine is asleep/off.

I can then check the event log, for which I’ve set up a filter as shown below for the “WSH” source, to see what updates have been applied by the script although they will also be visible in the standard update history in the Windows Update control panel applet.

Obviously you could modify the string passed via the /Match: option to enable auto updating of any updates you want – just create a new scheduled task for each different /Match: argument used. Note that the script uses a very simple, case insensitive, substring search so there’s no wildcards or regular expressions (yet!).

## Introduction

In Part 1 we covered how to make a very basic native Windows bootable image and Part 2 went further to show customisation methods. In this part I’ll discuss some further enhancements we can make, namely:

• Making a multi-image bootable

Over the (many) years that I’ve been making Windows bootables, frequently used for recovery purposes on physical devices, the need to add drivers to the images has reduced greatly since each Windows release seems to cope better with the array of PCs, Servers and laptops that I encounter (or perhaps there is less and less “exotic” hardware out there). However, given the extracted driver package for a specific device, e.g. storage controller or network card (note that wireless doesn’t work out of the box with Windows PE), it is incredibly easy to add the drivers to the bootable image once it has been mounted, to c:\temp\wimp in this case, which we covered in part 2. So to add the drivers simply run the following command once your wim file has been mounted:

Dism /Image:C:\temp\wimp /Add-Driver /Driver:C:\drivers\mydriver.inf

This should then parse the driver’s .inf file specified and copy all of the required files from the specified folder into the mounted image. If you have more than one driver to add and they are in a folder hierarchy then you can use a single command to add them all – use the /Recurse option instead of /Driver. If any of the drivers are unsigned then using the /ForceUnsigned option may help.

To see what 3rd party, as in non-Microsoft, drivers are in your image run the following:

Dism /Image:C:\test\wimp /Get-Drivers

Once you have finished adding drivers then remember to commit and unmount the image so that the changes you have made are written back to the .wim file and then make the bootable USB or ISO as before although for USB all you need to do is to copy the updated boot.wim file into the \sources folder.

Given that the out of the box experience with vanilla WinPE is a command prompt which not everyone is comfortable with, an easy way to provide a Windows style start menu is to use the excellent Nu2menu tool. It uses a simple XML configuration file to build a menu of tools – when adding a new tool I usually just copy and paste an existing entry and then modify it as necessary. There are sample XML snippets on the site and one in the download zip file. This then can look something like the following which also shows you the sorts of utilities that can be worth obtaining/purchasing to put into your recovery suite, such as the excellent Explorer++:

To launch nu2menu.exe automatically at boot, my startnet.cmd file simple has the following line:

start "" \programs\nu2menu\nu2menu.exe

Where \programs\nu2menu is a folder in the .wim image where I have copied the nu2menu executable, xml configuration file and the image to use for the start menu button itself which is in a file “nu2go.bmp”.

## Making a multi-image bootable

So now we have the knowledge to create an all singing, all dancing, Windows PE bootable that can be used for all kinds of things Windows wise but what if I’ve got a physical Linux system I need to do some offline servicing of? This is where we can add grub4dos to the existing bootable media, so USB drives have to be FAT32 format rather than NTFS, to allow other operating systems to be booted such as Linux and DOS by the following steps:

1. Rename “bootmgr” to “bootmgr8”
2. Extract Grub4Dos grldr and menu.lst to the root of the bootable media from here
3. Rename grldr to bootmgr
4. Modify the menu.lst file as required (see below)

For example, the following entry in menu.lst will give an option to boot your existing WinPE image:

title WinPE plus BartPE
find --set-root /bootmgr8
chainloader /bootmgr8

And the following will boot an Ubuntu (I used build 14.04, 64 bit) ISO image /Ubuntu.iso already present on the media:

title Ubuntu Live from ISO
find --set-root /ubuntu.iso
map --sectors-per-track=0 --heads=0 /ubuntu.iso (0xff) || map --sectors-per-track=0 --heads=0 --mem /ubuntu.iso (0xff)
map --hook
root (0xff)
kernel /casper/vmlinuz.efi boot=casper iso-scan/filename=/ubuntu.iso noprompt noeject noswap noapm nomodeset locale=en_GB
initrd /casper/initrd.lz

Note that as it is booting from the ISO which it loads into memory first, it can be quite slow to boot on physical systems and can spend several minutes at the stage below:

It does however have the advantage that because no files need to be extracted to the bootable media from the ISO then it can be updated with a newer ISO simply by copying any new ISO over the top of the old one on the bootable media.

To boot to DOS (very occasionally I need this for tasks like BIOS updates on very old systems), I have the following:

title DosFlashDisk
kernel /dos/memdisk
initrd /dos/idecdrom.img

Where the idecdrom.img file is a 1.44MB floppy disk image I created years ago from an actual floppy disk and memdisk comes from here. You may be able to obtain a FreeDOS image if you don’t have a bootable floppy disk and drive from which to make an image.

Note also that we can have multiple WinPE images booting from the same USB/ISO, e.g. 32 and 64 bit variants although I usually just have a 32 bit variant since that works fine on 64 bit systems, by using bcdedit to manipulate the \boot\BCD file on the USB stick (not in the .wim file itself). To show the current boot entries simply run the following (where G: is my USB stick mount point):

bcdedit /store G:\boot\bcd

We can then create new entries and modify them to point to other .wim files on our boot media. See here for a quick guide to using bcdedit for this but remember to always to also specify the /store option as well so that bcdedit manipulates the bcd file on the boot media and not your local Windows system!

That’s all for now folks – have fun making bootable media!