NTFS 8.3 short names – solving the issues

In Part 1 I explained the fundamentals of NTFS 8.3 short names and the typical sorts of problems that they can cause which are mostly when short name creation has been disabled or when a short name is not preserved when a file is copied and the new copy is referred to by its original short name.

Ok, so how do short names get disabled? This TechNet article covers the values for the registry value “NtfsDisable8dot3NameCreation” which controls the short name behaviour. In the days of old, as in Windows XP and 2000, this value was by default 0, meaning that short name creation was not disabled so it was enabled (which is the default for new installations, to this day, when original Microsoft installation media is used). In those days, the only other legal registry value was 1 which meant that short name creation was disabled on all local volumes, which needed a reboot to take affect. In later operating systems, further values have been introduced to make the settings per volume (2) or to all volumes except the system drive (3) and, since Vista, a reboot is not required to affect a change.

Disabling of short names has become quite common place since Microsoft recommend to do so in a security guide found here. There’s also some evidence that file operations in folders containing large numbers of similarly named files can be slowed down dramatically by short name creations as the OS has to figure out unique names for each newly created file/folder.

In addition, the “format” command takes an optional /S: argument to either enable or disable short names and the default is now to disable so any Windows OS installed to a manually formatted partition, e.g. via ADK/AIK, will have short names disabled. I have seen this with systems created via SCCM 2012 which then causes problems with software that uses appinit_dlls.

To check if short names are disabled on a given volume, run the following as an administrator in a cmd prompt:

fsutil 8dot3name query c:

which will return something like the following if short name creation is disabled:

The volume state is: 1 (8dot3 name creation is disabled).
The registry state is: 2 (Per volume setting - the default).
Based on the above two settings, 8dot3 name creation is disabled on c:

You could of course also create a new file with a space in the name (remembering to use quotes around the file name) and then run “dir /x” against it to see if a short name is created or not. This approach has the advantage that it does not require administrative privilege.

Enabling it is then just a case of running the following:

fsutil 8dot3name set c: 0

however, any file or folder created when short name creation was disabled will still have no short name since they are only created at initial file/folder creation, not on subsequent accesses to the file even if short name creation has subsequently been enabled.

All is not lost as we can add short names for items that are missing them using  the fsutil command again, for example:

fsutil file setshortname "c:\program files" PROGRA~1
fsutil file setshortname "c:\program files (x86)" PROGRA~2

This approach can also be used to “repair” short names when they are incorrect, such as my robocopy example in Part 1. Note that if you are switching around two short names for two files/folders, such as in the above example, then you will need to use a temporary short name for one of them since short names must be unique per folder at all times:

Notice in the above that the temporary short name I used was “SHORTY” which doesn’t include the “~” character since short names don’t actually require it, it’s just that the Microsoft algorithm used to generate short names uses it. You could in theory therefore run with the short name as “SHORTY” but only if there are no existing references to the expected short name, “PROGRA~1” in the above example, such as in the registry.

The above works fine for small numbers of files/folders, such as items that need to work in the good old appinit_dlls registry value but for large scale fixing of missing or incorrect short names then it would just be a little bit tedious to do this manually. Therefore I have written a utility that will copy the short names from existing files/folders and apply them to a copy of these files/folders in a new location. The idea is that you have used robocopy or similar to (selectively) copy files to a new location but as covered in Part 1, this will likely wreck the short names so that programs, like Office, may no longer work.

So if we have recursively copied say “c:\program files (x86)” to d:\ then run the following to apply the same short names from c: to d:

CloneShortNames.exe -s "\\?\c:\Program Files (x86)" -d \\?\d:\

Notice the use of the “\\?\” sequence to disable path parsing, as described here, to enable long file names to be used, up to 32K characters, rather than hitting the usual MAX_PATH (260) character limit. It’s optional but a good idea, where utilities support it.

You can also specify a -v to get verbose output of what is going on and -c to continue should any errors occur. It should be run as an administrative user.

It is available here, is used entirely at your own risk since it is completely unwarranted, and requires the Visual C++ 2013 redistributables which are available here.

NTFS 8.3 short names – a primer

If you’re not aware of what an 8.3 short name is then this article probably isn’t for you but feel free to carry on as it might be something that will affect you in the future. If you’ve ever had issues with any software product that uses the “legacy” Microsoft injection technology implemented in user32.dll via the “appinit_dlls” registry value (and if you ever call it a key I’ll never speak to you again!) then you have probably encountered 8.3 short names, possibly without realising it if everything worked ok. In their wisdom, Microsoft decided to not only use a comma character as a delimiter in the appinit_dlls value when multiple dlls are to be loaded but they also recognise a space as a delimiter. Yes, a space. But hold on a minute, isn’t a space now a valid character in a file name, like “Program Files” and “Program Files (x86)”? Yes it is so if we need to have multiple dlls listed in appinit_dlls (e.g. Citrix and AppSense hook dlls) then there needs to be a method to achieve this where the dlls are not in the standard system path, e.g. system32. Quoting the path(s) containing spaces, as you would in a command prompt and for file arguments to programs, does not work so we are left with having to use 8.3 short names.

8.3 short names, which are a feature of NTFS, are a throwback, sorry “compatibility feature”, to the early days of the FAT file system where filenames had to be in strict 8.3 format and couldn’t contain special characters such as spaces. To maintain compatibility with older (e.g. 16 bit) programs, a feature of NTFS, which is enabled by default is to create an 8.3 short name for a file or folder when it is created so that these legacy programs can access long file names by way of the 8.3 short name.

So let’s see this in action:

The /x argument to dir shows us the short names, if they exist. Note that the folder “Short” does not have a short name since it is already 8.3 compatible. The short names for the other folders which aren’t 8.3 compatible are generated in the order that the folders themselves are created so “Program Files”, since it is the first Program* folder created gets the “~1” and “Program Files (x86)” gets “~2” as it is the second created. Notice that the last folder created doesn’t get a “~4” but some seemingly random name instead which shows that you can’t predict short names.

Let’s take a look at a folder on a live system:

I’ve used the /od argument to sort by date and /tc to show the creation date attribute which strives to show that the short names are kind of related to the creation date in terms of the ~1, ~2, etc. although even I’ve no idea why the Visual Studio folder has the .0 extension!

This illustrates one of the two common problems with 8.3 short names – namely, pardon the pun, that if I restored these folders from a backup to a new “c:\program files” folder then the 8.3 short names might be different depending on the order that the new folders were created in.

The other problem is when 8.3 short name creation has been disabled, which we’ll cover in the next part, such that when a file/folder is created, it does not get a short name created at all so if it is required for appinit_dlls then it won’t work without some kind of manual intervention. It’s not just appinit_dlls though – ever heard of a productivity package called “Microsoft Office”? Interestingly, when you install the 64 bit version of Office 2010, it puts nearly 1600 registry values in HKLM in 8.3 short name format. But can this cause problems I hear you say? Yes, in a word. A while back, I decided to use robocopy to selectively copy some of the contents of a larger capacity hard drive to a smaller SSD, which included the Office installation folders, and afterwards none of the Office programs worked when run from the SSD. I eventually diagnosed this as being due to incorrect short names in the copied folders because these were created in the order that robocopy processed them, not in the order that they were originally created on the source drive so the short names in the registry, which hadn’t changed, were wrong and didn’t refer to files that existed on the SSD.

That’s all for this part – in the next part I’ll cover the ways that we can fix 8.3 short name problems.

Making a native Windows bootable USB stick (Part 2)

In part 1 I explained how to make a very basic bootable USB or ISO using the Microsoft Automated Deployment Kit. However, when booted that just gives a command prompt, often erroneously referred to as a “DOS prompt” (16 bit apps won’t even run on a 64 bit OS!), which although powerful in its own right, particularly those of us who shun GUIs and embrace command lines, doesn’t give a great user experience. I will therefore cover how we can customise the image which can be done for a variety of reasons:

1. Add required drivers (needed less often these days due to the large driver set provided by Microsoft)
2. Add startup scripts to give information about the environment such as disk volumes and network configuration (but not wireless (yet))
3. Add extra tools (e.g. defragmentation, backup , anti-virus, etc.)

The first thing we have to do is to mount the boot image (WIM) file so that we can manipulate it. If you have ever installed an operating system from Vista onwards then you will already have used a WIM file as this is usually the largest file on installation media and is effectively a file system within a file. See this link for a detailed explanation if you are so inclined.

To mount the WIM file we use the multi-talented DISM.exe (Deployment Image Servicing and Management) tool in the Deployment and Imaging Tools Environment administrative command prompt that I introduced in part 1. We must first create a folder in a local file system which will become our mount point, which is “c:\temp\wimp” in my examples below. So now we run:

dism /mount-wim /wimfile:"c:\WinPE\media\sources\boot.wim" /index:1 /MountDir:c:\temp\wimp

Which should then give us some folders in the mount point as shown below:

We can now add what we want to this folder hierarchy and once we’re done we simply unmount it, again using DISM.exe, and then create the bootable image again, exactly as described already in part 1.

Note that if you are adding files which you might want to update frequently, such as pattern files for an offline anti-virus scanner, like Trend Micro’s Sysclean, then I tend to not add these to the WIM file put just keep them in a folder on my bootable USB stick.

Now onto the customisation. There are a number of official ways to customise the image but by far and away the easiest in my opinion is to edit the c:\temp\wimp\Windows\System32\startnet.cmd script file which is what is invoked when the WIM file is booted. By default it just contains the “wpeinit” command which initialises the WinPE environment.

My usual startnet.cmd file looks like this:

@echo off
title Guy's Win8.1 x86 WinPE environment
color 0e
echo Initialising ...
wpeinit
\programs\bginfo\bginfo.exe \programs\bginfo\bginfo.bgi /accepteula /silent /timer:0
ipconfig
diskpart /s %SystemDrive%\programs\tools\diskpart.txt
start "" \programs\nu2menu\nu2menu.exe

Which results in the following at boot:

Where BGInfo is the excellent Microsoft/SysInternals Background Info tool that displays system information, such as processor, memory and disk information, as the wallpaper. The file bginfo.bgi is a config file that I saved previously from having running BGInfo interactively and saved the settings to this file.

The diskpart.txt file is just a list of commands to run the diskpart tool such that it will show us information about the hard drives on the system. It contains the following:

list disk
list vol
exit

The \programs folder is one I have created myself and then added in all of the extra, third party, tools that I want to use in the booted image. We’ll cover how to hook this into the excellent NU2menu tool in the next thrilling instalment. Also, look out for information on how we can extend the capabilities of the bootable so that it will also boot live Linux distributions and even DOS all from a single USB stick or ISO.

It’s a good idea to not use driver letters in scripts, etc. because although the system drive is usually X:, I hate making assumptions/hard coding so I either don’t put a drive letter in at all, as above, or use the %systemdrive% environment variable. Be aware though that anything that you actually change or add to this X: drive will be lost as soon as you shutdown, since it is dynamically created from the WIM file, so use a persistent drive, such as a USB stick, if necessary to maintain a file, etc.

Don’t forget to unmount the WIM file when you have finished customising it and before you make any bootable images from it. Ensure that no command prompts, explorer windows, etc. are in your c:\temp\wimp folder, or subfolders, and nothing has files open in this folder otherwise the image will not unmount properly. Then run:

dism /unmount-wim /mountdir:"c:\temp\wimp" /commit

Note that if you want to discard your changes then specify the /discard option in place of /commit.

Lastly, I always put file/path arguments in double quotes not only in case there are spaces in them but also so that file name completion works, by default via the <TAB> key, so the chances of a typo in a path name are greatly reduced.

Making a native Windows bootable USB stick (Part 1)

This is in fact incredibly easy and I find it invaluable to keep a bootable USB stick to hand for various purposes including:

1. Deleting temporary files, etc. that would otherwise be locked or difficult to access
2. Taking offline backups of files, e.g. “software” or “system” registry hives, or whole systems (make your backup USB hard drive bootable too using the exact same technique)
3. Resetting forgotten passwords (via temporary replacement of sethc.exe)

In part 1, I’ll cover creating a basic bootable image that can be either built into an ISO for use in virtual environments, or burnt to disc, or copied to USB stick and made bootable. In later parts, I’ll show how we can make the image much more customisable and user friendly – much like the old, but certainly not forgotten, BartPE which is how we had to do things before Microsoft made the WinPE tools available to the masses.

The steps:

• Install the Windows ADK (Assessment and Deployment Kit, formerly the Automated Installation Kit (AIK)) which is available here
• Run the “Deployment and Imaging Tools Environment” shortcut that gets created in the Start Menu. Run this with administrative privileges which should give a command prompt.
• Now we need to copy the required files to a working area so that we can customise them if required. So locate the “copype.cmd” batch file, which on a 64 bit system with the default installation path is in “C:\Program Files (x86)\Windows Kits\8.1\Assessment and Deployment Kit\Windows Preinstallation Environment” and run it with the following parameters:
copype.cmd   x86   c:\winpe
• This should then have created the following folders, with files in some of them, in the destination you selected (c:\winpe in the example above):
fwfiles
media
mount
• Now we have the files that can be copied onto a USB stick or drive, which doesn’t have to be empty, and then we just need to make it bootable so copy all folders and files from the “c:\winpe\media” folder to the root of your chosen USB stick, maintaining the same folder structure. The files are barely 200MB in total so you don’t need a particularly large capacity USB stick. Note that there are a fair few locale folders, e.g. “en-gb”, so you should only need to copy the one(s) you need, which is typically “en-us” for US English. For instance, the root of my USB stick looks like this:

• Once copied, we just need to make sure the USB stick is bootable which is achieved by running the following command in the administrative command prompt, where “G:” is the drive letter of my USB stick:
bootsect /nt60  G: /mbr

If you have multiple partitions on your USB stick, you will also need to ensure that the partition where you’ve copied the boot files to is the active partition. This can be done via the Disk Management snapin (diskmgmt.msc), right clicking on the partition and selecting “Mark Partition as Active”.

That’s it, you should now have a bootable USB stick, as long as your system is USB bootable – sometimes this has to be enabled in the BIOS, the boot order changed so USB comes before the hard disk(s) or, frequently, there is an option immediately after power on to get a one-time boot menu where you can then select to boot off the USB stick.

If you don’t mind losing the contents of the USB stick then instead of copying the files to the stick and making it bootable, you can just use the “MakeWinPEMedia” command in the same command prompt with the /ufd paramter.

To make a bootable ISO image, that can then either be used to boot up a virtual machine or burnt to CD to boot a physical system, run the following command in the same command prompt:

MakeWinPEMedia   /ISO  c:\winpe c:\temp\winpe.iso

When you boot though, you will eventually just get a command prompt, albeit one that has access to all the local file systems and tools built in to the image such as diskpart.exe and net.exe. Whilst a lot can be done from this command prompt, as long as you know the commands you need, such as to copy files off an unbootable system, it’s not very intuitive so in the next instalment I’ll cover how to customise the boot image to give a quasi start menu.

Troubleshooting Unknown Devices and Drivers that Refuse to Install

Ever had an “Unknown Device” in Device Manager and ended up spending a long time trying to find drivers for the device? This can be particularly difficult on laptops where the vendor has put in hardware that you don’t even know about and weren’t necessarily expecting.

Fortunately, it is actually quite easy, usually, to troubleshoot these situations and know what drivers to search for by looking at the hardware id in the details in Device Manager and then searching the web for that term.

Note that the “VEN_” tells us the vendor, “NVIDIA” in the example above, as can be found at http://www.pcidatabase.com and the “DEV_” is the device. Incidentally, vendor “VEN_8086” is Intel which always makes me chuckle! I usually don’t need to bother searching for the “SUBSYS_” or “REV_” strings since “VEN_” and “DEV_” is enough to identify the vendor and device and thence driver. Obviously it is always best to get the drivers from the Vendor’s website rather than a third party site since these latter drivers could contain malware, etc.

However, sometimes even with the seemingly correct driver package it still does not cure the problem. For instance, I recently had a battle with the drivers for an Epson printer. I knew I’d win but it was how long it would take and how much hair I would lose in the process!

I started with getting the latest driver package from the vendor’s web site rather than using the bundled CD as you never quite know how long these have been sat in a stock room somewhere. However, after installing the drivers, the USB attached printer was still showing as an unknown device. Even trying to update the driver, via Device Manager, and pointing it at the folder where the .inf file resided did not rectify the situation.

I then looked in the .inf file for the hardware id since the way Windows finds the right .inf file for a device is to look for an .inf file with that hardware id in it (see http://msdn.microsoft.com/en-us/library/windows/hardware/ff549520(v=vs.85).aspx). What I noticed was that the hardware id wasn’t listed despite the .inf file being the right one according to the descriptions in the file, i.e. had the name of the printer model I was trying to use in it. An easy way to search .inf files for a given id from a cmd prompt is:

findstr /i /c:”PCI\VEN_10DE^&DEV_0A6C” %systemroot%\inf\*.inf

Note the escaping of the ampersand (&) above otherwise the batch processor will treat the command line as two separate commands and will not run properly. Due to a limitation in “findstr” (and “find”), the above won’t work for Unicode files so instead I use Strings.exe from SysInternals/Microsoft in the following way which works for both Unicode and ASCII files:

strings.exe %systemroot%\inf\*.inf | find /i “PCI\VEN_10DE&DEV_0A6C”

I use “find” instead of “findstr” as I frequently get “out of memory” errors when using findstr in this pipelined way.

What I then did was to take a copy of the .inf file and add in the “missing” hardware id such that Windows now accepted it as being the “correct” .inf file for this device. It did then fail on some of the subsequent copying of driver files but that was soon rectified by creating a new, temporary, folder with a copy of this .inf file in, and then copying into there all of the missing files, which were dotted around the system in such places as the driver installation package folder and %systemroot%\ System32\spool. Identifying the missing files was achieved with a mixture of SysInternals/Microsoft Process Monitor running during the driver install and entries in the file %systemroot%\inf\setupapi.dev.log.

This setupapi.dev.log is “quite interesting” in that it keeps a record of all device installations, and statuses, from the birth of the specific computer. Think someone may have inserted a USB stick to “steal” something when you left your system unlocked (tut, tut)? Then either search for that date/time in the setupapi.dev.log file or look for the hardware id in this file, e.g. search for “Cruzer” if you think that the device was a SanDisk Cruzer flash drive, for instance.

Displaying text from a (log) file at a specific offset

When analysing product behaviour here at AppSense we often run SysInternals Process Monitor (procmon) to understand what is going on behind the scenes. In the procmon trace, we will see our log files being written to but all that is shown in the trace is the offset into the file that the data is written to and how much is written which, on the whole, is not that useful. What we sometimes need to know is exactly what text is written to the log file at this point as it will help us to correlate what procmon tells us with what our product’s log file tells us.

In order to be able to perform the correlation, I wrote a utility, called FindFileEntry.exe, which takes an offset (-o), a length (-l) and text log file name (-f) and will output the line number of the file and the text (up to “length” characters) . It will work with both Unicode and ANSI/ASCII text files.

For instance, if we consider the following, somewhat contrived, snippet of a procmon trace:

We can see that the AppSense Application Manager Agent process (AMAgent.exe) has written 88 bytes to the log file “C:\Temp\amlogs\demo\Application Manager_AGENT_06_11_2013_21_52_28.log” at an offset of 22,252,941 bytes. We therefore run the following options with the FindFileEntry utility:

c:\>FindFileEntry.exe -f "c:\temp\amlogs\demo\Application Manager_AGENT_06_11_2013_21_52_28.log"
-o 22252941 -l 88

And it outputs the following where the number immediately before the first colon character is the line number within the file

206504:     T031268 693093390 21:54:21.502 [AMPipeServer::Run] Event signalled. waitValue = 22

Given that we now have the line number as well as the text written, we can open the log file in a text editor such as Notepad++ so that we can see what other lines of (useful) debugging information were written around that same period.

PS. Wouldn’t it be nice if procmon had some kind of customisable capability to allow an event to be right clicked on, or similar, and an arbitrary program be selected to run, with parameters such as parts of the procmon trace line, e.g. to launch the FindFileEntry program seamlessly?

Capturing transitory files before they are deleted

Sometimes you need to take a copy of a file that another program creates when it runs and subsequently deletes but it may well have deleted it before you get a chance to take a copy. A colleague was struggling with such an issue recently so I suggested temporarily changing the permissions on the folder where the file was created such that the file could be created but not deleted. This should then allow the file to be copied at your leisure since the creating program should be unable to delete it (unless it manipulates the file’s Access Control List (ACL) itself but this is unlikely).

Owing to my age, I like command lines, although I shudder if anyone dares to call a cmd prompt a DOS prompt, and in this instance it’s much easier to show what we need to do to secure the folder which is:

icacls %temp% /deny appsense\ivan:(CI)(OI)(DE,DC)

This creates a deny Access Control Entry (ACE) in the ACL for the folder, sub folders and files where the file is created, which you can figure out with good old Process Monitor (procmon). The “OI” and “CI” are Object Inherit and Container Inherit respectively and without these specified the ACE would only be added to the folder and not be inherited by anything created in there. The “DE” and “DC” are Delete and Delete Child respectively which are the special permissions which we are explicitly denying and as we all know in Windows security it’s the most restrictive permission that wins so a Deny ACE beats an Allow ACE. We assume that there is already an ACE in the folder’s ACL that allows the user, “ivan” in this example, to create files – the user’s %temp% folder, for instance, would normally have a Full Control Allow ACE for the user to which it relates.

Once the required file(s) have been moved elsewhere, the ACE added above should be removed by running:

icacls %temp% /remove:d appsense\ivan

Note that as long as the user “ivan” already has permission to change the ACL on the required folder then the above icacls commands do not need be run elevated. You can run icacls with just the folder name as an argument to see what the current ACL contains.

Before the more pedantic amongst you contact me, I should point out that I know I ought not refer here to an ACL but instead to to a DACL (Discretionary Access Control List) but most of the time when we talk about security of Windows objects the “D” is silent. After all, the utility is not called icdacls.exe! Also, don’t confuse a DACL with a SACL (System Access Control List) which  contains audit settings, which you viewin Explorer’s security dialogues for NTFS objects for example.

See here and here for further discussions on DACLs, SACLs, ACEs and other Windows security concepts.