Chad
Fri Aug 22 00:03:56 CDT 2003
Hi Toby -
Luckily I was able to locate a previous post by Jeff on this very subject,
which originally appeared in the beta newsgroup. (I'm assuming there isn't
a problem reposting it here since Bobcat is RC'd - and this is actually more
about Windows 2003 than SBS 2003). I trust that Jeff will offer any
necessary clarifications / corrections if anything from this has changed
since the original post date.
------------------------------------------
Posted by: Jeff Middleton (SBS MVP)
Subject: My Bare Metal Restore Report
If you are looking for me to declare victory or defeat in black and white,
you are going to be disappointed. That's because I bring good news, and bad
news on this subject.
+ Bobcat Beta2 includes new functionality in the OS which provides the
ability for NTBackup to restore the original SFNs on all files, unless it
hits a condition with an exception not handled by the underlying API. This
is a 90% improvement from 0, but not good enough, IMHO.
+ Bobcat Beta2 still references SFNs in the registry to every primary folder
in the "c:\Program Files\Microsoft*.... " folder tree, there are four of
them, though I have not attempted to prove how to break the SFNs to see what
happens to the server. However, I can illustrate a simple, plausible
condition that will break a bare metal restore for this entire tree of
applications, or any similar condition anywhere in the file system with
similar conditions.
+ The SBS 2000 condition was unacceptable. The Bobcat Beta2 is barely
acceptable.
+ The problem here is once again, still, a Windows issue, not an SBS issue.
With this said, based upon Beta2, I will state:
- SBS2003 moves the rating for reasonable recovery from a 'bare metal
restore' up from 'not possible' to 'probable, but imperfect'. You need to
not break the restore by the process you use, therefore you need to know
what would break it.
- What has improved is that the historically predictable disaster recovery
practice of 100% backups to tape to recover a server returns up from a
rating of 'unacceptable' with SBS 2000 to 'probable with SBS 2003'. Stated
differently, if SBS servers were always used in practice the way we beta
test them, there's not problem. However, in the real world, the way that we
use server is likely to bring the SFN issue back in play, but in a smaller
percentage of the population meeting specific conditions we can predict.
- If nothing changes regarding the SFN handling with file by file
backup/restore with Win2003, I would strongly recommend one of the following
courses: (a) use drive imaging as your primary backup method for recovery,
supplemented by tape recovery as needed. (b) Do NOT do an in-place upgrade
of any prior version of SBS to get to SBS 2003, likewise Windows pre-2003 to
Windows 2003 (c) DEMAND that MS find a better way to document this issue,
what it means, what it impacts, when it is happening, and what to do about
solving it other than reinstalling applications following disaster recovery
or migration of hardware process.
- The nature of the remaining SFN flaws that can be predicted because they
are both plausible, prevalent, and not trapped by NTBackup, and that leaves
a restore from bare metal as a concept I would rate as "flawed, but
acceptable" because it is now possible to determine what flaws are
introduced, even though MS is doing nothing to identify that to you.
What I know of SFN related issues to date, and what I see revealed in Bobcat
Beta2 confirms to me that MS still doesn't see the issue at a root level,
they see it as a low-percentage risk at an application level, not one they
should bring to zero risk at the OS level. MS also doesn't yet see this
issue are deserving of an audit logging condition, rather it's something if
it's ignored, it will mostly remain a mystery of behavior in the OS, even
though it causes major heartaches in predictable conditions.
My number 1, highest priority recommendation to MS on this situation is that
MS should do two things with NTBackup (I know, pass it on to Veritas):
1. NTBackup now can restore the old SFN, therefore it has the ability to
determine with a verify process if the SFN originally stored is what was
established on the drive. This should be logged in the restore log itself.
2. NTBackup could provide a technique to identify prior to performing a
restore of files/folders if the results will contain any shifted SFNs, and
this should be included as a feature of NTBackup.
I would find it shocking if MS fails to act upon the first item, and I would
be shocked if MS delivered on the second. Those two points wouldn't even
solve the problem, just track it.
I find it unacceptable that a restore from 100% backup cannot return a
system entirely to the conditions that were recorded on the tape, the file
system present at the time of the original backup, and the certainty that a
100% backup will return you to a known good condition, more importantly
....a known condition.
Bobcat Beta2, because of Windows 2003, cannot return you to a "known good
condition" using file by file backup, it returns you to a "pretty much the
same, I'm probably not going to be bothered by the differences, and I'll
just reinstall anything funky" condition.
Alright, here are some details of what I have found in just about 2 hrs of
looking:
1. It appears that NTBackup 2003 will provide the means to do a bare
metal restore under ideal conditions, specifically in that the SFNs are now
preserved in the backup and are restored to original condition *when
building a new folder tree*. When creating the contents of a new folder, or
empty folder, the child files/folders are recreated with the original SFNs
recorded by the backup. However, when a collision occurs, not only does the
file responsible for the collision get a new SFN, so do all other associated
files/folders using the same "SFN root name" as they are restored,
essentially the condition Windows 2000, XP, NT, 95, 98 and ME did.
Therefore, with Windows 20003, instead of the undesirable certainty that
none of the SFNs would be preserved, we have the certainty that SFNs with no
collisions in the "SFN root name" will be preserved, otherwise not.
2. NTBackup 2003 will not repair SFNs that are present, but not matching.
In other words, if a condition is established where files/folders are
present on a drive that "are wrong", meaning, not consistent with a prior
condition you want to be returned to, you cannot use NTBackup to diagnose
this, identify it, or repair it....unless you either wipe the entire disk
clean, then perform an idealized restore.....or if you delete the entire
affected folder tree, then restore it from tape into "empty space".
3. The presence of any SFN matching file/folder during a restore will
cause a break of the SFNs associated for the entire set of files that share
a common "SFN root name" in that folder as they are restored. A comparison
after the restore will reveal that only coincidence will be responsible for
the correct SFN being assigned to any particular file in the affected set.
It's important to emphasize that this issue impacts only the handful of
coincidentally named files/folders in that one folder, and there's no impact
on the other files/folders in the same parent folder that don't share a
common SFN root name with them.
4. The changes present in the NTBackup 2003 manner of handling SFNs does
not appear to be also be applied in any other aspect of the Windows API
regarding other methods of file copy or create. Therefore, while a restore
from tape using NTBackup placing a file/folder tree in empty space would
faithfully reproduce the SFNs entirely, an XCOPY or Explorer drag/drop
cut/paste will always fail to reproduce the SFNs correctly. In addition, the
VBscript API calls I used from WSH 5.6 and earlier behave in the same
predictable manner as XCOPY, Explorer and every other means I've looked at
other than NTBackup restore in Windows 2003.
5. It appears that the Windows API has been modified so that a rename of
a parent folder does not cause a reset of the SFNs downlevel for entire
subtree of that folder. This is excellent news. It means that a folder
rename is reversible in most conditions, therefore certain Administrative
techniques/tests would no longer result in an irrevocable break of the SFN
tree.
6. As mentioned above, a redirected restore of a new file/folder tree
will also result in preservation of the SFNs. The integrity of the restore
is not dependent upon returning the files to the original location, only
dependent upon the coincidental presence of the existing file/folder tree
with similarly named LFNs that produce SFN collisions.
7. A restore using NTBackup 2003 to an existing tree of files/folders
which results in an append within that tree does not have a certain result
regarding SFNs if the existing tree and the files/folder being restored will
commingle branches together. The presence of existing files and folders can
cause a block, or shift in SFN naming, just as it did in Windows 2000/XP and
earlier.
Okay, what does this all mean?
What follows are the tests that I performed and the results of the tests, in
very simple summary explanation. Let me say first, at no time did I perform
a 100% bare metal test for the very simple reason that these test should
reveal.....that wasn't really a critical test to do. Instead, I tested the
compartmental conditions that occur in a bare metal restore process, without
actually getting involved with the sort of things that could well be changed
by MS in the course of Beta revisions to Windows 2003, namely: System State
restore and Directory Services Restores. I'm pretty sure that we can all
agree that I'm not qualified to test the minutia of System State or DSRM
conditions....they are too complex to do without additional documentation.
Steps to reproduce the results I've Found
------------------------------------------
Install Windows 2003 at a minimum, or to see the impact upon SBS 2003
specifically, complete the normal SBS 2003 Beta install. Make a 100% backup
of the system, or if you prefer a little faster test process, do a selective
backup of a folder tree that exists, or one you might create to "prove"
certain conditions.
Let's assume we have a machine with the entire SBS system installed on C:,
and an empty partition named E:.
To work with the normal "c:\program files" source folder, after an SBS 2003
installation, you will find at least 4 instances beneath "C:\Program Files\"
with a subfolder that starts with "Microsoft". Open a CMD prompt in the
"C:\Program Files\", run the command:
dir /x
This will show the file folder names including the LFN and the SFN. Note
that the display of the LFN column is by default in alphanumeric order, but
the SFNs with similar names are not.
If you do an NTBackup of this folder tree, then restore it to a different
partition with no existing folders, you will create a new folder tree with
identical SFNs, as we would hope.
Perform the similar test with XCOPY using a command similar to this:
xcopy c:\*.* e:\ /c/r/s/h/k/e/y/o/x
This will recreate the same LFN tree, but when you look at the "dir /x"
results, the SFNs will not match the originals from the c:\. This may not be
a surprise, but it proves that you don't want to use XCOPY to move files
that might have significant SFNs!
Now, to demonstrate how to break an SFN restore, do the following:
1. On Drive E, create a new folder named "E:\Program Files". (if you
have a similar folder already there from the earlier tests, you can simple
delete the entire contents, that won't matter.)
2. Now, create a new folder inside, below that folder named "Microsoft A
Folder", so the entire path will read "E:\Program Files\Microsoft A Folder".
Now, perform the same restore process of the "\Program Files" folder tree,
redirected to the E: location you have just "prepared". After the restore
completes, you will find that most of the folders with the similarly named
"Microsoft ...." now no longer have the same SFNs as in the original source
folder on C:.
Before going further, I want to emphasize that I'm not suggesting that you
would normally be doing a restore of the Program Files to a different drive.
What I'm demonstrating is that an important folder like the "Program Files"
can be restored normally to empty space, but with coincidentally named
existing contents that share a similar "SFN root name", the SFNs for the set
of files restored is altered.
Now, let me explain the meaning of my term "SFN root name". A simple
illustration would be the case of a long filename where the first 8
characters are identical, the first 8 letters might appear to be the similar
root name, but that's not correct for the deep meaning here. The deeper
meaning of the "SFN root name" is found by creating a LFN in an empty
folder, then looking at the SFN that is generated. That's the "SFN root
name". All SFN files/folders collide not based upon the LFN, rather what the
default SFN is generated in an empty folder condition. When two or more
files would independently generate the very same SFN in this condition, that
is the SFN root name. With this information, you can now determine what
individual files would collide with any other file because they all share
the common resolution of the SFN root name that is generated by default.
When two or more files/folders collide in a single folder, the incremental
change of name stems from there. In other words, you can have LFNs that
don't have the same first 8 characters, but do share a common SFN root name.
This has to do with the use of numbers, spaces, and punctuation.
Okay, there's another possible way to arrive at an SFN collision that will
break a restore. You can have a "SFN sequence gap". Suppose you have a
group of 4 files with a common "SFN root name", so they have their SFN names
assigned in sequence of the order of creation, what I refer to as Slot 1,
Slot 2, Slot 3, and Slot 4. If you now delete the three files in Slot 1-3,
what is left behind is the file named for the SFN in Slot 4. If you create a
new file now in this folder that collides with the "SFN root name" for this
file occupying Slot 4 position in the sequence, the new file actually
doesn't collide with the SFN of the existing file, but the "SFN root name"
collision triggers the break in SFN restore. In other words.......
By brief experimentation, I found that if you were to create the four
file/folders in a sequence I described just above, if you deleted any 3 of
the 4 files/folders, then now performed an NTBackup restore into this same
folder location, NTBackup will not attempt to preserve the SFNs on this same
set of files you just restored. Those SFNs will be assigned in alphanumeric
order by LFN. It's certainly possible that I don't understand everything
about what is going on here, after all, it took me longer to type up this
summary of observations than it did for me to do the test. Now maybe someone
else can help in proving if my tests were valid.
Final conclusion:
I believe that a 100% file by file backup of the OS should have the ability
to correctly return any file/folder from the backup media to the exact same
LFN, SFN, base attributes, security attributes.....everything the same as
what was recorded. When a restore is performed and this doesn't happen, it
should be reported as an exception. If it's possible to force the restore to
make the change, it should be an option.
Though not the heart of the problem, I do feel there is one additional
condition that is still being ignored by MS, and now, in the current
situation, NTBackup (any backup) would potentially resolve this problem 100%
if it were returned as a feature of Windows: 100% restore to a redirected
location. Just put it back as a feature!!!!!!
If we could again perform a 100% redirected restore, including System State,
to a partition other than the one hosting the OS running the machine to
perform the Backup/Restore process, then we would have the ability to
restore a machine entirely, completely, in the very same condition it was
recorded. A restore would once again be reliable, fast, and not require this
bullshit sequence of first installing an OS, then overwriting it, then
trying to decide if overwriting existing files has helped, hurt, or not
mattered to the process. Even if this process were provided to only support
"next boot" recovery where you would then restore System State on top, this
would be an improvement in process and reliability. Files are files, why
can't we simply restore them as files?????
--
Chad A Gross
Lerman's Law of Technology: Any technical problem can be overcome
given enough time and money. Corollary: You are never given enough
time or money.
Toby Watson wrote:
> Hi Chad,
>
> From looking around it does appear that if the SFN is in use it's a
> no-go - which I suppose makes a bit of sense. However, I agree with
> you that a bare metal restore now seems possible - I guess this is
> the main issue with SFN problems. I can see some problems with
> restores of documents, for example if you have spreadsheets linked by
> SFNs and the like and you want to restore just one file from backup
> overwriting the existing one, you will still have problems and the
> links between spreadsheets will be broken...
>
> If he's about, I look forward to Jeff's comments though :-)
>
> Toby.
>
> "Chad A Gross" <chad.gross@laytonflower.nospam.com> wrote in message
> news:O7uYGiFaDHA.1748@TK2MSFTNGP12.phx.gbl...
>> Hi Toby -
>>
>> As usual, we're going to need Jeff for the details - but the SFN API
>> in Win2k3 (and SBS2k3) is an improvement, but not the end-all,
>> cure-all fix we're hoping for. IIRC, Jeff's analysis of the SFN
>> restore function in SBS2k3 was that the restore process would
>> request the original SFN - however, there was no method for handling
>> the scenario where the requested SFN was already in use. So, the
>> bare-metal restore is now possible, but other disaster recoveries
>> that do not require a bare metal restore still have the potential to
>> be ugly.
>>
>> --
>> Chad A Gross
>>
>> Lerman's Law of Technology: Any technical problem can be overcome
>> given enough time and money. Corollary: You are never given enough
>> time or money.
>>
>>
>>
>> Toby Watson wrote:
>>> In the past couple of weeks since seeing a post on the subject I
>>> have been fretting over the SFN issue on restoring from backups
>>> (anyone
>>> else interested in this would do well to read the many knowledgeable
>>> posts by Jeff on this issue).
>>>
>>> My search has revealed that there is *possibly* some light at the
>>> end of the tunnel for SBS 2003. It seems that microsoft provides a
>>> new
>>> API to restore short file names for Windows Server 2003 (and
>>> apparently also in XP) - see MSDN
http://tinyurl.com/kslt . I got
>>> this pointer after looking at the Dantz Retrospect forums and found
>>> this KB article
http://tinyurl.com/kslw . This says this new API
>>> will be supported in the next major release of Retrospect, however
>>> it is actually referring to 6.5 which is out now and apparently
>>> supports
>>> this API - see
http://tinyurl.com/ksm0 .
>>>
>>> Now, I have no idea whether this works in practice but I thought it
>>> was interesting, and I have not looked to see what the situation is
>>> with Backup Exec (or even NTBackup) etc., but it seems promising.
>>> So, my question is whether SBS2003 supports this API (I presume it
>>> does since it has Win2003 Server underlying)...
>>>
>>> Any of you knowledgeable folk care to comment?
>>>
>>> Toby.