Testing MS Security Patches?

Does anyone know of specific tests that should be performed before
implementing MS security updates on production systems.

I am a member of a large organization and we are trying to enhance our
testing procedures before implementing MS security patches through out our
production environments.

Is there a document that shows what specific things should be considered?

I'm looking for a guideline that displays how to validate that a security
patch released by MS will not break any applications or the Operating System.

PA
Fri Jan 27 14:27:52 CST 2006

All Windows Updates are thoroughly tested in a wide range of environments
prior to being released. Contact your local MS representative to see if you
might participate in this testing program.
--
~Robear Dyer (PA Bear)
MS MVP-Windows (IE/OE, Shell/User, Security), Aumha.org VSOP, DTS-L.org

y2kbug_s97 wrote:
> Does anyone know of specific tests that should be performed before
> implementing MS security updates on production systems.
>
> I am a member of a large organization and we are trying to enhance our
> testing procedures before implementing MS security patches through out our
> production environments.
>
> Is there a document that shows what specific things should be considered?
>
> I'm looking for a guideline that displays how to validate that a security
> patch released by MS will not break any applications or the Operating
> System.

Roger
Fri Jan 27 17:26:09 CST 2006

Your validation needs to be accomplished relative to the essential
applications,
services, and capabilities in your environment.
You could start by developing the list of essentials per your environment.
Then, with a test harness (which might be quite expensive to make "real")
that represents these capabilities of your production environment you would
need to validate the list of essentials after injecting the patches into the
test
machines.
All is much harder done that sketched.

--
Roger Abell
Microsoft MVP (Windows Server : Security)

"y2kbug_s97" <y2kbugs97@discussions.microsoft.com> wrote in message
news:178C0A5D-51D0-46C6-8243-C3193B0C4871@microsoft.com...
> Does anyone know of specific tests that should be performed before
> implementing MS security updates on production systems.
>
> I am a member of a large organization and we are trying to enhance our
> testing procedures before implementing MS security patches through out our
> production environments.
>
> Is there a document that shows what specific things should be considered?
>
> I'm looking for a guideline that displays how to validate that a security
> patch released by MS will not break any applications or the Operating
> System.

reb93720
Sat Jan 28 18:58:26 CST 2006

Here is a MS article that discusses patch mgt -
http://www.microsoft.com/technet/security/topics/patchmanagement/secmod193.mspx

I would recommend

1. Working with the respective business units and IT dept to dentify the
critical apps within your in environment.

2. Map out key functions that are required for each of the applications to
work correctly (i.e. MS Word - open a word document, cut & paste info, create
a table, import graphics, close word, etc).

3. Pay particular attention to the MS bulletins and what functionality /
features / files are affected and adjust accordingly.



"y2kbug_s97" wrote:

> Does anyone know of specific tests that should be performed before
> implementing MS security updates on production systems.
>
> I am a member of a large organization and we are trying to enhance our
> testing procedures before implementing MS security patches through out our
> production environments.
>
> Is there a document that shows what specific things should be considered?
>
> I'm looking for a guideline that displays how to validate that a security
> patch released by MS will not break any applications or the Operating System.

alun
Sun Jan 29 17:00:07 CST 2006

In article <178C0A5D-51D0-46C6-8243-C3193B0C4871@microsoft.com>,
"=?Utf-8?B?eTJrYnVnX3M5Nw==?=" <y2kbugs97@discussions.microsoft.com> wrote:
>Does anyone know of specific tests that should be performed before
>implementing MS security updates on production systems.

There are no general recommendations to suggest here, because your goal should
be to test those applications on which your business depends. If your
business uses the same applications that everyone else uses, then it's
entirely likely that they've already been tested by Microsoft. If your
business uses different applications from others, then those are the
applications you need to test.

>I am a member of a large organization and we are trying to enhance our
>testing procedures before implementing MS security patches through out our
>production environments.

There's nothing specific about these being security patches - they are changes
you are making to the system, and you need to ensure that they don't impact
the rest of the business' operations in a way that will cause more damage than
the problem that they are aimed at fixing.

>Is there a document that shows what specific things should be considered?

You need to analyse your LOB (line-of-business) applications, and that's
really something that you need to establish for yourself. Establish a set of
functions that are required for most of your operations, and a means to test
that they happen correctly.

Then use that sequence on every change you make, and add to it any operations
that appear to be related to the parts of the system that have changed.

>I'm looking for a guideline that displays how to validate that a security
>patch released by MS will not break any applications or the Operating System.

I'd say it's a fairly good bet that Microsoft has already done more testing
than you'll be able to in regards to making it not break the OS, so you should
concentrate on those applications that are required for your business'
continued operations.

But, there are a few things you should consider:

1. Download the patch.

2. Test the install on a lab machine / machines that represent the majority of
your users.

3. Test the uninstall - when you roll out the patch to users, if it kills
their systems, you need to know that the uninstall is going to work.

4. Test the installed patch in your lab against some sample use cases. Engage
in a feedback loop with your users to discover the important use cases -
remind them that if they experienced significant problems with a previous
patch, that was because you didn't test what they need to do against the patch
in the lab. If that was because they didn't tell you what they need to do,
then they need to be more detailed next time.

5. Determine who your rollout will start with - pick a group of users that
will be first-adopters for the next patch. There are different theories on
this - you can find people who are advanced users, so they can cope with
instructions to resolve issues; you can pick people at random, to make a good
statistical selection; you can pick people that you know are trouble to any
piece of software they encounter; you can pick people that you like, you can
pick people that you hate. Okay, that's a piece of humour, I don't suggest
that you do exactly that - but pick a strategy, or a set of strategies. I
suggest informing the test group that they are to be used as a test case for a
new update that's rolling through, and that you need them to let you know
anything that's changed.

6. Review the success of the initial rollout in meetings with stakeholders in
those departments, and then roll out to the rest of the company if the result
is good - or if the anticipated result of not rolling it out is worse.

Alun.
~~~~

[Please don't email posters, if a Usenet response is appropriate.]
--
Texas Imperial Software | Find us at http://www.wftpd.com or email
23921 57th Ave SE | alun@wftpd.com.
Washington WA 98072-8661 | WFTPD, WFTPD Pro are Windows FTP servers.
Fax/Voice +1(425)807-1787 | Try our NEW client software, WFTPD Explorer.

y2kbugs97
Mon Jan 30 07:17:29 CST 2006

Thank you all for posting your comments. I can tell from your comments that
Microsoft probably does not publish general guidlines for testing patches but
your comments will help me mold a solid plan in our specific environment.

Thanks again...

"Alun Jones" wrote:

> In article <178C0A5D-51D0-46C6-8243-C3193B0C4871@microsoft.com>,
> "=?Utf-8?B?eTJrYnVnX3M5Nw==?=" <y2kbugs97@discussions.microsoft.com> wrote:
> >Does anyone know of specific tests that should be performed before
> >implementing MS security updates on production systems.
>
> There are no general recommendations to suggest here, because your goal should
> be to test those applications on which your business depends. If your
> business uses the same applications that everyone else uses, then it's
> entirely likely that they've already been tested by Microsoft. If your
> business uses different applications from others, then those are the
> applications you need to test.
>
> >I am a member of a large organization and we are trying to enhance our
> >testing procedures before implementing MS security patches through out our
> >production environments.
>
> There's nothing specific about these being security patches - they are changes
> you are making to the system, and you need to ensure that they don't impact
> the rest of the business' operations in a way that will cause more damage than
> the problem that they are aimed at fixing.
>
> >Is there a document that shows what specific things should be considered?
>
> You need to analyse your LOB (line-of-business) applications, and that's
> really something that you need to establish for yourself. Establish a set of
> functions that are required for most of your operations, and a means to test
> that they happen correctly.
>
> Then use that sequence on every change you make, and add to it any operations
> that appear to be related to the parts of the system that have changed.
>
> >I'm looking for a guideline that displays how to validate that a security
> >patch released by MS will not break any applications or the Operating System.
>
> I'd say it's a fairly good bet that Microsoft has already done more testing
> than you'll be able to in regards to making it not break the OS, so you should
> concentrate on those applications that are required for your business'
> continued operations.
>
> But, there are a few things you should consider:
>
> 1. Download the patch.
>
> 2. Test the install on a lab machine / machines that represent the majority of
> your users.
>
> 3. Test the uninstall - when you roll out the patch to users, if it kills
> their systems, you need to know that the uninstall is going to work.
>
> 4. Test the installed patch in your lab against some sample use cases. Engage
> in a feedback loop with your users to discover the important use cases -
> remind them that if they experienced significant problems with a previous
> patch, that was because you didn't test what they need to do against the patch
> in the lab. If that was because they didn't tell you what they need to do,
> then they need to be more detailed next time.
>
> 5. Determine who your rollout will start with - pick a group of users that
> will be first-adopters for the next patch. There are different theories on
> this - you can find people who are advanced users, so they can cope with
> instructions to resolve issues; you can pick people at random, to make a good
> statistical selection; you can pick people that you know are trouble to any
> piece of software they encounter; you can pick people that you like, you can
> pick people that you hate. Okay, that's a piece of humour, I don't suggest
> that you do exactly that - but pick a strategy, or a set of strategies. I
> suggest informing the test group that they are to be used as a test case for a
> new update that's rolling through, and that you need them to let you know
> anything that's changed.
>
> 6. Review the success of the initial rollout in meetings with stakeholders in
> those departments, and then roll out to the rest of the company if the result
> is good - or if the anticipated result of not rolling it out is worse.
>
> Alun.
> ~~~~
>
> [Please don't email posters, if a Usenet response is appropriate.]
> --
> Texas Imperial Software | Find us at http://www.wftpd.com or email
> 23921 57th Ave SE | alun@wftpd.com.
> Washington WA 98072-8661 | WFTPD, WFTPD Pro are Windows FTP servers.
> Fax/Voice +1(425)807-1787 | Try our NEW client software, WFTPD Explorer.
>

Roger
Mon Jan 30 09:14:37 CST 2006

On the contrary, MS has issued very much info on the topic
http://search.microsoft.com/results.aspx?mkt=en-US&setlang=en-US&q=patch+management
We were just trying to give the essentials.

"y2kbug_s97" <y2kbugs97@discussions.microsoft.com> wrote in message
news:236F2A1A-DD4B-4152-A2E6-C9CF9EA48508@microsoft.com...
> Thank you all for posting your comments. I can tell from your comments
> that
> Microsoft probably does not publish general guidlines for testing patches
> but
> your comments will help me mold a solid plan in our specific environment.
>
> Thanks again...
>
> "Alun Jones" wrote:
>
>> In article <178C0A5D-51D0-46C6-8243-C3193B0C4871@microsoft.com>,
>> "=?Utf-8?B?eTJrYnVnX3M5Nw==?=" <y2kbugs97@discussions.microsoft.com>
>> wrote:
>> >Does anyone know of specific tests that should be performed before
>> >implementing MS security updates on production systems.
>>
>> There are no general recommendations to suggest here, because your goal
>> should
>> be to test those applications on which your business depends. If your
>> business uses the same applications that everyone else uses, then it's
>> entirely likely that they've already been tested by Microsoft. If your
>> business uses different applications from others, then those are the
>> applications you need to test.
>>
>> >I am a member of a large organization and we are trying to enhance our
>> >testing procedures before implementing MS security patches through out
>> >our
>> >production environments.
>>
>> There's nothing specific about these being security patches - they are
>> changes
>> you are making to the system, and you need to ensure that they don't
>> impact
>> the rest of the business' operations in a way that will cause more damage
>> than
>> the problem that they are aimed at fixing.
>>
>> >Is there a document that shows what specific things should be
>> >considered?
>>
>> You need to analyse your LOB (line-of-business) applications, and that's
>> really something that you need to establish for yourself. Establish a
>> set of
>> functions that are required for most of your operations, and a means to
>> test
>> that they happen correctly.
>>
>> Then use that sequence on every change you make, and add to it any
>> operations
>> that appear to be related to the parts of the system that have changed.
>>
>> >I'm looking for a guideline that displays how to validate that a
>> >security
>> >patch released by MS will not break any applications or the Operating
>> >System.
>>
>> I'd say it's a fairly good bet that Microsoft has already done more
>> testing
>> than you'll be able to in regards to making it not break the OS, so you
>> should
>> concentrate on those applications that are required for your business'
>> continued operations.
>>
>> But, there are a few things you should consider:
>>
>> 1. Download the patch.
>>
>> 2. Test the install on a lab machine / machines that represent the
>> majority of
>> your users.
>>
>> 3. Test the uninstall - when you roll out the patch to users, if it kills
>> their systems, you need to know that the uninstall is going to work.
>>
>> 4. Test the installed patch in your lab against some sample use cases.
>> Engage
>> in a feedback loop with your users to discover the important use cases -
>> remind them that if they experienced significant problems with a previous
>> patch, that was because you didn't test what they need to do against the
>> patch
>> in the lab. If that was because they didn't tell you what they need to
>> do,
>> then they need to be more detailed next time.
>>
>> 5. Determine who your rollout will start with - pick a group of users
>> that
>> will be first-adopters for the next patch. There are different theories
>> on
>> this - you can find people who are advanced users, so they can cope with
>> instructions to resolve issues; you can pick people at random, to make a
>> good
>> statistical selection; you can pick people that you know are trouble to
>> any
>> piece of software they encounter; you can pick people that you like, you
>> can
>> pick people that you hate. Okay, that's a piece of humour, I don't
>> suggest
>> that you do exactly that - but pick a strategy, or a set of strategies.
>> I
>> suggest informing the test group that they are to be used as a test case
>> for a
>> new update that's rolling through, and that you need them to let you know
>> anything that's changed.
>>
>> 6. Review the success of the initial rollout in meetings with
>> stakeholders in
>> those departments, and then roll out to the rest of the company if the
>> result
>> is good - or if the anticipated result of not rolling it out is worse.
>>
>> Alun.
>> ~~~~
>>
>> [Please don't email posters, if a Usenet response is appropriate.]
>> --
>> Texas Imperial Software | Find us at http://www.wftpd.com or email
>> 23921 57th Ave SE | alun@wftpd.com.
>> Washington WA 98072-8661 | WFTPD, WFTPD Pro are Windows FTP servers.
>> Fax/Voice +1(425)807-1787 | Try our NEW client software, WFTPD Explorer.
>>

y2kbugs97
Mon Jan 30 09:43:26 CST 2006

Thank you roger. Although these links are very good at showing the patch
management process. There is little information regarding how to test the
individual patches internal to an organization.

There are some very general information like (but this is very general):

Testing
If the results of your assessment determine that a patch must be installed,
you should test that patch against your system to ensure that no breaking
changes are introduced or, if a breaking change is expected, how to work
around the change.

Methods for Testing Security Patches
Methods used to test the installation of security patches against your
systems include:

Testing security patches against a test mirror of your live server
configuration and scenario. This method allows you to both test the
installation offline, without disrupting service, and the freedom to test
workarounds if a breaking change is introduced, again without disrupting
service.
Testing the patch on a few select production systems prior to fully
deploying the update. If a test network that matches your live configuration
is not available, this is the safest method to introduce the security patch.
If this method is employed, you must perform a backup prior to installing the
update.
Confirming the Installation of a Patch
Before deploying a patch to production servers, confirm that the tested
patch has made the appropriate changes on the test servers. Each security
bulletin includes the information you need to confirm that the patch has been
installed. In each bulletin, the Additional information about this patch
section contains the entry Verifying patch installation. It includes registry
values, file versions, or similar configuration changes that you can use to
verify that the patch is installed.

Uninstalling a Security Patch
If you need to uninstall a patch, use Add/Remove Programs in the Control
Panel. If an uninstall routine is not an option for the patch and its
installation introduces breaking changes, you must restore your system from
backup. Make sure that your testing process also covers the patch uninstall
routine.



I guess I was looking for things like:

1) install patch
2) verify that the computer still has network connectivity by doing the
following: run these tools: ping, tracrt, dcdiag, etc
3) Check services- use these tools to verify services are working. 1st look
at all automatic starting services and verify that they are started. etc
4) Check Internet Explorer. Verify that the web is accessable via these
methods... etc
5) Check event viewer in these locations, application, systems logs. Run
this filter to trap common errors associated with security updates
6) etc

After checking the above things, you can be relatively sure that security
patch did not stop functionality of the server...




"Roger Abell [MVP]" wrote:

> On the contrary, MS has issued very much info on the topic
> http://search.microsoft.com/results.aspx?mkt=en-US&setlang=en-US&q=patch+management
> We were just trying to give the essentials.
>
> "y2kbug_s97" <y2kbugs97@discussions.microsoft.com> wrote in message
> news:236F2A1A-DD4B-4152-A2E6-C9CF9EA48508@microsoft.com...
> > Thank you all for posting your comments. I can tell from your comments
> > that
> > Microsoft probably does not publish general guidlines for testing patches
> > but
> > your comments will help me mold a solid plan in our specific environment.
> >
> > Thanks again...
> >
> > "Alun Jones" wrote:
> >
> >> In article <178C0A5D-51D0-46C6-8243-C3193B0C4871@microsoft.com>,
> >> "=?Utf-8?B?eTJrYnVnX3M5Nw==?=" <y2kbugs97@discussions.microsoft.com>
> >> wrote:
> >> >Does anyone know of specific tests that should be performed before
> >> >implementing MS security updates on production systems.
> >>
> >> There are no general recommendations to suggest here, because your goal
> >> should
> >> be to test those applications on which your business depends. If your
> >> business uses the same applications that everyone else uses, then it's
> >> entirely likely that they've already been tested by Microsoft. If your
> >> business uses different applications from others, then those are the
> >> applications you need to test.
> >>
> >> >I am a member of a large organization and we are trying to enhance our
> >> >testing procedures before implementing MS security patches through out
> >> >our
> >> >production environments.
> >>
> >> There's nothing specific about these being security patches - they are
> >> changes
> >> you are making to the system, and you need to ensure that they don't
> >> impact
> >> the rest of the business' operations in a way that will cause more damage
> >> than
> >> the problem that they are aimed at fixing.
> >>
> >> >Is there a document that shows what specific things should be
> >> >considered?
> >>
> >> You need to analyse your LOB (line-of-business) applications, and that's
> >> really something that you need to establish for yourself. Establish a
> >> set of
> >> functions that are required for most of your operations, and a means to
> >> test
> >> that they happen correctly.
> >>
> >> Then use that sequence on every change you make, and add to it any
> >> operations
> >> that appear to be related to the parts of the system that have changed.
> >>
> >> >I'm looking for a guideline that displays how to validate that a
> >> >security
> >> >patch released by MS will not break any applications or the Operating
> >> >System.
> >>
> >> I'd say it's a fairly good bet that Microsoft has already done more
> >> testing
> >> than you'll be able to in regards to making it not break the OS, so you
> >> should
> >> concentrate on those applications that are required for your business'
> >> continued operations.
> >>
> >> But, there are a few things you should consider:
> >>
> >> 1. Download the patch.
> >>
> >> 2. Test the install on a lab machine / machines that represent the
> >> majority of
> >> your users.
> >>
> >> 3. Test the uninstall - when you roll out the patch to users, if it kills
> >> their systems, you need to know that the uninstall is going to work.
> >>
> >> 4. Test the installed patch in your lab against some sample use cases.
> >> Engage
> >> in a feedback loop with your users to discover the important use cases -
> >> remind them that if they experienced significant problems with a previous
> >> patch, that was because you didn't test what they need to do against the
> >> patch
> >> in the lab. If that was because they didn't tell you what they need to
> >> do,
> >> then they need to be more detailed next time.
> >>
> >> 5. Determine who your rollout will start with - pick a group of users
> >> that
> >> will be first-adopters for the next patch. There are different theories
> >> on
> >> this - you can find people who are advanced users, so they can cope with
> >> instructions to resolve issues; you can pick people at random, to make a
> >> good
> >> statistical selection; you can pick people that you know are trouble to
> >> any
> >> piece of software they encounter; you can pick people that you like, you
> >> can
> >> pick people that you hate. Okay, that's a piece of humour, I don't
> >> suggest
> >> that you do exactly that - but pick a strategy, or a set of strategies.
> >> I
> >> suggest informing the test group that they are to be used as a test case
> >> for a
> >> new update that's rolling through, and that you need them to let you know
> >> anything that's changed.
> >>
> >> 6. Review the success of the initial rollout in meetings with
> >> stakeholders in
> >> those departments, and then roll out to the rest of the company if the
> >> result
> >> is good - or if the anticipated result of not rolling it out is worse.
> >>
> >> Alun.
> >> ~~~~
> >>
> >> [Please don't email posters, if a Usenet response is appropriate.]
> >> --
> >> Texas Imperial Software | Find us at http://www.wftpd.com or email
> >> 23921 57th Ave SE | alun@wftpd.com.
> >> Washington WA 98072-8661 | WFTPD, WFTPD Pro are Windows FTP servers.
> >> Fax/Voice +1(425)807-1787 | Try our NEW client software, WFTPD Explorer.
> >>
>
>
>

alun
Mon Jan 30 13:34:36 CST 2006

"y2kbug_s97" wrote:
> I guess I was looking for things like:
>
> 1) install patch
> 2) verify that the computer still has network connectivity by doing the
> following: run these tools: ping, tracrt, dcdiag, etc

Which is fine, if those tools access resources you are looking for, and
which Microsoft hasn't tested for. I would lay odds that you won't find as
blatant a failure as would come out in a "ping" or "tracert" failure in a
patch that has been released.

> 3) Check services- use these tools to verify services are working. 1st look
> at all automatic starting services and verify that they are started. etc

Again, this depends on you - what services do you need?

> 4) Check Internet Explorer. Verify that the web is accessable via these
> methods... etc

Not necessary on server machines that are not going to be used for accessing
web pages. Also, very likely to have been tested by Microsoft already. Why
would Microsoft provide a document telling you how to do tests that they
should have done before releasing the patch? If they haven't thought of
doing those tests in their own testing, they won't think of recommending them
to customers, and if they have thought of recommending them to customers,
they will have instituted them in the patch testing.

You seem intent on applying tests that should have already been done by
Microsoft. That may have been appropriate five years ago, when the classic
advice was "never be the first on your block to install a patch", but these
days, Microsoft's testing of patches is way ahead of the rest of the industry.

> 5) Check event viewer in these locations, application, systems logs. Run
> this filter to trap common errors associated with security updates

Security updates cover a wide range of applications. There are few "common
errors". Those that I can think of, such as RPC failures, are a result of
applying an RPC patch, not a security patch in general. Look at what the
patch modifies, and test applications that use those modified components.

> After checking the above things, you can be relatively sure that security
> patch did not stop functionality of the server...

"functionality" is what _you_ need the server to do, not what Microsoft
defines. Microsoft (and their patch validation beta testers) have already
verified that the patch does not break common functionality - it is now your
task to verify that the patch does not break _uncommon_ functionality - those
applications that you specifically use for your business.

Alun.
~~~~