Using compression from kernel mode...

hi
I am developing a kernel mode driver in which I need to
compress/decompress huge amount of data.Now all the available compression
libraries seem to contain the usage C run time libraries,making it very
unsuitable for use in windows kernel mode.
Is there any compression decompression library for windows kernel mode
environment?

Rgds

Gary
Tue Apr 11 09:59:54 CDT 2006

Is there a compelling reason to do this in the kernel?

--
The personal opinion of
Gary G. Little

"Debabrata" <debabrata@stellarinfo.com> wrote in message
news:Op7Lb%23WXGHA.4148@TK2MSFTNGP03.phx.gbl...
> hi
> I am developing a kernel mode driver in which I need to
> compress/decompress huge amount of data.Now all the available compression
> libraries seem to contain the usage C run time libraries,making it very
> unsuitable for use in windows kernel mode.
> Is there any compression decompression library for windows kernel mode
> environment?
>
> Rgds
>
>

Debabrata
Tue Apr 11 23:14:16 CDT 2006

hi
well yeah....actually I am developing a virtual disk driver for FAT and
NTFS which will be capable of loading used and compressed image of a
FAT/NTFS....The compression has to be done by an application program while
my driver will decompress the image and load it.
I have been able load the used clusters image of FAT32 but now I am struct
with the compression problem..
Can you plz help me out..

Rgds

"Gary G. Little" <gary.g.little@seagate.com> wrote in message
news:443bc2eb$0$15431$6d36acad@taz.nntpserver.com...
> Is there a compelling reason to do this in the kernel?
>
> --
> The personal opinion of
> Gary G. Little
>
> "Debabrata" <debabrata@stellarinfo.com> wrote in message
> news:Op7Lb%23WXGHA.4148@TK2MSFTNGP03.phx.gbl...
> > hi
> > I am developing a kernel mode driver in which I need to
> > compress/decompress huge amount of data.Now all the available
compression
> > libraries seem to contain the usage C run time libraries,making it very
> > unsuitable for use in windows kernel mode.
> > Is there any compression decompression library for windows kernel
mode
> > environment?
> >
> > Rgds
> >
> >
>
>

Maxim
Wed Apr 12 18:12:48 CDT 2006

Borrow some legally free code (or just ideas - they are always legally free
unless patented) from somewhere and write the compression yourself.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com

"Debabrata" <debabrata@stellarinfo.com> wrote in message
news:Op7Lb%23WXGHA.4148@TK2MSFTNGP03.phx.gbl...
> hi
> I am developing a kernel mode driver in which I need to
> compress/decompress huge amount of data.Now all the available compression
> libraries seem to contain the usage C run time libraries,making it very
> unsuitable for use in windows kernel mode.
> Is there any compression decompression library for windows kernel mode
> environment?
>
> Rgds
>
>

Kevin
Thu Apr 13 15:44:21 CDT 2006

Remove all CRT header files from the compression algorithm source code.

Compile for kernel mode. Re-implement all required CRT functions until all
errors are eliminated

Replace all memory allocations with fixed buffers, as much as possible.

Test performance with compression in the kernel.

Then, re-architect the driver to interface with a user-mode application
using the blocking-ioctl method, put the compression in the user-mode app,
and then test performance.

You may find that immense amount of work in the kernel is great when the
system is doing nothing but executing your codepath, but terrible when other
applications are running. You may find that performance is terrible with
certain access-patterns exhibited by particular applications. Or, you may
find that it works better in the kernel.

But, the main reason why implementing the blocking-ioctl method is best is
that silly bugs cause simple user-mode exceptions -- process termination --
instead of re-boots. Very helpful during the development and testing phase.

After doing it this way, when it comes time to move the code into the
kernel, you might think twice... what if... at the customer site... BSOD...
right now... I catch any exceptions in user-mode, relaunch the
driver-interface thread, and recover gracefully... but what if my testing
was not 100% complete... what about getting a surprise removal when coming
out of hibernation on a multi-core cpu... while the user is pressing the F10
key... ya know... in the interests of "public safety", the only code in the
driver is code that MUST execute in kernel-mode.

Aahh... now we can all sleep at night without worrrying about scrambled hard
disks due to a silly oversight / typo / build-bug....

The only code in the kernel is code that can not execute in user-mode.

"Debabrata" <debabrata@stellarinfo.com> wrote in message
news:uxPIwYeXGHA.1352@TK2MSFTNGP05.phx.gbl...
> hi
> well yeah....actually I am developing a virtual disk driver for FAT and
> NTFS which will be capable of loading used and compressed image of a
> FAT/NTFS....The compression has to be done by an application program while
> my driver will decompress the image and load it.
> I have been able load the used clusters image of FAT32 but now I am struct
> with the compression problem..
> Can you plz help me out..
>
> Rgds
>
> "Gary G. Little" <gary.g.little@seagate.com> wrote in message
> news:443bc2eb$0$15431$6d36acad@taz.nntpserver.com...
> > Is there a compelling reason to do this in the kernel?
> >
> > --
> > The personal opinion of
> > Gary G. Little
> >
> > "Debabrata" <debabrata@stellarinfo.com> wrote in message
> > news:Op7Lb%23WXGHA.4148@TK2MSFTNGP03.phx.gbl...
> > > hi
> > > I am developing a kernel mode driver in which I need to
> > > compress/decompress huge amount of data.Now all the available
> compression
> > > libraries seem to contain the usage C run time libraries,making it
very
> > > unsuitable for use in windows kernel mode.
> > > Is there any compression decompression library for windows kernel
> mode
> > > environment?
> > >
> > > Rgds
> > >
> > >
> >
> >
>
>

Maxim
Thu Apr 13 17:58:10 CDT 2006

> Replace all memory allocations with fixed buffers, as much as possible.

Impossible. On-stack buffers in the kernel are deadly, and the global buffers
are not thread-safe.

> After doing it this way, when it comes time to move the code into the
> kernel, you might think twice... what if... at the customer site... BSOD...
> right now... I catch any exceptions in user-mode, relaunch the

Another way:

- rework the compression code (it does not rely on a platform much) to be
portable across kernel/user using macros - OSAlloc is malloc() in user and
ExAllocatePoolWithTag in kernel, and so on.
- write a tiny test tool to compress/decompress some file - main() and a call
to this portable code.
- test it a lot using BAT files.
- then integrate it to the kernel project.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com