fprintf performance

Hallo,

I am using fprintf to write a very large byte-array to disk.
In order to do this, I am writing each byte with a separate
fprintf-call in a text file (text file is a hard requirement
for my application).

Unfortunately, the performance of this disk-write is very poor:
over all, I do get only ~17Mbit/s, which is not acceptable
for my purpose.

How can I speed up my application?

Doug
Tue May 29 00:34:41 CDT 2007

On 28 May 2007 22:21:40 -0700, Ezmeralda <ezmeralda@gmx.de> wrote:

>
>Hallo,
>
>I am using fprintf to write a very large byte-array to disk.
>In order to do this, I am writing each byte with a separate
>fprintf-call in a text file (text file is a hard requirement
>for my application).
>
>Unfortunately, the performance of this disk-write is very poor:
>over all, I do get only ~17Mbit/s, which is not acceptable
>for my purpose.
>
>How can I speed up my application?

Buffer your data and write the largest blocks possible. The stdio routines
buffer by default, but writing a byte at a time is still a lot slower than
writing large blocks, especially when using the multithreaded runtime,
which acquires and releases a CRITICAL_SECTION on every call.[*] And for
writing single bytes, fprintf is a lot slower than, say, fputc. If you need
to format the data, you're really not "writing a very large byte-array to
disk", but if in fact you really are writing an array of bytes, you should
be able to save all your data with a single call to fwrite. That will give
you the best performance you will find in stdio.

[*] VC2005 provides "unlocked" versions which do not perform the locking
done by the standard functions.

--
Doug Harrison
Visual C++ MVP

Alex
Tue May 29 05:04:32 CDT 2007

Ezmeralda wrote:
> I am using fprintf to write a very large byte-array to disk.
> In order to do this, I am writing each byte with a separate
> fprintf-call in a text file (text file is a hard requirement
> for my application).
>
> Unfortunately, the performance of this disk-write is very poor:
> over all, I do get only ~17Mbit/s, which is not acceptable
> for my purpose.
>
> How can I speed up my application?

If you want to achieve best performance, then do as few
writes as possible. Ideally, do one big write with fwrite,
as Doug suggests.

If you don't mind to use Win32 specific features, then look
at memory mapped files. It should give significant
performance boost.


Alex

Tom
Tue May 29 05:36:31 CDT 2007

Ezmeralda wrote:
> Hallo,
>
> I am using fprintf to write a very large byte-array to disk.
> In order to do this, I am writing each byte with a separate
> fprintf-call in a text file (text file is a hard requirement
> for my application).
>
> Unfortunately, the performance of this disk-write is very poor:
> over all, I do get only ~17Mbit/s, which is not acceptable
> for my purpose.
>
> How can I speed up my application?


Use fwrite or WriteFile rather than fprintf, making sure each call is
passed a large buffer (say 16K bytes). You should format the text in the
buffer manually (avoid sprintf too!).

e.g.
int const buffer_size = 16384;
char buffer[buffer_size];
int bufferPos = 0;
for (int i = 0; i != byte_array_size; ++i)
{
char const currentByte = byte_array[i];
//format currentByte into buffer, incrementing bufferPos
//and checking whether buffer is full
//(code depends on how you were using fprintf)


if (bufferPos == buffer_size)
{
//empty buffer
fwrite(buffer, bufferPos, file);
bufferPos = 0;
}
}

if (bufferPos > 0)
{
//empty the last bit
fwrite(buffer, bufferPos, file);
}

Error checking should be added.

Tom