Importing from a text file to a Dataset

I have a large(1.25M), fixed field delimited, text file which I want to read into a DataSet. So, I opened a StreamReader, read a line, assigned the values to a new DataRow, and finally added the DataRow to a DataTable in the DataSet. Seemed to be a staight forward task but boy did it progress at a snail's pace!

So I tried to speed things up by first importing the data from the text file into SQL Server. To my amazement, the text import went like lightning! How can I get lightning fast text import?

Michael

Miha
Thu Jan 22 07:21:02 CST 2004

Hi,

See this thread
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=ivkblu05hqdqosre0d18butjdfrmnpc400%404ax.com&rnum=3&prev=/groups%3Fas_q%3Dtext%2520file%26safe%3Dimages%26ie%3DUTF-8%26oe%3DUTF-8%26as_ugroup%3D*adonet*%26as_uauthors%3Dpaul%2520clement%26lr%3D%26hl%3Den
Basically you should use OleDb provider to read from text file.
It should be faster.

--
Miha Markic - RightHand .NET consulting & software development
miha at rthand com
www.rthand.com

"michael" <michael@hockstein.org> wrote in message
news:1FB3BBD3-4BD1-4458-B5C8-3C04D438D9F6@microsoft.com...
> I have a large(1.25M), fixed field delimited, text file which I want to
read into a DataSet. So, I opened a StreamReader, read a line, assigned the
values to a new DataRow, and finally added the DataRow to a DataTable in the
DataSet. Seemed to be a staight forward task but boy did it progress at a
snail's pace!
>
> So I tried to speed things up by first importing the data from the text
file into SQL Server. To my amazement, the text import went like lightning!
How can I get lightning fast text import?
>
> Michael

anonymous
Thu Jan 22 07:51:07 CST 2004

OK! Let's see if I can get it to work.

Michael

Paul
Thu Jan 22 08:58:39 CST 2004

On Thu, 22 Jan 2004 05:51:07 -0800, "Michael" <anonymous@discussions.microsoft.com> wrote:

¤ OK! Let's see if I can get it to work.
¤

If you have fixed length fields (as opposed to comma or custom character delimited) you will need a
schema.ini file:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcjetsdk_98.asp


Paul ~~~ pclement@ameritech.net
Microsoft MVP (Visual Basic)

Miha
Thu Jan 22 10:59:20 CST 2004

Hi Keith,

"Keith Matzen" <kmatzen@hotmail.com> wrote in message
news:4A1B9F36-4808-4602-98F5-4D8589B1436F@microsoft.com...
> I have had similar performance problems loading a database from text
files. But the text files I am usually importing contain headings and
fields that need some kind of conversion. Would it make sense to build a
clean temporary text file and then import it by this method?

It would be faster, yes. But I don't know how faster. You'll have to measure
it by yourself.

Can this method be used on a Stream instead of a file?

I don't think so because it uses OleDb...

--
Miha Markic - RightHand .NET consulting & development
miha at rthand com
www.rthand.com