Kawarjit
Sun Dec 05 20:04:51 CST 2004
In .Net 2.0 the DataSet and it's related classes have been significantly
extended to scale and perform for large number of rows. However 10GB - 100GB
is kind of large for hosting in a single in-memory data structure. Are you
considering using 64 bit machines, as simple 32 bit machines may only go
upto 4GB?.
It'd help if you give more details, for instance
1. Are you considering partitioning data across multiple systems or does it
have to be constrained to a single system ?
2. Does the complete 10GB - 100GB data needs to be cached in main memory or
can it be paged in from secondary storage
3. What is the performance requirement, is it more around insert, update and
delete or querying? is it a mix of all of these
4. What is the kind of querying support in terms of complexity of
expressions and performance you are looking for.
you may want to take a look at the following article that describes some of
the dataset related enhancements in .NET 2.0
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnadonet/html/datasetenhance.asp
Thanks,
Kawarjit Bedi
Program Manager - ADO.NET Team
Microsoft Corp.
This posting is provided "AS IS" with no warranties, and confers no rights.
"Extreme Datasets" <Extreme Datasets@discussions.microsoft.com> wrote in
message news:95216E1E-8B90-4246-B369-95D04DC8BEED@microsoft.com...
> Hello All,
>
> Does anyone have any suggestions on working with large datasets (10GB -
> 100GB)?
>
> Best initialization?
> Transfer from dataset to database?
>
> Anything would be helpful, articles, code examples, discussions, etc
>
> Thanks In Advance