Handling very huge data

Hi All

Wish all of u Happy New Year

I am developing a application where in I retrieve data from Database which
can go up millions records, do some processing on that and update the same in
single transaction,
now with limited RAM say 500MB how should I handle this data (I know one
solution might be paging) so that my ASP.NET process does not fall short of
memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
recycled because memory consumption exceeded the 305 MB (60 percent of
available RAM). "

Regards
Makarand

Miha
Wed Jan 05 12:21:30 CST 2005

Did you consider using a stored procedure for processing the data?

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
SLODUG - Slovene Developer Users Group
www.rthand.com

"Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in message
news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
> in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
> of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

Sahil
Wed Jan 05 12:29:11 CST 2005

This much data is not a good candidate for putting in a dataset or the likes
of dataset. Keep it in the database for processing if you can.
What is the compelling reason that you must pull it out of the db to work on
so much data?

- Sahil Malik
http://dotnetjunkies.com/weblog/sahilmalik



"Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in message
news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

Patrice
Wed Jan 05 12:30:34 CST 2005

I would try to do this in the DB using SQL...

Patrice

--

"Makarand Keer" <MakarandKeer@discussions.microsoft.com> a écrit dans le
message de news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

Eric
Wed Jan 05 12:32:09 CST 2005

Makarand,

It sounds to me like you have to real options here.

1) Chunk thru the data. I would choose this if there is some specific
business logic that you cannot perform directly on the database layer.

2) Process the Data in the Database. I would choose this if you can
perform the logic on the database layer. If you can go this route it
will probably be the most efficient.

HTH,
-eric


Makarand Keer wrote:
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

MakarandKeer
Wed Jan 05 12:45:06 CST 2005

Sahil
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand

"Sahil Malik" wrote:

> This much data is not a good candidate for putting in a dataset or the likes
> of dataset. Keep it in the database for processing if you can.
> What is the compelling reason that you must pull it out of the db to work on
> so much data?
>
> - Sahil Malik
> http://dotnetjunkies.com/weblog/sahilmalik
>
>
>
> "Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in message
> news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> > Hi All
> >
> > Wish all of u Happy New Year
> >
> > I am developing a application where in I retrieve data from Database which
> > can go up millions records, do some processing on that and update the same
> in
> > single transaction,
> > now with limited RAM say 500MB how should I handle this data (I know one
> > solution might be paging) so that my ASP.NET process does not fall short
> of
> > memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> > recycled because memory consumption exceeded the 305 MB (60 percent of
> > available RAM). "
> >
> > Regards
> > Makarand
>
>
>

MakarandKeer
Wed Jan 05 12:51:17 CST 2005

Miha
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand


"Miha Markic [MVP C#]" wrote:

> Did you consider using a stored procedure for processing the data?
>
> --
> Miha Markic [MVP C#] - RightHand .NET consulting & development
> SLODUG - Slovene Developer Users Group
> www.rthand.com
>
> "Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in message
> news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> > Hi All
> >
> > Wish all of u Happy New Year
> >
> > I am developing a application where in I retrieve data from Database which
> > can go up millions records, do some processing on that and update the same
> > in
> > single transaction,
> > now with limited RAM say 500MB how should I handle this data (I know one
> > solution might be paging) so that my ASP.NET process does not fall short
> > of
> > memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> > recycled because memory consumption exceeded the 305 MB (60 percent of
> > available RAM). "
> >
> > Regards
> > Makarand
>
>
>

Sahil
Wed Jan 05 16:20:36 CST 2005

I'd recommend getting peices of the data rather than millions of rows
together. Maybe in another 4-5 years when we have terabytes of RAM and 64
bit CPUs my suggestion will change, but today this is what I'd recommend.

- Sahil Malik
http://dotnetjunkies.com/weblog/sahilmalik




"Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in message
news:BF23980B-0C35-4C6D-8496-426C5DDF051F@microsoft.com...
> Sahil
> Thanx for prompt reply,
> Thing is I get the data from database server, add feed this data to MS
> RuleEngine for applying Business Rules and after that I take same data and
> update it in database.
> So I don't c any scope of doing all these things in database.
> Hope this time I am more clear on my problem
>
> Makarand
>
> "Sahil Malik" wrote:
>
> > This much data is not a good candidate for putting in a dataset or the
likes
> > of dataset. Keep it in the database for processing if you can.
> > What is the compelling reason that you must pull it out of the db to
work on
> > so much data?
> >
> > - Sahil Malik
> > http://dotnetjunkies.com/weblog/sahilmalik
> >
> >
> >
> > "Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in
message
> > news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> > > Hi All
> > >
> > > Wish all of u Happy New Year
> > >
> > > I am developing a application where in I retrieve data from Database
which
> > > can go up millions records, do some processing on that and update the
same
> > in
> > > single transaction,
> > > now with limited RAM say 500MB how should I handle this data (I know
one
> > > solution might be paging) so that my ASP.NET process does not fall
short
> > of
> > > memory, right now it is giving me error as "aspnet_wp.exe (PID: 596)
was
> > > recycled because memory consumption exceeded the 305 MB (60 percent of
> > > available RAM). "
> > >
> > > Regards
> > > Makarand
> >
> >
> >

Joyjit
Wed Jan 05 22:57:22 CST 2005

Hi,

try using API server side cursors that would fetch you data in batches, so
that your app. will not block for that period, which greatly improves user
experience.

regards
Joyjit

"Makarand Keer" <MakarandKeer@discussions.microsoft.com> wrote in message
news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand