ctacke/>
Mon Feb 26 13:21:52 CST 2007
This application can either :
1- Load the same DLL (copied with a different name) over and over.
2- Fill an ArrayList with DateTime[1000] while checking free RAM on
the device.
<ctacke>
Checking RAM by what mechanism?
<ctacke>
We noticed that :
- after 60 DLLs loaded with (1), we get a MissingMethodException.
<ctacke>
Sure. Loading native DLLs takes at least 64k each. 60 DLLs would be ~3.8MB
of virtual space. How is this valid? Does your app actually need 60+
native DLLs?
</ctacke>
- after 25 Mb used with (2), we get an OOMException.
<ctacke>
Again, exactly what is this testing? I'm not sure exactly how big a
DateTime is (your number or 1k items indicates 25k which seems really,
really high), but does your app actually contain 25MB of data like this? If
so, this is probably not an ideal design.
</ctacke>
- if we run (1) all the way, we can only load 13 Mb with (2).
- after 12 Mb of (2) we can't load a single DLL with (1).
If we duplicate our test application :
- we can run (2) all the way to 25 Mb each in parallel, for a total of
50 Mb used.
- if we load 20 DLLs with (1), then run (2) all the way (up to ~21
Mb), we can load 40 other DLLs with the second application before
getting a MissingMethodException.
<ctacke>
Again I'm not sure I "get" the point of all of this. First, are you really
loading 60 DLLs? If not, then there's no reason to be testing that. If you
*must* load that many DLLs, are they all out of your control or can they be
combined? Doug points out that lots of small DLLs is not a good thing due
to the 64k minimum virtual memory size each will take.
Are you really using 25 or even 12MB of arrays of small objects? If so,
this seems like a not-so-wise implementation, but we have no idea on how you
could work around it because the sample/test doesn't seem terribly valid.
</ctacke>
Based on these measures, we decided to pre-load all our Native DLLs
prior to loading our ".NET Modules". And it works fine for everything
we could pre-load.
> Your managed app could be taking up all of the virtual space with the
> GCHEap
> and preventing native DLLs from loading. I've never seen a managed app
> take
> up 25, or even 12MB of heap space though. If it is in fact doing that, I
> suspect you're holding some really big items that you shouldn't be, but I
> don't knwo what your "data" is. 25MB of process space usage is huge for
> any
> app, managed or otherwise.
We're not at 25 Mb yet, but probably around 14 Mb at worst. Although
<ctacke>
Again, 14MB by what measure? Is it 14MB of process space, or 4MB or process
and 10 or shared? You must be able to answer that to make any headway on
this problem.
</ctacke>
the application is still going to grow with GPRS synchronisation
working in background, etc...
<ctacke>
Not sure how SYnchronization has an affect on it. Are you using colossal
DataSets that you carry around?
</ctacke>
I know it's a pretty big application, and we tried to reduce what we
could.
We are getting close to having 100 C# DLLs in our application, among
which 40 are interfaces. Most of our data (~200 Mb) is stored in
databases though, and we already cleaned a lot of our data cache.
<ctacke>
Again, managed DLLs have no affect of the process slot virtual memory - they
are loaded as memory mapped files from shared space. 100 DLLs or 1000 won't
matter.
</ctacke>
We load our ".NET Modules" using reflection, and then store them in a
HashTable for future use, and a lot of these modules are interacting,
and have to be loaded in parallel. Maybe that's part of our problem...
<ctacke>
When you load the interface classes, you're not walking each class and its
methods and properties via reflection are you? If so that's going to take
up some serious space.
</ctacke>
> You can't "keep" any memory. The GC handles all allocations and frees for
> you. Again, if you've got 25MB allocated and the GC is unable to release
> any of it because it's all got active roots, there's not much that can be
> done other than change the way your app works.
The GC does release some of the memory, but as long as it doesn't
reduce the allocations under 12 Mb, we are still stuck when trying to
load Native DLLs.
<ctacke>
How much is "some"? It would be well worth building this against 2.0 just
so you can run RPM against it and see what it's doing.
</ctacke>
> I'm still unclear what 25Mb of "usage" means in your app. How are you
> deriving this nuymber? Used from where? Again, I've never seen an app
> using that much virtual space (not one that's behaving well anyway), so
> without knowing what's making the allocations or from where, all I can do
> is
> guess.
I got the number from our test application, we are not there yet in
our real application.
Part of my problem comes from that our customer wants his application
to be able to read and control RFID cards and 2D barcodes, print
tickets (and barcodes) on a bluetooth printer, use very large
databases to determine the price of the tickets, keep track of the
daily activity of the user, access WebServices and WebSites via GPRS,
synchronise and update the PDA via WiFi or Ethernet, etc. while having
desktop-like performance.
<ctacke>
For the most part this shouldn't be a problem (using the items, not
necessarily achieving desktop speed of course)
</ctacke>
A lot of this requires the use of a few Native DLLs, and we have 70
different screens... so far, we had architectured the application
towards modularity and performance rather than memory usage, although
we did try to reduce that lately. The user constantly switches between
the different modules, and loading them seems pretty long (even if it
only takes a few seconds), and could be a reason for them not to use
the application anymore if we unloaded them everytime (we do unload
the screens though, but usually not the business parts and Native
DLLs). The technical or transverse modules also have to be loaded at
all times since they are used by most functions.
<ctacke>
I'm beginning to think it's misuse of System.Data elements. You could
probably greatly reduce your footprint using data readers.
</ctacke>
I'm afraid we will have to rethink part of the application to reduce
the memory usage, but as I said, it's frustrating when it's so close
to working perfectly that way (by pre-loading Native stuff before
loading our different modules).
Our test application seemed to indicate that we still had a good
margin (up to 25 Mo) before having OOM issues.
The MissingMethodException issue after 12 Mb is disturbing though.
<ctacke>
Again, saying "XXMB" is not of value unless we know what type of memory it
is. Process or shared.
</ctacke>
> did you run RPM to see?
I haven't heard of RPM yet. I had looked into Entrek a bit, but
haven't used it yet.
Is there any chance of finding what is loaded during the first HTTPS
connection with such a tool? I still believe that would be our easy
way out, since reachitecturing our application will not be an easy
thing to do right now.
<ctacke>
Entrek is great for native apps, but of no value for managed apps. I'd look
at the CF 1.0 perf counters first, and then watch the app with RPM and see
what's up. Do a web search for RPM or Remote Perfomrnace Monitor with the
CF. It's likely to show you pretty quickly how things are going bad
(unfortunately its granularity doesn't yet show the "where" so that'll take
some detective work).
</ctacke>
Thanks a lot for your quick answers,
Regards,
--
Patrick AVENEL
On 26 fév, 17:46, "<ctacke/>" <ctacke[@]opennetcf[dot]com> wrote:
> 1. You said the GC Heap size is unbound, yet we manage to load 25 Mb
> of data in one Process before getting an OOM Exception, and still load
> another 25 Mb of data in another process without unloading the data
> from the first process. (The system free memory is down 50 Mb). That
> doesn't fit with the way I understand the way shared memory works.
>
> <ctacke>
> The GCHeap is allocated out of the 32MB process slow, so it is bounded by
> available memory, but oinbounded by the CF itself (unlike in 1.0). You
> "loaded" 25Mb how? Where? how do you know it's all in the GCHeap (did
> you
> run RPM to see?).
> </ctacke>
>
> 2. You talk about "Native stuff" at the beginning of your
> presentation, but I didn't understand where these Native DLLs get
> loaded.
>
> <ctacke>
> Native DLLs follow the normal model and load into slot 0, like the article
> you referenced shows
> <c/tacke>
>
> Our problem comes from the fact that our CF program seems to
> be taking memory space which could otherwise be used to load Native
> DLLs. Although we can use up to 25 Mb for our application, once the
> application uses over 12 Mb, it's no longer possible to load Native
> DLLs in the same process (yet we can load Native DLLs in another
> process, as long as the total doesn't exceed 12 Mb of Native DLLs
> loaded).
>
> <ctacke>
> Your managed app could be taking up all of the virtual space with the
> GCHEap
> and preventing native DLLs from loading. I've never seen a managed app
> take
> up 25, or even 12MB of heap space though. If it is in fact doing that, I
> suspect you're holding some really big items that you shouldn't be, but I
> don't knwo what your "data" is. 25MB of process space usage is huge for
> any
> app, managed or otherwise.
> </ctacke>
>
> I though it would be possible to keep some critical memory space in
> the 12Mb range of our process, and free it when we would need to load
> new DLLs, but the part about GC Compaction ruined my hopes there : it
> seems the memory freed by the CF will always be on top.
>
> <ctacke>
> You can't "keep" any memory. The GC handles all allocations and frees for
> you. Again, if you've got 25MB allocated and the GC is unable to release
> any of it because it's all got active roots, there's not much that can be
> done other than change the way your app works.
> </ctacke>
>
> It's pretty frustrating that 99% of our application works fine up to
> 20-25 Mb of memory usage (by loading all the "Native stuff" before
> loading the heavy parts of our application), and we might have to
> rearchitecture it completely to remain under 12 Mb of memory usage at
> all times (probably at a great performance cost) just because we can't
> find a way to load whatever is required to establish an HTTPS
> connection before the connection is actually established.
>
> <ctacke>
> I'm still unclear what 25Mb of "usage" means in your app. How are you
> deriving this nuymber? Used from where? Again, I've never seen an app
> using that much virtual space (not one that's behaving well anyway), so
> without knowing what's making the allocations or from where, all I can do
> is
> guess.
> </ctacke>
>
> Thanks again for your help!
>
> Regards,
>
> --
> Patrick AVENEL
>
> On 26 fév, 15:03, "<ctacke/>" <ctacke[@]opennetcf[dot]com> wrote:
>
>
>
> > The Compact Framework doesn't follow the model you're looking at. Its
> > DLLs
> > load into sharedmemoryas amemory-mapped file and have no impact on the
> > 32MB process space. The only thing that uses substantial process space
> > is
> > the GCHeap, and if you're hitting 12MB of heap usage, you might consider
> > rearchitecting how the app works.
>
> > Here's a webcast that might clarify things.
>
> >
http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=10...
>
> > --
> > Chris Tacke - Embedded MVP
> > OpenNETCF Consulting
> > Managed Code in the Embedded Worldwww.opennetcf.com
> > --
>
> > "Patrick A." <patrick.ave...@gmail.com> wrote in message
>
> >news:1172479898.418334.54450@k78g2000cwa.googlegroups.com...
>
> > > Hello,
>
> > > We are currently havingmemoryissues with our Pocket PC application.
> > > We are using PsionTeklogix WorkAbout Pro devices withWindowsMobile
> > > 2003 and the Compact Framework 1.0 SP3.
> > > The device has 95.80 Mb of RAM, 30 Mb of ROM and a 512 Mb SD-Card.
> > > Most of our application is loaded directly from the SD-Card.
>
> > > The limitations we have experienced are the following:
> > > - 25 Mb per process
> > > - 12 Mb for the DLLs (independently from the number of
> > > processes)
> > > - DLLs can only be loaded in the "first" 12 Mb of each
> > > process
>
> > > For example:
> > > 1. If a process loads 4 Mb of DLLs, then 8 Mb of data, he can no
> > > longer load DLLs, but can still load data. A second process launched
> > > in parallel will be able to load a maximum 8 Mb of new DLLs, then
> > > data.
> > > 2. If a process loads 12 Mb of data, it can no longer load a single
> > > DLL.
>
> > > What happens in our application is that at some point we reach the 12
> > > Mblimitand every attempt at loading a library after that fails.
>
> > > Such behaviour is described here:
> > >
http://msdn2.microsoft.com/en-us/library/ms836325.aspx
> > > "Now consider what to do if you encounter a potential problem.
> > > Suppose
> > > Process 2 loaded DLL C that was quite large as shown in Figure 6.
> > > Note
> > > that process 3 has the bad luck of being both a large .exe file and
> > > loading a DLL after process 2 had loaded its rather immense DLL C.
> > > Clearly, process 3 is close to trouble if it attempted to load any
> > > more DLLs that had not been loaded already by other processes. This
> > > is
> > > a somewhat contrived example because the size of DLL C would have to
> > > be incredibly large, or Process 2 would have to load a large number
> > > of
> > > DLLs for this problem to occur naturally."
>
> > > We do have a large number of DLLs, some of which are quite large and
> > > the previous example is no longer contrived in our case.
>
> > > To avoid the problem, we tried several approaches:
> > > - Free somememoryin the critical 12 Mb area when we need to load
> > > a new DLL
> > > o The compact framework doesn't seem to free thememoryfor
> > > native DLLs, but only for its own future use.
> > > o We tried to artificially create 12 Mb of data before
> > > loading our real data, and then free the 12 Mb of artificial data to
> > > restore some freememoryin the criticalmemoryspace, but no new DLL
> > > could be loaded afterwards.
> > > o We even tried to broadcast a WM_HIBERNATE message to
> > > force the Compact Framework to free somememory, but in vain.
>
> > > - Migrate our application toWindowsMobile2005 and the Compact
> > > Framework 2.0
> > > o We had the samememoryissues.
>
> > > - Split our application in several processes.
> > > o We still reach the 12 Mblimitfor the DLLs, no matter
> > > how many processes are involved.
> > > o Loading our DLLs in a separate process doesn't change
> > > the
> > > behaviour.
>
> > > - Load all our DLLs before loading data.
> > > o This approach has been quite successful so far.
> > > o We still have some issues with several functionalities
> > > such as HTTPS WebService calls, which fail if the very first call is
> > > made when thememoryhas already passed the 12 Mblimit:
> > > "System.Net.WebException - Could not establish secure channel for
> > > SSL/
> > > TLS".
>
> > > Is there something to pre-load so that our HTTPS call will already
> > > have its requirements loaded before thememoryhas passed the 12 Mb
> > >limit?
>
> > > Did we miss something in our understanding of howmemoryis managed
> > > in
> > >WindowsMobile2003?
>
> > > Our application requires more than 12 Mb ofmemory(most of which is
> > > data and C# .Net libraries), and needs to be able to load new DLLs at
> > > all time, what would
> > > you recommend in this situation?
>
> > > Thank you in advance,
>
> > > Sincerely,
>
> > > --
> > > Patrick AVENEL- Masquer le texte des messages précédents -
>
> > - Afficher le texte des messages précédents -- Masquer le texte des
> > messages précédents -
>
> - Afficher le texte des messages précédents -