Using CSharpCodeProvider to create MANY assemblies

I am planning to use the CSharpCodeProvider to generate some compiled
functions in my app.

In my current implementation, All of these functions are generated in
one swipe... thus they all invoke the compiler once, and create only
one DLL. My new requirement tells me that I need to be more
dynamic... that I can't queue up all of these functions... I actually
need to do them more on demand.

The number of times this happens can be as many as 10,000 times...
each with a different calculation. This would mean that I invoke the
compiler 10,000 times and create 10,000 in-memory DLLs. This idea
scares me... can .NET handle that many DLLs being linked to the app?

Does anyone know about the performance overhead of such an endeavor?
I am about to write some tests to gather data, but I was wondering if
anyone out there has experience with this type of execution. Am I
going to run into performance penalties for invoking the compiler that
many times? Am I going to run into performance penalties for having
that many DLLs in the app domain? If so, are there any workarounds
that anyone knows of?

Thanks,
Brian

Nicholas
Mon Nov 26 11:03:12 PST 2007

Brian,

I would say you are probably going to start seeing some issues when you
try and load 10,000 assemblies into an app-domain. The assemblies
themselves are small, but the CLR is still going to have overhead in the
form of assembly, module, and type info for each assembly.

As for your requirement, does it state that you have to have separate
assemblies, or that you have to be able to call them dynamically (and not
queue it all up).

If the case is the former, then I would say revisit the requirements.
What's the justification for generating all those assemblies?

If the case is the former, I would look into creating an in-memory
assembly, with a single module, and add new global methods/types as needed
and implementing them using IL, instead of the codedom provider and
compiling separately.


--
- Nicholas Paldino [.NET/C# MVP]
- mvp@spam.guard.caspershouse.com

"Bilz" <BrianGenisio@gmail.com> wrote in message
news:9c89a7d6-08e8-4a8e-a839-d6e53a3a0bb8@y5g2000hsf.googlegroups.com...
>I am planning to use the CSharpCodeProvider to generate some compiled
> functions in my app.
>
> In my current implementation, All of these functions are generated in
> one swipe... thus they all invoke the compiler once, and create only
> one DLL. My new requirement tells me that I need to be more
> dynamic... that I can't queue up all of these functions... I actually
> need to do them more on demand.
>
> The number of times this happens can be as many as 10,000 times...
> each with a different calculation. This would mean that I invoke the
> compiler 10,000 times and create 10,000 in-memory DLLs. This idea
> scares me... can .NET handle that many DLLs being linked to the app?
>
> Does anyone know about the performance overhead of such an endeavor?
> I am about to write some tests to gather data, but I was wondering if
> anyone out there has experience with this type of execution. Am I
> going to run into performance penalties for invoking the compiler that
> many times? Am I going to run into performance penalties for having
> that many DLLs in the app domain? If so, are there any workarounds
> that anyone knows of?
>
> Thanks,
> Brian

Nicholas
Mon Nov 26 11:04:53 PST 2007

Sorry, the second "former" should be "latter".


--
- Nicholas Paldino [.NET/C# MVP]
- mvp@spam.guard.caspershouse.com

"Nicholas Paldino [.NET/C# MVP]" <mvp@spam.guard.caspershouse.com> wrote in
message news:%23Mwkj5FMIHA.3976@TK2MSFTNGP03.phx.gbl...
> Brian,
>
> I would say you are probably going to start seeing some issues when you
> try and load 10,000 assemblies into an app-domain. The assemblies
> themselves are small, but the CLR is still going to have overhead in the
> form of assembly, module, and type info for each assembly.
>
> As for your requirement, does it state that you have to have separate
> assemblies, or that you have to be able to call them dynamically (and not
> queue it all up).
>
> If the case is the former, then I would say revisit the requirements.
> What's the justification for generating all those assemblies?
>
> If the case is the former, I would look into creating an in-memory
> assembly, with a single module, and add new global methods/types as needed
> and implementing them using IL, instead of the codedom provider and
> compiling separately.
>
>
> --
> - Nicholas Paldino [.NET/C# MVP]
> - mvp@spam.guard.caspershouse.com
>
> "Bilz" <BrianGenisio@gmail.com> wrote in message
> news:9c89a7d6-08e8-4a8e-a839-d6e53a3a0bb8@y5g2000hsf.googlegroups.com...
>>I am planning to use the CSharpCodeProvider to generate some compiled
>> functions in my app.
>>
>> In my current implementation, All of these functions are generated in
>> one swipe... thus they all invoke the compiler once, and create only
>> one DLL. My new requirement tells me that I need to be more
>> dynamic... that I can't queue up all of these functions... I actually
>> need to do them more on demand.
>>
>> The number of times this happens can be as many as 10,000 times...
>> each with a different calculation. This would mean that I invoke the
>> compiler 10,000 times and create 10,000 in-memory DLLs. This idea
>> scares me... can .NET handle that many DLLs being linked to the app?
>>
>> Does anyone know about the performance overhead of such an endeavor?
>> I am about to write some tests to gather data, but I was wondering if
>> anyone out there has experience with this type of execution. Am I
>> going to run into performance penalties for invoking the compiler that
>> many times? Am I going to run into performance penalties for having
>> that many DLLs in the app domain? If so, are there any workarounds
>> that anyone knows of?
>>
>> Thanks,
>> Brian
>
>

Bilz
Mon Nov 26 12:45:49 PST 2007

On Nov 26, 2:03 pm, "Nicholas Paldino [.NET/C# MVP]"
<m...@spam.guard.caspershouse.com> wrote:
> Brian,
>
> I would say you are probably going to start seeing some issues when you
> try and load 10,000 assemblies into an app-domain. The assemblies
> themselves are small, but the CLR is still going to have overhead in the
> form of assembly, module, and type info for each assembly.
>

No, there is no requirement to create multiple DLLs... just that the
functions be dynamically generated. The functions look to the user
like math equations... wrap their math equations with C# methods,
compile them, and run them against their inputs.

I will look into the idea of inserting methods via IL. I have never
done anything like that. If I can't use C# code, it might defeat the
purpose, since I am using this method to avoid parsing their
functions. They write something like ((input1 * input2)/input3) and I
write that out directly as a C# method with 3 parameters.

Thanks,
B

Nicholas
Mon Nov 26 13:05:57 PST 2007

Brian,

Well, you would have to parse it anyways to get the parameter names.
Beyond that, you would have to map the operators and whatnot to IL (which
might be no small task).

You might be able to get away with compiling to memory and then taking
the IL from the compiled unit and then placing it dynamically in your type.

I would test creating separate assemblies, and see if you see any
noticable change in your app for the load you would expect to normally
handle. If it works, then I would leave it at that. Only if it doesn't
would I venture down this path.


--
- Nicholas Paldino [.NET/C# MVP]
- mvp@spam.guard.caspershouse.com

"Bilz" <BrianGenisio@gmail.com> wrote in message
news:f977ead5-3344-4eca-bac5-1182068dd23c@e25g2000prg.googlegroups.com...
> On Nov 26, 2:03 pm, "Nicholas Paldino [.NET/C# MVP]"
> <m...@spam.guard.caspershouse.com> wrote:
>> Brian,
>>
>> I would say you are probably going to start seeing some issues when
>> you
>> try and load 10,000 assemblies into an app-domain. The assemblies
>> themselves are small, but the CLR is still going to have overhead in the
>> form of assembly, module, and type info for each assembly.
>>
>
> No, there is no requirement to create multiple DLLs... just that the
> functions be dynamically generated. The functions look to the user
> like math equations... wrap their math equations with C# methods,
> compile them, and run them against their inputs.
>
> I will look into the idea of inserting methods via IL. I have never
> done anything like that. If I can't use C# code, it might defeat the
> purpose, since I am using this method to avoid parsing their
> functions. They write something like ((input1 * input2)/input3) and I
> write that out directly as a C# method with 3 parameters.
>
> Thanks,
> B

Bilz
Mon Nov 26 13:57:29 PST 2007

On Nov 26, 4:05 pm, "Nicholas Paldino [.NET/C# MVP]"
<m...@spam.guard.caspershouse.com> wrote:
> I would test creating separate assemblies, and see if you see any
> noticable change in your app for the load you would expect to normally
> handle. If it works, then I would leave it at that. Only if it doesn't
> would I venture down this path.

Yes, I agree with that... I am really just investigating right now. I
just don't want to go down a design path that falls apart as it
scales.

I looked into the DynamicMethod class... I thought about what you
said... compile it into a new assembly and transfer the IL over to the
primary assembly. Is there a way to unload the temp assembly? I
don't know that I have ever UNloaded an assembly dynamically.

Thanks for ALL the help,
B

Nicholas
Tue Nov 27 08:22:16 PST 2007

Bilz,

If you compile the assembly, I don't know that it gets loaded by the
CLR. There might be a managed representation of it in memory, but I don't
know that it's considered loaded.

If it is loaded into memory though, you are going to have to do it in a
separate app domain, so that you can unload it.

Is the parsing that difficult to do? I ask, because if it is not, then
I would recommend building some sort of parser/graph tree for your
expressions and then creating the IL from that.

If you are using .NET 3.5/C# 3.0, you could even use Expressions to help
build your methods for you, as it should make it much, much easier.


--
- Nicholas Paldino [.NET/C# MVP]
- mvp@spam.guard.caspershouse.com

"Bilz" <BrianGenisio@gmail.com> wrote in message
news:a65699ab-f6cf-432b-be57-04b552667866@d4g2000prg.googlegroups.com...
> On Nov 26, 4:05 pm, "Nicholas Paldino [.NET/C# MVP]"
> <m...@spam.guard.caspershouse.com> wrote:
>> I would test creating separate assemblies, and see if you see any
>> noticable change in your app for the load you would expect to normally
>> handle. If it works, then I would leave it at that. Only if it doesn't
>> would I venture down this path.
>
> Yes, I agree with that... I am really just investigating right now. I
> just don't want to go down a design path that falls apart as it
> scales.
>
> I looked into the DynamicMethod class... I thought about what you
> said... compile it into a new assembly and transfer the IL over to the
> primary assembly. Is there a way to unload the temp assembly? I
> don't know that I have ever UNloaded an assembly dynamically.
>
> Thanks for ALL the help,
> B

Martin
Tue Nov 27 09:01:39 PST 2007

Bilz <BrianGenisio@gmail.com> writes:

> I looked into the DynamicMethod class... I thought about what you
> said... compile it into a new assembly and transfer the IL over to the
> primary assembly. Is there a way to unload the temp assembly? I
> don't know that I have ever UNloaded an assembly dynamically.

Assemblies are only unloaded, if the (last) AppDomain using it is
unloaded. So temporary assemblies should be loaded only in an own
AppDomain to allow later unloading.

Best regards,
Martin