procedure in separate files = slow execution?

I had an application with a large procedure file. The application uses SET
PROCEDURE TO <this file>

I'm writing another application which uses a few of the procedures in the
file. I don't want to include the large procedure file in my new application
so I took the common procedures out of the large file and created a separate
.PRG for each one.

These procedures are used in small loops and when I build the original
application, the code runs at least twice as slowly now these procedures are
in separate files.

Is this a known effect?

(having written that post and thought a bit,) I'm thinking I should #INCLUDE
the files in the procedure file. But I find it odd that performance has been
affected; the whole thing is built into a single exe, why should code
location in the source have such a negative impact?

--
TIA
Andrew Howell

Andrew
Wed Oct 20 05:29:33 CDT 2004

Andrew Howell wrote:
> (having written that post and thought a bit,) I'm thinking I should
> #INCLUDE the files in the procedure file. But I find it odd that
> performance has been affected; the whole thing is built into a single
> exe, why should code location in the source have such a negative
> impact?


Errrrr, there's no such thing as #INCLUDE :-\
SURELY I've missed something?

I think it's time to take a break.

--
Regards
Andrew Howell

Andrew
Wed Oct 20 05:43:22 CDT 2004

Andrew Howell wrote:
> Andrew Howell wrote:
> I think it's time to take a break.

Errrr, I've had a break. I put the code back into the large procedure file
and removed the separate .PRGs and rebuilt. Sure enough the application ran
quickly again.

I removed the procedures from the procedure file and rebuilt. Sure enough,
the separate files were added to the project again but now the application
is still running quickly.

Does anyone have a plausible explanation for what went on here or should I
pretend it never happened?
I'd already done a "rebuild all" when I first split the procedures out so I
know that's out of the question.

--
WTF?
Andrew Howell

Andrew
Wed Oct 20 05:50:20 CDT 2004

Andrew Howell wrote:
> Andrew Howell wrote:
>> Andrew Howell wrote:
>> I think it's time to take a break.
>
> Errrr, I've had a break. I put the code back into the large procedure
> file and removed the separate .PRGs and rebuilt. Sure enough the
> application ran quickly again.
>
> I removed the procedures from the procedure file and rebuilt. Sure
> enough, the separate files were added to the project again but now
> the application is still running quickly.

(head spinning) My bad, sorry. I'd commented out the code that triggered all
these procedures.

My original question still stands, I'm just a bit angrier with FPW now. Why
does having a bunch of separate procedures in separate .PRGs run a LOT
slower than having them in a single file which is opened with SET PROCEDURE?

And why can't I include files at compile time?

--
Regards
Andrew Howell

Andrew
Wed Oct 20 06:39:12 CDT 2004

Andrew Howell wrote:
> Andrew Howell wrote:
>> Andrew Howell wrote:
>>> Andrew Howell wrote:

> And why can't I include files at compile time?

Workaround found:

Create screen called "procs" and
#INSERT <procfile>
in a code snippet somewhere.

Then
SET PROCEDURE TO procs.spr

Unfortunately it seems that a file that is #INSERTed must have a full path,
if I use a relative path it cannot find the file so it's still no good for
me, I keep source and development branches of code and copy from dev to
source when I put new modifications in. I don't want to have to change a
hardcoded path in the code each time, it's too messy.

--
Regards
Andrew Howell

GaryZ
Wed Oct 20 07:59:04 CDT 2004

"Andrew Howell" wrote:

> Workaround found:
>
> Create screen called "procs" and
> #INSERT <procfile>
> in a code snippet somewhere.
>
> Then
> SET PROCEDURE TO procs.spr

Andrew,

By now, you probably realize this is what you'd've gotten if
you'd've copied the needed procs from the original procedure file into a new
procedure file, except it would have a prg extension.

However, this is an interesting round-about way of creating a procedure file.

GaryZ

Rick
Wed Oct 20 08:19:47 CDT 2004

Andrew,
In VFP, I've found that using separate files in a project actually can =
run faster. However a lot does depend on where the code file exists - on =
the file server or a copy on each local machine, and how the separate =
procs/funcs are called. Ultimately in a .EXE or .APP, all the individual =
pieces are in one file, with what ammounts to a table-of-contents with =
links to each component of the project.

Rick

"Andrew Howell" <ajh@work> wrote in message =
news:uLiU8votEHA.444@TK2MSFTNGP10.phx.gbl...
>I had an application with a large procedure file. The application uses =
SET
> PROCEDURE TO <this file>
>=20
> I'm writing another application which uses a few of the procedures in =
the
> file. I don't want to include the large procedure file in my new =
application
> so I took the common procedures out of the large file and created a =
separate
> .PRG for each one.
>=20
> These procedures are used in small loops and when I build the original
> application, the code runs at least twice as slowly now these =
procedures are
> in separate files.
>=20
> Is this a known effect?
>=20
> (having written that post and thought a bit,) I'm thinking I should =
#INCLUDE
> the files in the procedure file. But I find it odd that performance =
has been
> affected; the whole thing is built into a single exe, why should code
> location in the source have such a negative impact?
>=20
> --
> TIA
> Andrew Howell
>=20
>

Andrew
Wed Oct 20 08:29:33 CDT 2004

GaryZ wrote:
> Andrew,
>
> By now, you probably realize this is what you'd've gotten if
> you'd've copied the needed procs from the original procedure file
> into a new procedure file, except it would have a prg extension.
>
> However, this is an interesting round-about way of creating a
> procedure file.
>
> GaryZ

Well, yes. To expand a little, there are about 10 files I want to include,
not just 1. Unfortunately there is no such thing as

SET PROCEDURE TO xxx ADDITIVE

the other obvious way to make it work is to have 2 sets of the procedures;
one in the procedure file for my (sales order processing) application and
the other in a procedure file for my (auto-importing) application. But my
original idea was to _not_ have multiple copies of the same procedures, I
did not expect it to make things slow.

The only thing I can think that makes it slow is Foxpro's search order for
procedures:
1) look in executing program
2) look in procedure file
3) look in chain of executing programs
4) look for separate file called procedure name

They have moved from case 2) to case 4)
I develop on a fairly slow machine because I can get a good feel for whether
or not the code is running fast enough. In this case it most definately is
not but I even tested on one of the target machines thinking "If it goes OK
on that then I'll do it like this anyway" but it's just as slow on the
target machine.

At this rate I'm going to duplicate the code and have 2 separate procedure
files but I cannot see a good way out of this.
Why no #INCLUDE !?

--
Regards
Andrew Howell

Andrew
Wed Oct 20 08:40:22 CDT 2004

Rick Bean wrote:
> Andrew,
> In VFP, I've found that using separate files in a project actually
> can run faster. However a lot does depend on where the code file
> exists - on the file server or a copy on each local machine, and how
> the separate procs/funcs are called. Ultimately in a .EXE or .APP,
> all the individual pieces are in one file, with what ammounts to a
> table-of-contents with links to each component of the project.

Rick,
the situation I have is a shared .EXE living on the server that the
clients run. Most things are called in this fashion:

value=function(parameter1,parameter2...)

A couple of things are called in this fashion:

value=-1
DO function WITH value, parameter1, parameter2...



I thought it would end up resolving the links something like you describe
but the observed behaviour makes me think it could even be looking on disc
before finding it in the EXE.

It is unacceptably slow, even on a fast machine. As soon as I paste the code
back into the main procfile it runs "instantly". The fact that it runs just
as slowly on a machine several times faster than my dev box makes me wonder
if it's thrashing the network a bit. Time to run filemon I think.
It's good that they seem to have fixed it in later versions of Fox but this
app isn't a great candidate to move up to VFP (it's the most complex/central
component in our systems and we are _still_ using Fox as an interim solution
<g> (been running for more than twice the original planned lifetime))

--
Regards
Andrew Howell

Ook
Wed Oct 20 09:07:03 CDT 2004

Andrew, have you compiled it into an EXE and tested performance? With VFP,
generally speaking, using PRG files slows things down in the development
environment, but not when you build an EXE. Build an EXE and see how it
performs.

"Andrew Howell" <ajh@work> wrote in message
news:uLiU8votEHA.444@TK2MSFTNGP10.phx.gbl...
> I had an application with a large procedure file. The application uses SET
> PROCEDURE TO <this file>
>
> I'm writing another application which uses a few of the procedures in the
> file. I don't want to include the large procedure file in my new
application
> so I took the common procedures out of the large file and created a
separate
> .PRG for each one.
>
> These procedures are used in small loops and when I build the original
> application, the code runs at least twice as slowly now these procedures
are
> in separate files.
>
> Is this a known effect?
>
> (having written that post and thought a bit,) I'm thinking I should
#INCLUDE
> the files in the procedure file. But I find it odd that performance has
been
> affected; the whole thing is built into a single exe, why should code
> location in the source have such a negative impact?
>
> --
> TIA
> Andrew Howell
>
>

Andrew
Wed Oct 20 09:13:30 CDT 2004

Andrew Howell wrote:
> It is unacceptably slow, even on a fast machine. As soon as I paste
> the code back into the main procfile it runs "instantly". The fact
> that it runs just as slowly on a machine several times faster than my
> dev box makes me wonder if it's thrashing the network a bit. Time to
> run filemon I think.

Confirmed.

Every function call, it is actually looking for a file called <functionname>
on disc through the whole SET('PATH') then failing BEFORE it finds the links
within the .EXE itself.

Silly thing :-(

The other options it brings to mind are to either put these files on disc
(but I don't really want source kicking around the data directory though)
or keep swapping procedure files throughout my calling programs so it finds
these procedures before scanning the whole path.

--
Regards
Andrew Howell

Andrew
Wed Oct 20 09:17:33 CDT 2004

Ook wrote:
> Andrew, have you compiled it into an EXE and tested performance? With
> VFP, generally speaking, using PRG files slows things down in the
> development environment, but not when you build an EXE. Build an EXE
> and see how it performs.

It's FPW and it's built to an EXE.
I've just discovered+posted the reason why to Rick - it's looking for the
files on disc every function call, through the whole path before failing and
then finding it in the EXE :-\

--
Regards
Andrew Howell

Fred
Wed Oct 20 10:23:13 CDT 2004

Another possible slow down in FPW with separate files: if you have FUNCTION
or PROCEDURE as the first line of your code, that runs slower than just
leaving it off. I think this was something that got fixed in VFP, but was a
problem in FPW/FPD.

--
Fred
Microsoft Visual FoxPro MVP


"Andrew Howell" <ajh@work> wrote in message
news:upPeM%23qtEHA.3016@TK2MSFTNGP12.phx.gbl...
> Ook wrote:
>> Andrew, have you compiled it into an EXE and tested performance? With
>> VFP, generally speaking, using PRG files slows things down in the
>> development environment, but not when you build an EXE. Build an EXE
>> and see how it performs.
>
> It's FPW and it's built to an EXE.
> I've just discovered+posted the reason why to Rick - it's looking for the
> files on disc every function call, through the whole path before failing
> and
> then finding it in the EXE :-\
>
> --
> Regards
> Andrew Howell
>
>

Rick
Wed Oct 20 12:51:28 CDT 2004

Andrew,
Actually the "ADDITIVE" clause of SET PROCEDURE has been available in =
VFP since at least 5.0 (1995!).

Rick

"Andrew Howell" <ajh@work> wrote in message =
news:%23CHHZjqtEHA.3292@TK2MSFTNGP12.phx.gbl...
> GaryZ wrote:
>> Andrew,
>>
>> By now, you probably realize this is what you'd've gotten if
>> you'd've copied the needed procs from the original procedure file
>> into a new procedure file, except it would have a prg extension.
>>
>> However, this is an interesting round-about way of creating a
>> procedure file.
>>
>> GaryZ
>=20
> Well, yes. To expand a little, there are about 10 files I want to =
include,
> not just 1. Unfortunately there is no such thing as
>=20
> SET PROCEDURE TO xxx ADDITIVE
>=20
> the other obvious way to make it work is to have 2 sets of the =
procedures;
> one in the procedure file for my (sales order processing) application =
and
> the other in a procedure file for my (auto-importing) application. But =
my
> original idea was to _not_ have multiple copies of the same =
procedures, I
> did not expect it to make things slow.
>=20
> The only thing I can think that makes it slow is Foxpro's search order =
for
> procedures:
> 1) look in executing program
> 2) look in procedure file
> 3) look in chain of executing programs
> 4) look for separate file called procedure name
>=20
> They have moved from case 2) to case 4)
> I develop on a fairly slow machine because I can get a good feel for =
whether
> or not the code is running fast enough. In this case it most =
definately is
> not but I even tested on one of the target machines thinking "If it =
goes OK
> on that then I'll do it like this anyway" but it's just as slow on the
> target machine.
>=20
> At this rate I'm going to duplicate the code and have 2 separate =
procedure
> files but I cannot see a good way out of this.
> Why no #INCLUDE !?
>=20
> --
> Regards
> Andrew Howell
>=20
>

Andrew
Thu Oct 21 03:01:42 CDT 2004

Fred Taylor wrote:
> Another possible slow down in FPW with separate files: if you have
> FUNCTION or PROCEDURE as the first line of your code, that runs
> slower than just leaving it off. I think this was something that got
> fixed in VFP, but was a problem in FPW/FPD.

Thanks for the answers everyone.

Good call Fred. This makes it noticeably faster although filemon is still
showing foxpro searching the whole path for a file of the function name
before giving up and finding it in the .EXE itself. There is absolutely no
sense in this: If I place a file of the relevant name in the search path,
FPW finds it, DOESN'T execute it and uses the procedure from the file built
in the EXE anyway.

I cannot find a good way around this problem, all the workarounds I've tried
have been imperfect in one way or another to me. I'll leave them as separate
files without FUNCTION or PROCEDURE at the beginning and make do with that.

--
Regards
Andrew Howell

Andrew
Thu Oct 21 03:55:12 CDT 2004

Andrew Howell wrote:
> Fred Taylor wrote:
>> Another possible slow down in FPW with separate files: if you have
>> FUNCTION or PROCEDURE as the first line of your code, that runs
>> slower than just leaving it off. I think this was something that got
>> fixed in VFP, but was a problem in FPW/FPD.
>
> Thanks for the answers everyone.
>
> Good call Fred. This makes it noticeably faster although filemon is
> still showing foxpro searching the whole path for a file of the
> function name before giving up and finding it in the .EXE itself.

Actually, I'm wrong there (there were a couple of other functions I'd
forgotten.)

Having PROCEDURE or FUNCTION as the first line in a separate program makes
FPW search the path every call. Removing PROCEDURE or FUNCTION (from all
separate programs <g>) stops this search and this eliminates the slowdown. I
guess they got something a bit kooky in the routine that searches for the
symbols/links in FPW.

Not the first time I owe you Fred ;)
Many thanks, I'm happy again.

--
Regards
Andrew Howell

Zeke
Thu Oct 21 09:22:12 CDT 2004

Is this really a relevant discussion? In a data centric application the time
that the database engine spends moving record pointers, filtering and other
heave data related tasks is 90 % of the execution time or more, and is
probably completely or almost completely independant on how the procedures
are organized. So - in order to optimize your application you should
primarily optimize the data centric code.

Agree?

Zeke


"Rick Bean" <rgbean@unrealmelange-inc.com> wrote in message
news:%23bbQ6dqtEHA.272@TK2MSFTNGP12.phx.gbl...
Andrew,
In VFP, I've found that using separate files in a project actually can run
faster. However a lot does depend on where the code file exists - on the
file server or a copy on each local machine, and how the separate
procs/funcs are called. Ultimately in a .EXE or .APP, all the individual
pieces are in one file, with what ammounts to a table-of-contents with links
to each component of the project.

Rick

"Andrew Howell" <ajh@work> wrote in message
news:uLiU8votEHA.444@TK2MSFTNGP10.phx.gbl...
>I had an application with a large procedure file. The application uses SET
> PROCEDURE TO <this file>
>
> I'm writing another application which uses a few of the procedures in the
> file. I don't want to include the large procedure file in my new
application
> so I took the common procedures out of the large file and created a
separate
> .PRG for each one.
>
> These procedures are used in small loops and when I build the original
> application, the code runs at least twice as slowly now these procedures
are
> in separate files.
>
> Is this a known effect?
>
> (having written that post and thought a bit,) I'm thinking I should
#INCLUDE
> the files in the procedure file. But I find it odd that performance has
been
> affected; the whole thing is built into a single exe, why should code
> location in the source have such a negative impact?
>
> --
> TIA
> Andrew Howell
>
>

Andrew
Thu Oct 21 09:37:21 CDT 2004

Zeke Galama wrote:
> Is this really a relevant discussion? In a data centric application
> the time that the database engine spends moving record pointers,
> filtering and other heave data related tasks is 90 % of the execution
> time or more, and is probably completely or almost completely
> independant on how the procedures are organized. So - in order to
> optimize your application you should primarily optimize the data
> centric code.
>
> Agree?

I think almost everything I've read by Rick Bean in this newsgroup has been
relevant to a greater or lesser extent.

In my application the bulk of the work is heaving data about. However, when
I reorganised some code, that action (and that alone) caused the execution
time to take many times longer than before, the result being about 1 second
to flick between "previous" and "next" records (instead of fast enough to
keep up with full key repeat rate when holding down the "next" and
"previous" keys.)

But luckily it's all solved now :-) It seems to be due to a quirk in FPW
searching the network unnecessarily.

Sorry if I missed your point, it has been a long day.
--
Regards
Andrew Howell

Zeke
Thu Oct 21 09:50:03 CDT 2004

Well - I'm not arguing against Rick's advise - just had the impression that
you were busy optimizing somethings that is not very important from
performance point of view.
Zeke



"Andrew Howell" <ajh@work> wrote in message
news:unsW6t3tEHA.1336@tk2msftngp13.phx.gbl...
> Zeke Galama wrote:
> > Is this really a relevant discussion? In a data centric application
> > the time that the database engine spends moving record pointers,
> > filtering and other heave data related tasks is 90 % of the execution
> > time or more, and is probably completely or almost completely
> > independant on how the procedures are organized. So - in order to
> > optimize your application you should primarily optimize the data
> > centric code.
> >
> > Agree?
>
> I think almost everything I've read by Rick Bean in this newsgroup has
been
> relevant to a greater or lesser extent.
>
> In my application the bulk of the work is heaving data about. However,
when
> I reorganised some code, that action (and that alone) caused the execution
> time to take many times longer than before, the result being about 1
second
> to flick between "previous" and "next" records (instead of fast enough to
> keep up with full key repeat rate when holding down the "next" and
> "previous" keys.)
>
> But luckily it's all solved now :-) It seems to be due to a quirk in FPW
> searching the network unnecessarily.
>
> Sorry if I missed your point, it has been a long day.
> --
> Regards
> Andrew Howell
>
>

Fred
Thu Oct 21 10:03:34 CDT 2004

Great! There's yet another quirk in versions of VFP that's probably even
worse, though it only seems to be a problem at startup. If you've used UNC
naming and have portions of your application (classlibs, etc.) on different
servers, when you go to execute your .EXE if the server isn't avaialable,
there's a several second delay before your .EXE starts. If the server is
available, no delay occurs. Once the delay happens, it doesn't appear to
occur again other than at startup.

VFP8 has SYS(2450) that should help with this. I've only noticed the
problem with UNC naming, but even drive letter shares should benefit.

--
Fred
Microsoft Visual FoxPro MVP


"Andrew Howell" <ajh@work> wrote in message
news:OE$Pwu0tEHA.3056@TK2MSFTNGP10.phx.gbl...
> Andrew Howell wrote:
>> Fred Taylor wrote:
>>> Another possible slow down in FPW with separate files: if you have
>>> FUNCTION or PROCEDURE as the first line of your code, that runs
>>> slower than just leaving it off. I think this was something that got
>>> fixed in VFP, but was a problem in FPW/FPD.
>>
>> Thanks for the answers everyone.
>>
>> Good call Fred. This makes it noticeably faster although filemon is
>> still showing foxpro searching the whole path for a file of the
>> function name before giving up and finding it in the .EXE itself.
>
> Actually, I'm wrong there (there were a couple of other functions I'd
> forgotten.)
>
> Having PROCEDURE or FUNCTION as the first line in a separate program makes
> FPW search the path every call. Removing PROCEDURE or FUNCTION (from all
> separate programs <g>) stops this search and this eliminates the slowdown.
> I
> guess they got something a bit kooky in the routine that searches for the
> symbols/links in FPW.
>
> Not the first time I owe you Fred ;)
> Many thanks, I'm happy again.
>
> --
> Regards
> Andrew Howell
>
>