I have a large local database (about 65,000 records) and need to be
able to populate a grid with continually updated information. The
search is performed so that partial names, and other info can be
entered in any order. As soon as the record appears enter will end
the interactivesearch. So in the interactivechange event of a text
field on the form I have placed the following code. In the setup form
I give the the user the option of turning off the incremental search
(interactivechange event).

"
IF butintch=.t.
loopvar=getwordcount(this.value)
filtervalue=''
FOR counter = 1 TO loopvar
filtervalue=filtervalue+'"'+ GETWORDNUM(this.Value,counter)+ '"'+" $
namepath and "
ENDFOR
filtervalue=UPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))
SELECT karaoke
SET FILTER TO &filtervalue
GO top
thisform.grid1.refresh
endif

"

On a small database this works great but when used with a 65,000
record table the filter slows the program considerably. Is there any
way to perform a search such as this without using a filter to narrow
the results down. I realize that the continual calling of the set
filter to is causing the bottleneck but cannot see anyother way to
perform this kind of search. The field it is searching on is 150
characters. and includes the filename and path.

It is for a karaoke song look up. The file structure is such that
directories could appear as below.
c:\karaoke\full discs\ah8001
and an example filename would be
ah8001-01 - Healey, Jeff - Angel Eyes.zip

The use might only remember that some guy named Jeff sung a song about
eyes and the disc number was 8001

So with the above search box I could type in
"
jef eye 8001
"
and the program would narrow it down to this file.

Any help would be appreciated. Thank you in advance.

Re: Search using locate taking too long by swdev2

swdev2
Wed Apr 09 01:28:42 CDT 2008

Have you tried to make your table optimized with RUSHMORE ?
basically, you set up an index that matches the search criteria, or in this
case, the filter expression of
UPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))

so you might set up an index (wait a minute) ..
eek - you have embedded double quote in your filter string.
hmmmmm

I think I'm missing something -

show me the last value inside the for loop of filtervalue variable, prior to
the loop exit,
and the value of
filtervalue=UPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))

Then maybe I can formulate a good index expression for RUSHMORE.
Also look at the RUSHMORE topic in the help file, it might give you some
more ideas about
creating an index expression.

Mondo Regards [Bill]

--
===================
William Sanders / EFG VFP / mySql / MS-SQL
www.efgroup.net/vfpwebhosting
www.terrafox.net www.viasqlserver.net

"Graham" <grahamskaraoke@gmail.com> wrote in message
news:12579e7c-18ed-4105-801a-6dd516f1db83@k37g2000hsf.googlegroups.com...
> I have a large local database (about 65,000 records) and need to be
> able to populate a grid with continually updated information. The
> search is performed so that partial names, and other info can be
> entered in any order. As soon as the record appears enter will end
> the interactivesearch. So in the interactivechange event of a text
> field on the form I have placed the following code. In the setup form
> I give the the user the option of turning off the incremental search
> (interactivechange event).
>
> "
> IF butintch=.t.
> loopvar=getwordcount(this.value)
> filtervalue=''
> FOR counter = 1 TO loopvar
> filtervalue=filtervalue+'"'+ GETWORDNUM(this.Value,counter)+ '"'+" $
> namepath and "
> ENDFOR
> filtervalue=UPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))
> SELECT karaoke
> SET FILTER TO &filtervalue
> GO top
> thisform.grid1.refresh
> endif
>
> "
>
> On a small database this works great but when used with a 65,000
> record table the filter slows the program considerably. Is there any
> way to perform a search such as this without using a filter to narrow
> the results down. I realize that the continual calling of the set
> filter to is causing the bottleneck but cannot see anyother way to
> perform this kind of search. The field it is searching on is 150
> characters. and includes the filename and path.
>
> It is for a karaoke song look up. The file structure is such that
> directories could appear as below.
> c:\karaoke\full discs\ah8001
> and an example filename would be
> ah8001-01 - Healey, Jeff - Angel Eyes.zip
>
> The use might only remember that some guy named Jeff sung a song about
> eyes and the disc number was 8001
>
> So with the above search box I could type in
> "
> jef eye 8001
> "
> and the program would narrow it down to this file.
>
> Any help would be appreciated. Thank you in advance.



Re: Search using locate taking too long by Stefan

Stefan
Wed Apr 09 03:27:07 CDT 2008

In addition to Bill's comment -
FoxPro does not natively support "fulltext" indexing. There used
to be a brillant tool called phDbase www.hallogram.com/phdbase
Or you can use a backend that does have that feature (if you
cannot avoid the requirement by restructuring the data design).

Another optimization issue in your example is that VFP's Set Filter
feature is extremly quick - but only for "invisible" aliases, not when
you Browse a filtered alias or use it as grid.RecordSource.
That's because the filter expression gets evaluated almost permanently,
e.g on mouse move.
Parameterized (Local) Views make a better grid.RecordSource.
In Vfp9, the new grid.Optimize property tries to handle that issue.

Open Database your.dbc
Create View vTest As ;
Select ... ;
Where Lower( Nvl( <yourFieldExpr>, '' ) Like Lower(?yourParameter)
...
Use vTest NoData In 0
Local yourParameter
yourParameter = '%song title%'
Requery("vTest")




hth
-Stefan


"Graham" <grahamskaraoke@gmail.com> schrieb im Newsbeitrag
news:12579e7c-18ed-4105-801a-6dd516f1db83@k37g2000hsf.googlegroups.com...
>I have a large local database (about 65,000 records) and need to be
> able to populate a grid with continually updated information. The
> search is performed so that partial names, and other info can be
> entered in any order. As soon as the record appears enter will end
> the interactivesearch. So in the interactivechange event of a text
> field on the form I have placed the following code. In the setup form
> I give the the user the option of turning off the incremental search
> (interactivechange event).
>
> "
> IF butintch=.t.
> loopvar=getwordcount(this.value)
> filtervalue=''
> FOR counter = 1 TO loopvar
> filtervalue=filtervalue+'"'+ GETWORDNUM(this.Value,counter)+ '"'+" $
> namepath and "
> ENDFOR
> filtervalue=UPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))
> SELECT karaoke
> SET FILTER TO &filtervalue
> GO top
> thisform.grid1.refresh
> endif
>
> "
>
> On a small database this works great but when used with a 65,000
> record table the filter slows the program considerably. Is there any
> way to perform a search such as this without using a filter to narrow
> the results down. I realize that the continual calling of the set
> filter to is causing the bottleneck but cannot see anyother way to
> perform this kind of search. The field it is searching on is 150
> characters. and includes the filename and path.
>
> It is for a karaoke song look up. The file structure is such that
> directories could appear as below.
> c:\karaoke\full discs\ah8001
> and an example filename would be
> ah8001-01 - Healey, Jeff - Angel Eyes.zip
>
> The use might only remember that some guy named Jeff sung a song about
> eyes and the disc number was 8001
>
> So with the above search box I could type in
> "
> jef eye 8001
> "
> and the program would narrow it down to this file.
>
> Any help would be appreciated. Thank you in advance.



--
|\_/| ------ ProLib - programmers liberty -----------------
(.. ) Our MVPs and MCPs make the Fox run....
- / See us at www.prolib.de or www.AFPages.de
-----------------------------------------------------------



Re: Search using locate taking too long by Anders

Anders
Tue Apr 08 18:04:49 CDT 2008

CREATE CURSOR Words (searchfor Char(20)
FOR i = 1 TO GetWordCount(M.loopvar,' ,')
INSERT INTO Words VALUES (GETWORDNUM(M.loopvar, i))
NEXT
SELECT Karaoke.* FROM Karaoke JOIN Words ;
ON Karaoke.namepath LIKE '%'+TRIM(Words.searchfor)+'%';
INTO CURSOR xx

Your are of course running the risk that they download 65000 rows. You can
guard against that.
One way would be using a pre-count
SELECT COUNT(*) FROM Karaoke JOIN Words ;
ON Karaoke.namepath LIKE '%'+TRIM(Words.searchfor)+'%';
INTO ARRAY xx.
If there are more than 250 rows, tell the user to please be more specific.

-Anders


"Graham" <grahamskaraoke@gmail.com> wrote in message
news:12579e7c-18ed-4105-801a-6dd516f1db83@k37g2000hsf.googlegroups.com...
>I have a large local database (about 65,000 records) and need to be
> able to populate a grid with continually updated information. The
> search is performed so that partial names, and other info can be
> entered in any order. As soon as the record appears enter will end
> the interactivesearch. So in the interactivechange event of a text
> field on the form I have placed the following code. In the setup form
> I give the the user the option of turning off the incremental search
> (interactivechange event).
>
> "
> IF butintch=.t.
> loopvar=getwordcount(this.value)
> filtervalue=''
> FOR counter = 1 TO loopvar
> filtervalue=filtervalue+'"'+ GETWORDNUM(this.Value,counter)+ '"'+" $
> namepath and "
> ENDFOR
> filtervalue=UPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))
> SELECT karaoke
> SET FILTER TO &filtervalue
> GO top
> thisform.grid1.refresh
> endif
>
> "
>
> On a small database this works great but when used with a 65,000
> record table the filter slows the program considerably. Is there any
> way to perform a search such as this without using a filter to narrow
> the results down. I realize that the continual calling of the set
> filter to is causing the bottleneck but cannot see anyother way to
> perform this kind of search. The field it is searching on is 150
> characters. and includes the filename and path.
>
> It is for a karaoke song look up. The file structure is such that
> directories could appear as below.
> c:\karaoke\full discs\ah8001
> and an example filename would be
> ah8001-01 - Healey, Jeff - Angel Eyes.zip
>
> The use might only remember that some guy named Jeff sung a song about
> eyes and the disc number was 8001
>
> So with the above search box I could type in
> "
> jef eye 8001
> "
> and the program would narrow it down to this file.
>
> Any help would be appreciated. Thank you in advance.



Re: Search using locate taking too long by Graham

Graham
Wed Apr 09 12:43:11 CDT 2008

Here is a sample of the filter swdev2

To use the below example of some guy named jeff singing something with
eyes in it on a disc numbered 8001, the filter would be as follows
"
Set Filter to "JEFF" $ NAMEPATH AND "EYES" $ NAMEPATH AND "8001" $
NAMEPATH
"

I don't know how I can make an optimized index file on the above
without including each word as a seperate index expression which could
conceivably give me an index expression containing 300,000 or more
pointers.

Stefan do you have any suggestions on a backend that supports the
"fulltext" indexing. I went looking for the phdbase and it appears to
no longer be available.

Also I am not sure on the parameterized views. Guess I have to do
some reading on these.

Anders, I think I am a little confused about using a cursor as I think
the nature of the beast would be to download the whole cursor the
first time that they entered the search string. What I am trying to
accomplish is to populate the grid with the most minimum info to try
to help the user narrow their search. I'm also a little rusty on
using cursors.


Thank you everyone.

On Apr 8, 7:04=A0pm, "Anders Altberg" <anders.altberg> wrote:
> CREATE CURSOR Words (searchfor Char(20)
> FOR i =3D 1 TO GetWordCount(M.loopvar,' ,')
> =A0INSERT INTO Words VALUES (GETWORDNUM(M.loopvar, i))
> NEXT
> SELECT Karaoke.* FROM Karaoke JOIN Words ;
> ON Karaoke.namepath LIKE '%'+TRIM(Words.searchfor)+'%';
> INTO CURSOR xx
>
> Your are of course running the risk that they download 65000 rows. You can=

> guard against that.
> One way would be using a pre-count
> SELECT COUNT(*) FROM =A0Karaoke JOIN Words ;
> ON Karaoke.namepath LIKE '%'+TRIM(Words.searchfor)+'%';
> INTO ARRAY xx.
> If there are more than 250 rows, tell the user to please be more specific.=

>
> -Anders
>
> "Graham" <grahamskara...@gmail.com> wrote in message
>
> news:12579e7c-18ed-4105-801a-6dd516f1db83@k37g2000hsf.googlegroups.com...
>
>
>
> >I have a large local database (about 65,000 records) and need to be
> > able to populate a grid with continually updated information. =A0The
> > search is performed so that partial names, and other info can be
> > entered in any order. =A0As soon as the record appears enter will end
> > the interactivesearch. =A0So in the interactivechange event of a text
> > field on the form I have placed the following code. =A0In the setup form=

> > I give the the user the option of turning off the incremental search
> > (interactivechange event).
>
> > "
> > IF butintch=3D.t.
> > loopvar=3Dgetwordcount(this.value)
> > filtervalue=3D''
> > FOR counter =3D 1 TO loopvar
> > filtervalue=3Dfiltervalue+'"'+ GETWORDNUM(this.Value,counter)+ '"'+" $
> > namepath and "
> > ENDFOR
> > filtervalue=3DUPPER(LEFT(filtervalue,LEN(ALLTRIM(filtervalue))-4))
> > SELECT karaoke
> > SET FILTER TO &filtervalue
> > GO top
> > thisform.grid1.refresh
> > endif
>
> > "
>
> > On a small database this works great but when used with a 65,000
> > record table the filter slows the program considerably. =A0Is there any
> > way to perform a search such as this without using a filter to narrow
> > the results down. =A0I realize that the continual calling of the set
> > filter to is causing the bottleneck but cannot see anyother way to
> > perform this kind of search. =A0The field it is searching on is 150
> > characters. and includes the filename and path.
>
> > It is for a karaoke song look up. =A0The file structure is such that
> > directories could appear as below.
> > c:\karaoke\full discs\ah8001
> > and an example filename would be
> > ah8001-01 - Healey, Jeff - Angel Eyes.zip
>
> > The use might only remember that some guy named Jeff sung a song about
> > eyes and the disc number was 8001
>
> > So with the above search box I could type in
> > "
> > jef eye 8001
> > "
> > and the program would narrow it down to this file.
>
> > Any help would be appreciated. =A0Thank you in advance.- Hide quoted tex=
t -
>
> - Show quoted text -


Re: Search using locate taking too long by swdev2

swdev2
Wed Apr 09 15:24:24 CDT 2008

Yikes.
I would suggest you look at Regular Experssions and how to parse them -
They are supposed to be mondo fast when properly formed.
See http://fox.wikis.com/wc.dll?Wiki~RegExp~VFP for all the Scoop.

Good Luck ! [Bill]
--
===================
William Sanders / EFG VFP / mySql / MS-SQL
www.efgroup.net/vfpwebhosting
www.terrafox.net www.viasqlserver.net

"Graham" <grahamskaraoke@gmail.com> wrote in message
news:791b8cb0-dda3-46b0-bee3-8eb96d394549@l42g2000hsc.googlegroups.com...
Here is a sample of the filter swdev2

To use the below example of some guy named jeff singing something with
eyes in it on a disc numbered 8001, the filter would be as follows
"
Set Filter to "JEFF" $ NAMEPATH AND "EYES" $ NAMEPATH AND "8001" $
NAMEPATH
"
[snip]



Re: Search using locate taking too long by Bernhard

Bernhard
Thu Apr 10 04:27:26 CDT 2008

Hi Graham

> Stefan do you have any suggestions on a backend that supports the
> "fulltext" indexing. I went looking for the phdbase and it appears to
> no longer be available.
MySQL has some fulltext indexing support:
http://dev.mysql.com/doc/refman/6.0/en/fulltext-search.html

I have not yet worked with it and I think other DB servers also have such a feature.

Regards
Bernhard Sander

Re: Search using locate taking too long by Stefan

Stefan
Fri Apr 11 01:28:41 CDT 2008


"Bernhard Sander" <fuchs@no.spam> schrieb im Newsbeitrag
news:%23VcBY0umIHA.5944@TK2MSFTNGP03.phx.gbl...
> Hi Graham
>
>> Stefan do you have any suggestions on a backend that supports the
>> "fulltext" indexing. I went looking for the phdbase and it appears to
>> no longer be available.
> MySQL has some fulltext indexing support:
> http://dev.mysql.com/doc/refman/6.0/en/fulltext-search.html
>
> I have not yet worked with it and I think other DB servers also have such a feature.

Right on, same for PostgreSQL, MS SqlServer Express, IBM
DB2, Oracle and others.

IIRC, the PostgreSQL approach works a little like Anders'
suggestion, and could be implemented in VFP too:
insert/update/delete triggers would check each word of a text
whether it occurs in a secondary "wordindex" table and if not,
do something like "Insert Into wordindex (foreignKey,word) ..."
That approach might probably slow down updates a little but
increase query performance.


hth
-Stefan



--
|\_/| ------ ProLib - programmers liberty -----------------
(.. ) Our MVPs and MCPs make the Fox run....
- / See us at www.prolib.de or www.AFPages.de
-----------------------------------------------------------