Unicode OpenTextFile and Search Problem

I have written a script in vbs that basically opens a set of selected files,
performs a search on each file and replaces each occurrence of a string with
a different string. This works fine on ASCII files.

However the original files that I will be searching are Unicode files. With
these files, even though I have stated in the OpenTextFile command that the
files are to be opened as Unicode, the search finds nothing. I have
confirmed that the files are unicode by checking the first two characters
are ASCII 255 and 254. If I copy the contents of the Unicode file to a new
text file the search works fine.

What is my problem? Do I have to have a search string that is written in
Unicode? I'm using RegExp to do the search. I do not want to save or convert
the original files.

I thought I'd find a solution in the newsgroups quickly on this one. But
have had no success yet. Please help as the script is fully functional
except for this "simple" problem.

Thanx in advance.

Rob

tomthumbkop
Tue Apr 27 15:41:08 CDT 2004


What is the pattern you are searching for?

Robert Bevington wrote:
> *I have written a script in vbs that basically opens a set o
> selected files,
> performs a search on each file and replaces each occurrence of
> string with
> a different string. This works fine on ASCII files.
>
> However the original files that I will be searching are Unicod
> files. With
> these files, even though I have stated in the OpenTextFile comman
> that the
> files are to be opened as Unicode, the search finds nothing. I have
> confirmed that the files are unicode by checking the first tw
> characters
> are ASCII 255 and 254. If I copy the contents of the Unicode file t
> a new
> text file the search works fine.
>
> What is my problem? Do I have to have a search string that is writte
> in
> Unicode? I'm using RegExp to do the search. I do not want to save o
> convert
> the original files.
>
> I thought I'd find a solution in the newsgroups quickly on this one
> But
> have had no success yet. Please help as the script is full
> functional
> except for this "simple" problem.
>
> Thanx in advance.
>
> Rob


-
tomthumbko
-----------------------------------------------------------------------
Posted via http://www.codecomments.co
-----------------------------------------------------------------------

Robert
Wed Apr 28 02:34:53 CDT 2004

Hi Tom,

thanx for the reply. Sorry it's a bit late but I'm sitting in Germany.

My pattern is: "<Tu MatchPercent=" & Chr(34) & "([0-9]{1,2})" & Chr(34) &
">(.*?)<Tuv Lang=" & Chr(34) & "DE-DE" & Chr(34) & ">"

I even tried searching for something simple like "Match", but that did not
work either.

Any help would be great.

Rob

"tomthumbkop" <tomthumbkop.15e4s1@mail.codecomments.com> wrote in message
news:tomthumbkop.15e4s1@mail.codecomments.com...
>
> What is the pattern you are searching for?
>
> Robert Bevington wrote:
> > *I have written a script in vbs that basically opens a set of
> > selected files,
> > performs a search on each file and replaces each occurrence of a
> > string with
> > a different string. This works fine on ASCII files.
> >
> > However the original files that I will be searching are Unicode
> > files. With
> > these files, even though I have stated in the OpenTextFile command
> > that the
> > files are to be opened as Unicode, the search finds nothing. I have
> > confirmed that the files are unicode by checking the first two
> > characters
> > are ASCII 255 and 254. If I copy the contents of the Unicode file to
> > a new
> > text file the search works fine.
> >
> > What is my problem? Do I have to have a search string that is written
> > in
> > Unicode? I'm using RegExp to do the search. I do not want to save or
> > convert
> > the original files.
> >
> > I thought I'd find a solution in the newsgroups quickly on this one.
> > But
> > have had no success yet. Please help as the script is fully
> > functional
> > except for this "simple" problem.
> >
> > Thanx in advance.
> >
> > Rob *
>
>
>
> --
> tomthumbkop
> ------------------------------------------------------------------------
> Posted via http://www.codecomments.com
> ------------------------------------------------------------------------
>

Han
Wed Apr 28 02:29:30 CDT 2004

There are two kinds of unicode, utf-8 and utf-16. Try something like this

set strm=createobject("adodb.stream")
with strm
.Charset="utf-16" 'or utf-8
.type=2
.open
.loadfromfile "c:/x.xml"
t=.readText
.close
end with

msgbox t
"Robert Bevington" <rbevington@t-online.de> wrote in message
news:c6m5r5$4v1$02$1@news.t-online.com...
> I have written a script in vbs that basically opens a set of selected
files,
> performs a search on each file and replaces each occurrence of a string
with
> a different string. This works fine on ASCII files.
>
> However the original files that I will be searching are Unicode files.
With
> these files, even though I have stated in the OpenTextFile command that
the
> files are to be opened as Unicode, the search finds nothing. I have
> confirmed that the files are unicode by checking the first two characters
> are ASCII 255 and 254. If I copy the contents of the Unicode file to a new
> text file the search works fine.
>
> What is my problem? Do I have to have a search string that is written in
> Unicode? I'm using RegExp to do the search. I do not want to save or
convert
> the original files.
>
> I thought I'd find a solution in the newsgroups quickly on this one. But
> have had no success yet. Please help as the script is fully functional
> except for this "simple" problem.
>
> Thanx in advance.
>
> Rob
>
>
>

Robert
Wed Apr 28 04:02:04 CDT 2004

Hi Han,

thanx for your reply. However, as I'm not a script profi I would like to
explain my case somewhat further as your solution does not seem to help or
may be I have not understood it.

The files that I am processing are located on the file system and not in a
database; I'm presuming "adodb" is something to do with databases. I my
vbscript manual I cannot find any reference to Charset.

Here's a snippet of the function that opens the file and reads the content.

'Open file and get the text contents
Function GetFile(FileName)
If FileName<>"" Then
Dim fso, file
Set fso = CreateObject("Scripting.FileSystemObject")
on error resume Next
Set file = fso.OpenTextFile(FileName, ForReading, AsUnicode)
GetFile = file.ReadAll
End If
End Function

May be this will help you help me. Any assistance is greatly appreciated.

Rob

"Han" <hp4444@kornet.net.korea> wrote in message
news:e0uwPIPLEHA.3216@tk2msftngp13.phx.gbl...
> There are two kinds of unicode, utf-8 and utf-16. Try something like this
>
> set strm=createobject("adodb.stream")
> with strm
> .Charset="utf-16" 'or utf-8
> .type=2
> .open
> .loadfromfile "c:/x.xml"
> t=.readText
> .close
> end with
>
> msgbox t
> "Robert Bevington" <rbevington@t-online.de> wrote in message
> news:c6m5r5$4v1$02$1@news.t-online.com...
> > I have written a script in vbs that basically opens a set of selected
> files,
> > performs a search on each file and replaces each occurrence of a string
> with
> > a different string. This works fine on ASCII files.
> >
> > However the original files that I will be searching are Unicode files.
> With
> > these files, even though I have stated in the OpenTextFile command that
> the
> > files are to be opened as Unicode, the search finds nothing. I have
> > confirmed that the files are unicode by checking the first two
characters
> > are ASCII 255 and 254. If I copy the contents of the Unicode file to a
new
> > text file the search works fine.
> >
> > What is my problem? Do I have to have a search string that is written in
> > Unicode? I'm using RegExp to do the search. I do not want to save or
> convert
> > the original files.
> >
> > I thought I'd find a solution in the newsgroups quickly on this one. But
> > have had no success yet. Please help as the script is fully functional
> > except for this "simple" problem.
> >
> > Thanx in advance.
> >
> > Rob
> >
> >
> >
>
>

Christoph
Wed Apr 28 04:01:55 CDT 2004

28.04.2004 11:02, Robert Bevington schrieb:

> thanx for your reply. However, as I'm not a script profi I would like to
> explain my case somewhat further as your solution does not seem to help or
>
> Here's a snippet of the function that opens the file and reads the content.
>
> 'Open file and get the text contents
> Function GetFile(FileName)
> If FileName<>"" Then
> Dim fso, file
> Set fso = CreateObject("Scripting.FileSystemObject")
> on error resume Next
> Set file = fso.OpenTextFile(FileName, ForReading, AsUnicode)

Little logical error here. OpenTextFile has actually *4* params, the
*4th* of which specifies, if the stream is opened as ASCII/Unicode.x
If it is left out, the stream is opened as ASCII.
The *3rd* param is boolean and specifies if the file will be created
if it doesn't yet exist (useful only for writing/appending).

You should change this line to:

Set file = fso.OpenTextFile(FileName, ForReading, False, AsUnicode)

Make sure that 'ForReading' (value: 1) and 'AsUnicode' (value: -1)
are explicitly declared somewhere in your code prior to the call
of 'GetFile'.
Both are no intinsic keywords/constants in VBscript and/or WSH.

ffr:
http://msdn.microsoft.com/library/en-us/script56/html/jsmthopentextfile.asp
--
Gruesse, Christoph

Han
Wed Apr 28 08:49:09 CDT 2004

Yes, I know you're using text file with opentextfile, and not using
database.

However adodb.stream is used for every binary/text files. Especially when
the target is encoded of not Windows default, adodb.stream is useful. Why
not try the full code waiting only your execution?

"Robert Bevington" <rbevington@dspace.de> wrote in message
news:e$0$UcPLEHA.2576@TK2MSFTNGP12.phx.gbl...
> Hi Han,
>
> thanx for your reply. However, as I'm not a script profi I would like to
> explain my case somewhat further as your solution does not seem to help or
> may be I have not understood it.
>
> The files that I am processing are located on the file system and not in a
> database; I'm presuming "adodb" is something to do with databases. I my
> vbscript manual I cannot find any reference to Charset.
>
> Here's a snippet of the function that opens the file and reads the
content.
>
> 'Open file and get the text contents
> Function GetFile(FileName)
> If FileName<>"" Then
> Dim fso, file
> Set fso = CreateObject("Scripting.FileSystemObject")
> on error resume Next
> Set file = fso.OpenTextFile(FileName, ForReading, AsUnicode)
> GetFile = file.ReadAll
> End If
> End Function
>
> May be this will help you help me. Any assistance is greatly appreciated.
>
> Rob
>
> "Han" <hp4444@kornet.net.korea> wrote in message
> news:e0uwPIPLEHA.3216@tk2msftngp13.phx.gbl...
> > There are two kinds of unicode, utf-8 and utf-16. Try something like
this
> >
> > set strm=createobject("adodb.stream")
> > with strm
> > .Charset="utf-16" 'or utf-8
> > .type=2
> > .open
> > .loadfromfile "c:/x.xml"
> > t=.readText
> > .close
> > end with
> >
> > msgbox t
> > "Robert Bevington" <rbevington@t-online.de> wrote in message
> > news:c6m5r5$4v1$02$1@news.t-online.com...
> > > I have written a script in vbs that basically opens a set of selected
> > files,
> > > performs a search on each file and replaces each occurrence of a
string
> > with
> > > a different string. This works fine on ASCII files.
> > >
> > > However the original files that I will be searching are Unicode files.
> > With
> > > these files, even though I have stated in the OpenTextFile command
that
> > the
> > > files are to be opened as Unicode, the search finds nothing. I have
> > > confirmed that the files are unicode by checking the first two
> characters
> > > are ASCII 255 and 254. If I copy the contents of the Unicode file to a
> new
> > > text file the search works fine.
> > >
> > > What is my problem? Do I have to have a search string that is written
in
> > > Unicode? I'm using RegExp to do the search. I do not want to save or
> > convert
> > > the original files.
> > >
> > > I thought I'd find a solution in the newsgroups quickly on this one.
But
> > > have had no success yet. Please help as the script is fully functional
> > > except for this "simple" problem.
> > >
> > > Thanx in advance.
> > >
> > > Rob
> > >
> > >
> > >
> >
> >
>
>

Robert
Wed Apr 28 10:47:23 CDT 2004

Hi to everyone that answered,

and what was the solution. I suggest changing the programmer.

I missed the fact the the OpenTextFile has four parameters and my AsUnicode
param was being ignored in third place. This meant that the text file was
always opened as ascii.

Thanx Christoph for the solution. That's made my day. I now know that I'm
only a beginner and have a lot to learn.

Rob
"Christoph Basedau" <e_tonne@hotmail.com> wrote in message
news:408f72e8$0$26345$9b4e6d93@newsread4.arcor-online.net...
> 28.04.2004 11:02, Robert Bevington schrieb:
>
> > thanx for your reply. However, as I'm not a script profi I would like to
> > explain my case somewhat further as your solution does not seem to help
or
> >
> > Here's a snippet of the function that opens the file and reads the
content.
> >
> > 'Open file and get the text contents
> > Function GetFile(FileName)
> > If FileName<>"" Then
> > Dim fso, file
> > Set fso = CreateObject("Scripting.FileSystemObject")
> > on error resume Next
> > Set file = fso.OpenTextFile(FileName, ForReading, AsUnicode)
>
> Little logical error here. OpenTextFile has actually *4* params, the
> *4th* of which specifies, if the stream is opened as ASCII/Unicode.x
> If it is left out, the stream is opened as ASCII.
> The *3rd* param is boolean and specifies if the file will be created
> if it doesn't yet exist (useful only for writing/appending).
>
> You should change this line to:
>
> Set file = fso.OpenTextFile(FileName, ForReading, False, AsUnicode)
>
> Make sure that 'ForReading' (value: 1) and 'AsUnicode' (value: -1)
> are explicitly declared somewhere in your code prior to the call
> of 'GetFile'.
> Both are no intinsic keywords/constants in VBscript and/or WSH.
>
> ffr:
>
http://msdn.microsoft.com/library/en-us/script56/html/jsmthopentextfile.asp
> --
> Gruesse, Christoph