HTML screen scraping and IE Object Models

Have been writing cool code (believe it or not) in VBA, for going against
certain web sites and collecting data for the database.
Everything works well with Excel's VBA going against IE data and parsing the
HTML and getting innerHTML's etc.
This is with Office 2007, Windows XP, IE 70.0.5730.13

But when you use the same VBA with Vista, Office 2007 and IE 7.0.6000.16389,
it is totally different.
DOM is 90% different and nothing works. How come? How can there be so much
difference.

Example:
in XP something like this, works :
Set oE = createObject("InternetExplorer.Application") : oE.Visible = Ture :
oE.Navigate http://www.junk.com" : iForms = oE.Document.forms.length

But in Vista and IE 7.0.6000.16389
Set oE = createObject("InternetExplorer.Application") : oE.Visible = Ture :
oE.Navigate http://www.junk.com" is fine....
But then after that.....it becomes oE.Application and then
oE.Application.Document .....and the rest does not make any sense.

What is the Problem? Is the progId ("InternetExplorer.Application") does
not apply anymore?

Also, does anybody know the best tool and/or environemnt to write Scraping
vb scripts against web pages?
My vba stuff works fine, but occasionally IE dies due to leaks and garbage
collection and stops functioning after a few thousand hits.

Anything will help

Travis
Thu Jul 10 12:29:14 CDT 2008

What is the best way of loading the DOM and then walking through the
Elements>
Does anybody have any sampe code for Screen Scraping database output from
HTML text?

Regards


"Travis McGee" <travisGatesMcGee@hotmail.com> wrote in message
news:e4Dc60E4IHA.4448@TK2MSFTNGP05.phx.gbl...
> Have been writing cool code (believe it or not) in VBA, for going against
> certain web sites and collecting data for the database.
> Everything works well with Excel's VBA going against IE data and parsing
> the HTML and getting innerHTML's etc.
> This is with Office 2007, Windows XP, IE 70.0.5730.13
>
> But when you use the same VBA with Vista, Office 2007 and IE
> 7.0.6000.16389, it is totally different.
> DOM is 90% different and nothing works. How come? How can there be so
> much difference.
>
> Example:
> in XP something like this, works :
> Set oE = createObject("InternetExplorer.Application") : oE.Visible = Ture
> : oE.Navigate http://www.junk.com" : iForms = oE.Document.forms.length
>
> But in Vista and IE 7.0.6000.16389
> Set oE = createObject("InternetExplorer.Application") : oE.Visible = Ture
> : oE.Navigate http://www.junk.com" is fine....
> But then after that.....it becomes oE.Application and then
> oE.Application.Document .....and the rest does not make any sense.
>
> What is the Problem? Is the progId ("InternetExplorer.Application") does
> not apply anymore?
>
> Also, does anybody know the best tool and/or environemnt to write Scraping
> vb scripts against web pages?
> My vba stuff works fine, but occasionally IE dies due to leaks and garbage
> collection and stops functioning after a few thousand hits.
>
> Anything will help
>
>

mr_unreliable
Sat Jul 12 08:56:52 CDT 2008

Travis McGee wrote:
> Also, does anybody know the best tool and/or environemnt to write Scraping
> vb scripts against web pages? My vba stuff works fine, but occasionally
> IE dies due to leaks and garbage collection and stops functioning after
> a few thousand hits.
>
> Anything will help
>

hi Travis, there have been numerous complaints posted here
about scripting with IE7, although I can't tell whether or
not they are applicable to your situation.

If you are looking for something "new-and-different", you
might try downloading with microsoft's xmlHTTP object, which
has little-or-nothing to do with IE. Here's a little code
to show how to download a webpage:

--- <code> ---
Dim xmlHTTP : Set xmlHTTP = CreateObject("Microsoft.XMLHTTP")

Const sSource = "http://www.dilbert.com"
Const bGetAsAsync = False ' wait for response

' formulate a request to get the Dilbert website...
xmlHTTP.Open "GET", sSource, bGetAsAsync
xmlHTTP.Send ' send it (to the web, wait for result)

sHTMLPage = xmlHTTP.responseText ' (note: as TEXT)
--- </code> ---

For something even more basic (lower level) than that,
(and if you are really desperate), since you are coding
in vba you can use the system api directly. The library
with the interfaces to the internet is "wininet.dll",
and the declarations are downloadable in (vb) module
form as "wininet.bas", found here:

http://support.microsoft.com/kb/q185519/

As microsoft says, the "wininet.bas" doesn't contain
every interface declaration, but it does contain most of
them. And you will have to use msdn to find more specific
information on exactly HOW to use the interfaces. If you
are looking for sample code that uses wininet.dll, there
is the "Slurp" demo written in vb, and posted on
Karl E. Peterson's site:

http://vb.mvps.org/samples/project.asp?id=Slurp

Versions of IE may come and go, but I am doubtful that
microsoft will change the basic system interfaces anytime
soon.

cheers, jw
____________________________________________________________

You got questions? WE GOT ANSWERS!!! ..(but,
no guarantee the answers will be applicable to the questions)