New very subtle HTTPRequest problem/issue

[JRS]

Hello All,

Once again I'm experiencing issue(s) and challenges with
the HTTPRequest command. This time I think the issue(s)
is/are more subtle than with my Syntax command problem
that was expertly resolved from the help from this forum
a few days ago (as of this post).

I hope I'll explain my new issue/problem adequately. If
I don't use the correct terminology please excuse me.

First in a nutshell here's what I'm trying to accomplish:

There are several web pages and some applications that
post the updated status of flights; IN FLIGHT, ARRIVED,
DELAYED, ETA etc.

My application is the user/subscriber E-MAils me a
Flight Notification request with the date they want to
be notified, airline name or abbreviation, flight number
and a list of E-mail addresses for multiple flight
status notifications.

This concept is nothing new and several airlines and
flight tracking web pages already offer services in a
similar regard or context but what I have in mind is
taking the basic concept and enhancing the service, for
instance (off the cuff) just E-Mail certain addresses in
the subscriber's list just if the given flight has a
DELAYED status. The complexity of what can realized is
open ended.

For a "proof of concept" I use http://www.fboweb.com as a flight
status "source". In "production" I'll have to get a
flight status source that I'll pay for/subscribe to. I
consider whatever interaction coding/logic will be
replaced by my "formal" flight status source.

From the fight status request E-mail I derive an Airline
abbreviation (i.e. if the subscriber's spells out the
Airline instead of already providing the abbreviation)

Following is a sample of my E-Mail request which the
subscriber sends to a special E-Mail address I
designate:

+==
Name Of Airline [Either complete name or use abbreviation]: IDE
Flight Number: 1198
Date: 11/30/05
Service Code: 1200
E-Mail Address(es):
[email protected]
[email protected]
[email protected]
==+

Here the (hypothetical) subscriber already provides me the
airline abbreviation for Independence Air and flight
1193 is a daily hour flight between Huntsville, Alabama
and Washington D.C. that departed today 6:13 AM Central
Time (CT U.S.) from Huntsville and arrived a little late
8:26 AM Eastern Standard Time (EST U.S) in Washington D.C.
with a scheduled ETA of 8:22 AM EST. Essentially the
in-flight time takes a little more than an hour as a time
zone is been crossed.

From the reguest the following initial URL is derived:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

resulting in http://www.fboweb.com that has a status frame such as
the following:

Information as of 11/30/2005 1:37:31 PM GMT
FlightID: IDE1193 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Dest Apt: IAD WASHINGTON DULLES INTERNATIONAL
Departed: 06:13 AM CST (1213Z)
Original ETA: 08:22 AM EST (1322Z)
Actual Arrival Time: 08:26 AM EST (1326Z)

Using Microsoft IE or Firefox if I do a "Page Source" (or an
HTTPRequest) of:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

I don't get the preceding text of the frame ... but I do get a
"reference to it":



The substring: /30/f/it.aspx?f= is unique as well as the frame
code number: 105001892.

In my Macro Scheduler script I search for: /30/f/it.aspx?f=
and ultimately derive the following URL where I do another
HTTPRequest:

http://www.fboweb.com/30/f/it.aspx?f=105001892

Now I get the "frame".

Now here's where the "weirdness starts" and I can only
put it best the where I thought an HTTPRequest results
the HTMLResponse variable being "packed" with the
equivalent of a the current Page Source ... apparently
this is not occurring.

My frame "reference URL" for IDE 1193 yesterday
11/29/05 was/is:

http://www.fboweb.com/30/f/it.aspx?f=105031249

If you bring up the URL in your browser your result
(probably) will show the following in a stand-
alone window:

Information as of 11/29/2005 10:49:14 PM GMT
FlightID: IDE1198 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: IAD WASHINGTON DULLES INTERNATIONAL
Dest Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Departed: 04:07 PM EST (2107Z)
Original ETA: N / A
Actual Arrival Time: 04:49 PM CST (2249Z)

Notice the Origin and Destination are reversed.

Later in the afternoon yesterday I did a:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

The URL resulted with the Origin and Destinations (correctly)
reversed with the flight destination IAD and the arrival time of
the frame maintained on the main page.

When I did a "Source Page" I got the same "105031249" code and
on initiating the:

http://www.fboweb.com/30/f/it.aspx?f=105031249

URL on my browser I got the preceding "inverse result".

=== That's my first subtle issue. ===

Today I did a fight request while IDE 1193 was: IN FLIGHT

When I did "Source Page" for:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

from my either browser IE or Firefox ... sure enough there
was "105001892" today's Tuesday 11/30/05 code for IDE 1193.

However, when I did an HTTPRequest for the same URL:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

I got yesterday's page (results) with the "105031249" code.

My intention is to use HTTPRequest instead of having to
inefficiently/ineffectively interact with the browser "camping
out" on the web page but with the subtle erratic behavior I'm
experiencing I'm at a loss. It's like the HTTPRequest isn't
playing with a full deck (which probably isn't the case),
something is lost in transmission and/or http://www.fboweb.com's
somehow sending info out to the HTTPRequest and posting
something else (probably is the case) ...

I can always revert back to my old code that interacts with
http://www.fboweb.com from a browser (which contains the status frame)
on the main page but I want to avoid this if at all possible.

Am I describing something that can't be avoided? Has anyone
else experience anything similar in another Macro Scheduler
app that uses HTTPRequest where HTTPRequest gets
different results from doing a "Source Page"?
Is there a certain setting I have to make?

Thanks again for any help.

Regards,

Joel S.

[Marcus Tettmar]

Hi,

I just did this:

Let>URL=http://www.fboweb.com/cob/google.asp?ai ... htnum=1193
HTTPRequest>URL,,GET,rHTML
MessageModal>rHTML

And manually pulled out the iframe src from the message box and then did:

//Let>URL=http://www.fboweb.com/cob/google.asp?ai ... htnum=1193
Let>URL=http://www.fboweb.com/30/f/it.aspx?f=105001892
HTTPRequest>URL,,GET,rHTML
MessageModal>rHTML

So am getting correct info. Will try again tomorrow if you like and see what happens ....

[JRS]

Once again thank you for your indulgence with me
on this strange HTTPRequest problem I'm experiencing.
What I think makes it so strange and hard to pin
down is it seems at different times of day or on
different days I'm experiencing different results
focusing on (no pun intended!) the "frame" URL
interacting with the http://www.fboweb.com page/site
with respect to the "main page" and Macro Scheduler
using the HTTPRequest.

For instance the following (not even using Macro
Scheduler): when I'm using the "105031249" from
yesterday's Monday 11/29/05 I'm getting the
following results which makes it appear there is
an IDE 1193 flight in progress from Washington
D.C. back to Huntsville, Alabama today Tuesday
11/30/05.

But on doing the "main page" http://www.fboweb.com URL:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

I get the destination Washington D.C. and the
main page and the frame coinciding in the same
state as the flight arrival in Washington D.C.
from this morning (Tuesday 11/30/05). As you can see
from the first frame URL I'm getting an 8:17:46 PM GMT
date/time stamp on my "outdated" URL which
concides with the 3:17:46 EST (US) I initiated the
the first URL.

You'll notice the main page and today's "105001892"
remains at 1:37:31 PM GMT. I did these URL's a
minute or two after the first thus 3:19 PM or 3:20
PM EST (or respectively 8:19 PM or 8:20 PM GMT)
showing no flights from Washington D.C. to
Huntsville, Alabama on the http://www.fboweb.com for
IDE 1193.

Perhaps if I did one of my Macro Scheduler E-Mail
requests as I described it will show the results of
the frame on the main page. But with this strange
web page I'm not so sure.

If you could redo a try tomorrow I would greatly
appreciate it. It jsut seems there's something
weird going on with the frames that show on the
http://www.fboweb.com main page that sometimes and
sometimes not coincide with taking the frame
in a context of its own window.

Perhaps, as the Flight is in progress the same
issue will crop up again. All I can say is you
take any flight on http://www.fboweb.com and compare
the main page which contains the frame and the
frame in its own window and/or do a Macro Scheduler
HTTPRequest for the main page for Flight IDE 1193

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

it can/may be different results than what you see
on the browser such as Firefox or Microsoft IE.
Can you think of any reason?

============================================

http://www.fboweb.com/30/f/it.aspx?f=105031249

Information as of 11/30/2005 8:17:46 PM GMT
FlightID: IDE1198 (Commercial Jet)
Status: IN FLIGHT
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Ground Spd: 346 kts Altitude: 22,300
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: IAD WASHINGTON DULLES INTERNATIONAL
Dest Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Departed: 03:05 PM EST (2005Z)
Original ETA: 03:33 PM CST (2133Z)
Calculated ETA: 03:43 PM CST (2143Z), In 1h 20m
Dist Fm Dept: 61 nm Dist To Dest: 460 nm

=============================================

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

Information as of 11/30/2005 1:37:31 PM GMT
FlightID: IDE1193 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Dest Apt: IAD WASHINGTON DULLES INTERNATIONAL
Departed: 06:13 AM CST (1213Z)
Original ETA: 08:22 AM EST (1322Z)
Actual Arrival Time: 08:26 AM EST (1326Z)

=============================================

http://www.fboweb.com/30/f/it.aspx?f=105001892

Information as of 11/30/2005 1:37:31 PM GMT
FlightID: IDE1193 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Dest Apt: IAD WASHINGTON DULLES INTERNATIONAL
Departed: 06:13 AM CST (1213Z)
Original ETA: 08:22 AM EST (1322Z)
Actual Arrival Time: 08:26 AM EST (1326Z)

=============================================

[JRS]

All ...

For anyone who's been following this new subtle and convoluted
HTTPRequest issue thread hopefully the following will put it
into succinct context. But in doing so I'm left no less
perplexed and incredulous how I'm experiencing the difference
between my HTTPRequest results and the results of going to
the browser (whether Firefox or Microsoft IE) and doing a "Source
Page" ... and my finding is nothing to do with anything wrong
with Macro Scheduler ...

To maintain consistency I'm still using the "Main Page"
http://www.fboweb.com URL for IDE (Independence Air) flight #: 1193

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

Take any 3 character airline name abbreviation and a valid
flight number for the airline and an http://www.fboweb.com "Main
Page" will appear for the flight. Within the main page there
is a "frame" with the current flight status: ARRIVED, IN FLIGHT
etc.

If you do "Page Source" from the browser (Firefox or IE) you
won't see the frame data but you will see a 9-digit reference
number or code" to the frame. The identifier string for the
code is: /30/f/it.aspx?f=

Now what you do (and what my Macro Scheduler script does)
is take the identifier string and the 9-digit code and
concatenate to string the "base" http://www.fboweb.com
string to realize the frame in its own window.

Example:

http://www.fboweb.com/30/f/it.aspx?f=105001892

It seems (this is really weird) http://www.fboweb.com uses a discrete
codes for each flight day in and out. IDE 1193's appears to be:

For Independence Air Flight #1193 (IDE 1193) the code appears to
be: 105001892

Independence Airline FLight #1198 is a daily afternoon from
Washington D.C back to Huntsville, Alabama.

For Independence Air Flight #1198 (IDE 1198) the code appears to
be: 105031249

For the moment disregard Macro Scheduler and focus on (no
pun intended) just the browser. At different times of the
day accessing the IDE 1193 "main page" URL:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

will either show the legit code doing a "Page source" in
your browser resulting in the "legit" frame (1) for IDE 1193

-or- ....

a "transpositon" of IDE #1198 data or frame reference from the
IDE 1193 URL "call" (2)

I verified this to be the case checking the status from the
"horse's mouth" so to speak ... in other words Independence Air's
own flight status for its own flight 1193, there is no return
Flight 1193 from Washington D.C. to Huntsville, Alabama.
However, there is an IDE Flight 1198 from Washington D.C. to
Huntsville, Alabama. (3) and (4)

What seems to be happening is in the morning (CST and EST US)
if you initiate a:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

the main "Source Page" from the browser will correctly show a
105001892 frame reference code.

Later in the day (afternoon "my" US EST or CST) the same URL:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

on "Source Paging" from my browser I'm getting the code for
a IDE 1198 frame: 105031249

Then still later on (next day) early morning (5AM) same URL:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

"Source paging" from the browser I'm reverting to the "correct"
105001892 result again.

If this is not complicated enough on doing a HTTPRequest
of: http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

while IDE 1193 is IN FLIGHT I'm still realizing the
incorrect 105031249 or IDE 1198 "transposition" in my
HTMLResponse variable from my HTTPRequest call to URL:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

Has anyone else ever experienced any kind of "transposition"
making a "call" to the same URL using HTTPRequest with different
results concurrently occurring in the browser doing a
"Source Page"?

I don't know if there is anything that could be done ...
(Hmmmm ... multiple HTTPRequest calls to the same URL is
something I haven't tried yet. Maybe the third or fourth
HTTPRequest call in quick succession to:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

will "right the HTTPRequest return" to the "correct" 105001892
result.

I'm just wondering if anyone else who has looked into what I'm
describing (the best I can) for themselves has experienced the
Flight IDE 1198 "transposition results" from a URL:

http://www.fboweb.com/cob/google.asp?ai ... htnum=1193

call using their browser?

I know the this description of the issue is verbose but the
issue is really a subtle one.

Thanks,

Joel S.

===============================================

(1)


http://www.fboweb.com/30/f/it.aspx?f=105001892

Information as of 11/30/2005 1:37:31 PM GMT
FlightID: IDE1193 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Dest Apt: IAD WASHINGTON DULLES INTERNATIONAL
Departed: 06:13 AM CST (1213Z)
Original ETA: 08:22 AM EST (1322Z)
Actual Arrival Time: 08:26 AM EST (1326Z)

===============================================

(2)


http://www.fboweb.com/30/f/it.aspx?f=105031249

Information as of 11/30/2005 9:38:47 PM GMT
FlightID: IDE1198 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: IAD WASHINGTON DULLES INTERNATIONAL
Dest Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Departed: 03:05 PM EST (2005Z)
Original ETA: 03:33 PM CST (2133Z)
Actual Arrival Time: 03:39 PM CST (2139Z)

===============================================

(3)

http://www.flyi.com/tools/FlightStatus/ ... =12/1/2005

Thursday, December 01, 2005

Huntsville, AL (HSV) To Washington-Dulles, VA (IAD)
FLIGHT#

1193 STATUS

ON TIME DESTINATION


Boarding Gate: 3
Boarding Time: 5:30 AM

Scheduled Departure: 6:00 AM
Estimated Departure: 6:00 AM

Scheduled Arrival: 8:40 AM
Estimated Arrival: 8:40 AM

Flight Duration: 1 hrs 40 mins

=============================================
(4)

http://www.flyi.com/tools/FlightStatus/ ... =12/1/2005

Thursday, December 01, 2005

Washington-Dulles, VA (IAD) To Huntsville, AL (HSV)
FLIGHT#

1198 STATUS

ON TIME DESTINATION


Boarding Gate: A5
Boarding Time: 2:15 PM

Scheduled Departure: 2:45 PM
Estimated Departure: 2:45 PM

Scheduled Arrival: 3:42 PM
Estimated Arrival: 3:42 PM

Flight Duration: 1 hrs 57 mins

=============================================
(5)

Going into the following Macro Scheduler script segment
the first MessageModal>%HoldURL% (the first command)
The HoldURL variable is "packed with" string:
http://www.fboweb.com/cob/google.asp?ai ... htnum=1198

.
.
.
MessageModal>%HoldURL%
DeleteFile>c:\flight\Flight_Http_dump2.txt
HTTPRequest>HoldURL,c:\flight\Flight_Http_Frame_dump2.txt,GET,,HTMLResponse
HTTPRequest>HoldURL,,GET,,HTMLResponse
StringReplace>HTMLResponse,",^,HTMLResponse
Position>/30/f/it.aspx?f=,HTMLResponse,1,StartPos,FALSE
Let>StartPos=StartPos+16
Let>InterimVal=
MidStr>HTMLResponse,%StartPos%,9,InterimVal
MessageModal>%InterimVal%
Let>HoldURL=http://www.fboweb.com/30/f/it.aspx?f=
ConCat>HoldURL,InterimVal
MessageModal>%HoldURL%

=============================================

[Marcus Tettmar]

I really think the only way to answer this properly would be to ask the webmaster of the site what his script does. It may well do something based on the browser, or your location, or ... something passed by or dependent on the browser. Maybe it uses a cookie - all sorts of things could be going on which will be bypassed by HTTPRequest. My point being there are all kinds of reasons why a server may serve up different content in IE than to HTTPRequest.

[JRS]

Thanks for the reply. Ah, so an HTTPrequest may not appear like
a browser ... interesting ... I tried it again just now and same result.

From Firefox browser: 105001892

Information as of 12/1/2005 1:42:45 PM GMT
FlightID: IDE1193 (Commercial Jet)
Status: ARRIVED
Owner: INDEPENDENCE AIR
Radio Call: INDEPENDENCE
Type: CRJ2 (Canadair Bombardier RJ-200 Regional Jet)
Origin Apt: HSV HUNTSVILLE INTL-CARL T JONES FIELD
Dest Apt: IAD WASHINGTON DULLES INTERNATIONAL
Departed: 06:12 AM CST (1212Z)
Original ETA: 08:23 AM EST (1323Z)
Actual Arrival Time: 08:32 AM EST (1332Z)

Browser is correct, HTTPRequest/HTMLResponse not
returning the 105031249 HTML.

LOL me trying to talk to the webmaster ... It's hard enough
for me to contact http://www.fboweb.com to subscribe to their
flight tracking.

There are other flight status sites with same info. I just know to be
careful to keep especially initally for a site I plan to access
often to lookout that my HTTPRequest/HTMLResponse
contains a valid return.

Thanks - end of discussion. If you want to delete my verbose thread
be my guest.

Joel S.

[JRS]

One very quick addendum or after note if I could ....
The "transposition" problem I described in this thread seemed
to have gone away in the interim.

So very strange.

Anyway thanks again.

Joel S.