Screen scraping values from a page

[mightycpa]

Hi all,

I'm trying to grab some values off a public web page, not sure how to approach this. I looked at the underlying HTML, and while the container is pretty well defined, there don't seem to be any values I can capture.

The web page: https://finance.yahoo.com/chart/SPY#eyJ ... QifX19fQ--

Here's the things I want to capture:


Specifically, at the top left, I want to capture the current price. Then on the right, you can see colored boxes that have prices in them. They move up and down sometimes, and sometimes one will overlay another, so they might be a little difficult to grab. The long url ensures that the rectangle that I want to search in for those values will always remain the same size and in the same position in the same browser, and the colors will not vary either. Only the numbers.

I want to capture all the values in all the colors into variables: Red, Yellow, Fuschia, Kelly Green, Orange, Lime, Dirty Yellow, Deep Green.



Can somebody tell me how? I've never been too good at capturing this web stuff. I'm browser agnostic if that helps. Thanks in advance.

[Grovkillen]

Let me guess, they use canvas elements to draw these graphics? If so it's possible to inject Javascript code to snatch the info instead of using image recognition.

[mightycpa]

Thanks. It's difficult for me to say, because I don't understand what you're talking about!!

That said, when I look at the HTML, I think these two things are what make this page work:
1. The red box highlights for a few seconds when I hover over any of the colored prices
2. I think the ratechange object is the listener for the changing price.



It doesn't help me, but does is help you?

[Grovkillen]

Looks doable. But yes, you would need to inject Javascript and perhaps output the values to a external file or a custom HTML element.

[mightycpa]

I've asked some people to look into it, one tells me that the values are encrypted. Have a few more inquiries out there.

[Grovkillen]

Encrypted... How so? They are clearly visible and I don't see how they would be able to hide them except if they write them as bits/pixel values. I doubt it.

[hagchr]

If you click on "Try the new and improved charts" and "join beta" then it seems they have added a "Table View" where you can access all the Bollinger bands per day in table format. This should make your life much easier.

[mightycpa]

I just saw that today and joined up! They sure do make it easy!

[mightycpa]

@Grovkillen,

I don't know, I was just passing along what they told me. What I did learn is that if you just download the html, you won't find those values in there anywhere. So I suppose they are delivering pictures rather than text data. Maybe those pictures are created server-side, I'm not sure. I do know that none of the values can be captured as text from the screen.

But now, as hagchr points out, they're providing the data in text format in a table, which is a lot easier to deal with.

[Grovkillen]

Great you found a way. But for the future, those values are stored within the canvas and if you know a little Javascript it's relativity trivial to get the data.