Web scrape slow speed

[ZFalcon]

I was having an issue and hoped there was a solution. When I go to scrape regular data from a website (i.e. stock data that may change multiple times per second) it doesn't seem to extract the data fast enough to move on to the next step and misses what I need quite a bit.

I am using the standard web recorder and the command in the final script is simply:

Extract data from Control type : Division, Webpage (link is here) to $CurPrc$

With only one variable extracted it isn't totally horrible but still too slow. When I go to extract multiple pieces of data at the same time it misses more and more as it pauses slightly at each line as can be seen by the step counter while the script is running. As an example if there are three things I want to scrape from the page continuously like:

Extract data from Control type : Division, Webpage : (link is here) to $CurPrc$
Extract data from Control type : Division, Webpage : (link is here) to $PctCg$
Extract data from Control type : Division, Webpage : (link is here) to $TlPcv$

This takes a lot of time prior to getting to the next step after the data is completely extracted. With 4-10 pieces of data being scraped it can take well over a second, in which time the first part of the data has already changed.

How can I speed this up? My computer is state of the art, and I'm on a 40 Megabit connection to the net so there is nothing that can be done with hardware or internet speeds.

Any ideas?

[forumsupport]

Hello,

Can you please upload your task so we can look into it and suggest solution to speed up task execution?

You can locate your task by selecting your task from Task List, right clicking that particular task and selecting 'Locate on disk' from context menu.

It would help us investigate further.

[ZFalcon]

Hello and thank you for the response.

I have attached a sample file that illustrates the problem I am having. The best way I could show it is from a simple time and date site instead of the actual site my real script works on since the area I am accessing is password protected.

What you will see with this attached script is the exact same problem with the speed that I am having where it pauses each time it goes to extract data from the webpage by roughly 1/2 a second. When I am trying to extract multiple pieces of data until certain criteria are met, this causes a delay of more than a second which just doesn't work.

If you run the attached script and watch the counter as it is executing, it has this delay at lines 3 and 13 where I have it extracting data from the site. The other lines are just there as filler so you can see where the pause is occurring.

With my real script running, due to this pause, by the time it extracts the data and runs through the script the data is outdated. If I had to put a number on it for seconds, the entire loop of 30 lines in my real script including the data extraction of just 2 items would need to execute entirely in 1/2 a second or less for the data to be current enough for my purposes. Please let me know how to accomplish this.

Thank you in advance for your help. Your programs flexibility and the support you provide is amazing. It is greatly appreciated.

[forumsupport]

Hello,

We can also use 'Get Text' option of 'Manage Web Controls' command for faster data extraction.

We have amended the task for your reference. Please copy it under location, 'C:\...\My Documents\Automation Anywhere\Automation Anywhere\My Tasks' folder.

[ZFalcon]

I tried the sample script you attached (removing the popup) and it still extracts very slow. If you look at how long it pauses on the extraction line it is still over 1/2 a second for each piece of data it extracts. Using something such as a packet sniffer I can see a LOT more data coming in much faster compared to what Automation Anywhere is capturing. I would just use the sniffer but it doesn't have any other commands and just sees the raw data in real time.

I also tried highlighting the text I wanted to extract and copying it to the clipboard then reading from that to see if it was any better. This doesn't work either though as it seems that windows doesn't give the clipboard high priority and sometimes the data is lagged due to windows being too slow with the copy/paste.

Do you have any other suggestions for how to speed this extraction up in Automation Anywhere? I love the program, but really need it to perform at a reasonable speed when extracting this data.

[forumsupport]

Hello,

Speed of data extraction also depends on machine and internet speed.

It took half a second at our end to extract the data. We tested the task on 7-8 machines in our lab and on all the machines, 'Get Text' command extracted the data within half a second.

Another way is, using keystrokes to copy the entire webpage and using String Manipulation command extract the required data from $Clipboard$ variable. Attached is the sample task that runs faster. To run this task, please copy it under location, 'C:\...\My Documents\Automation Anywhere\Automation Anywhere\My Tasks' folder. Create test.txt under D:\.

We logged the extracted into test.txt to re-view the time.

Hope that helps.