Extract Regular Data

[dsn076]

I am having a rough time extracting pieces of data over several web pages. I use a csv file to feed URLs into the regular data extraction process. There are server variables over a series of pages. The problem I am running into is my output will vary even if I run the task over the same URLs multiple times. For example I will mine the following variables:

Product Name
Sale Price
Date
Time

I will log these variables to a .txt or .csv file. The output is vary over the same URLs. On the first run I may grab 4 pages of data just fine then the sale price, or product name will be missing for several other pages. Then I will run the same automation over the same URLs and then complete different lines of data will be missing. There seems to be no consistency as to when it will pull the data and when it will miss it.

I thought that maybe the automation script was moving too fast and some of the data was not fully loading so I used a delay to ensure the page had completed loaded, but that did not help. I have tried to extract the data associated with the exact html tag and it will work for several pages of data, then just totally mess up. Again, if I run the automation script over the same set of test URLs then the results will vary each time.

Any suggestions? HELP

[dsn076]

By using several if statements along with loops I was able to get the data I need each time however it adds about 6 sec to the automation. This ends up to be a lot of time when going over 1000 pages each go around. Why does AA not consistently grab the data and am I forced to tell it to loop until the data is grabbed? Is there a better way?

[forumstaff]

Hello,

Based on the description provided above it seems that the website coding is getting changed for each record, it could be possible that the website might be getting updated frequently. This is reason you might have to set different
IF condition to take care of different pattern.

Please let us know if you can share the Website with us, this will help us to analyze the problem. You can send the link of website to support@automationanywhere.com in case you do not want to share on forum.

[dsn076]

Originally Posted by forumstaff

Hello,

Based on the description provided above it seems that the website coding is getting changed for each record, it could be possible that the website might be getting updated frequently. This is reason you might have to set different
IF condition to take care of different pattern.

Please let us know if you can share the Website with us, this will help us to analyze the problem. You can send the link of website to support@automationanywhere.com in case you do not want to share on forum.

I will email the website you you guys. The site uses AJAX I believe. The loop system I have setup works, but it will occasionally get caught on a loop never grabbing the data and looping continuously. Other times the task will stop on a step I think waiting for the web page to load? This happens about 1 to 2 times out of a 1000.

The reason why I think AA may wait for the page is because I have noticed the page load to 100% then a second loading will appear for a few microseconds. I think that it is possible that some of the content due to being AJAX is not finished loading although the HTML has loaded. This may confuse AA.

I will email the URL so you can suggest a solution.

Thank you

[forumstaff]

Hi,

Thanks for the note. We will be on the lookout for your e-mail.