A Full Data Scraping Solution?

[TylerMitton]

I've been using the trial version of AutomationAnywhere for a while now and want to get some opinions from experienced users.

I would really appreciate your feedback on what you foresee as the major challenges if I were to use AutomationAnywhere as a data scraping solution where:

• All scraping tasks must be run on the same machine (a win 2008 server)
• Tasks take data from websites, ftp locations or java apps and insert to SQL server 2008
• I have to be able to track the status of all tasks in a central location
• I have to be able to stop individual tasks manually when I want to
• Multiple tasks must be able to run at the same time
• Up to 5 tasks will be set to run once per minute
• A total of over 30 tasks will be scheduled to run at varying intervals
• Tasks must start running on their schedules after a cold boot with no user login or action of any kind

What do you think would be the major challenges?

I've had several conversations with "Automation Experts" and sales reps and they all tell me that everything is "possible", but I want some feedback from people that have tried to make these things happen and work together... as I have had some problems when I try to get them to work seamlessly.

Thanks so much for your time!

Tyler

[forumsupport]

Hello,

Regarding your queries,

. You can create exe of scrapping tasks and can run those exes simultaneously. Automation Anywhere Premier/Enterprise license provide 'Create Exe' feature that allows you to create independent exes of task. These exes can run on several machines without needing the software to be installed on that machine.
For Create Exe demo, please visit: http://www.tethyssolutions.com/AADeployExe.html

. You can use Web Recorder tool to extract structured or unstructured data from any website into user variables and then using Database command, you can connect SQL Server database and issue 'Insert SQL Statement' to transfer the extracted data into database.
You can view demos on how Web Recorder works at,
Automation Anywhere - Learning Center
Automation Anywhere - Learning Center
Automation Anywhere - Learning Center

You can view online sample tasks for web data extraction at, http://www.tethyssolutions.com/autom...dataextraction

. Can you please share more information on status that you want to track? You can use 'Log to file' command in your task to log the status in a common text file.

. Use the Task Manager to kill the specific exe.

. Unfortunately, you cannot run multiple tasks at the same time on same machine. However, multiple exes of the tasks can be run at the same time on same machine.

. You can set the repeat options of the task as per the requirement.

. Set Schedule property to schedule tasks to run at varying interval of time. You can also set 'Time between Repeats' option to repeat the task after certain interval.

. Schedule the tasks to run when computer starts. You can enable Auto-login option and specify the Auto-login Settings to run the task even when the machine is logged off. From Security property You can set the task to run in stealth mode and can also disable mouse and keyboard actions for the task.

To get started easily, you can view the demo on Automation Anywhere 5.5 usage, at Automation Anywhere - Learning Center

In addition, there are many online sample tasks available at Automation Anywhere Examples - The Automation Software

We also have a manual for Automation Anywhere 5.5. For that, please refer the AAUserManual.pdf link given below:
http://www.automationanywhere.com/su...UserManual.pdf

Hope that helps.

[TylerMitton]

Thanks for your detailed response... there are just a few concerns that I would like cleared up:

• Creating EXEs limits the logging capability, right? I don't see any log information when I schedule and run EXE's I have created (clicking on View Log does nothing)
• You'd asked what I'd like to track: I want to track any time a task has been run, how long it took, and if it completed successfully.
• Is there no central location where I can control (and stop) the tasks? You suggested I use the Task Manager to kill specific tasks... that seems odd.
• I specified that I will be using Windows Server 2008, but I have heard that Auto-login doesn't work with that... are you saying that it does?

If anyone that doesn't work for AA has some feedback on this stuff I would love to hear from you!

Cheers!

[forumsupport]

Hello,

. You can create custom report using 'Log to File' command. Create a task that would include 'If Task Successful/If Task Unsuccessful' command to monitor the task execution process. This command will run the Automation Anywhere task and depending on return code decide if it was successful or not. You can log the result inside If-Endif block. You can also log the start time and end time using system variables, $Hour$, $Minute$ and $Second$ to get duration of task. The text format of task would appear something like this,

1) Log to File: Start Time- $Hour$:$Minute$:$Second$ in "C:\VIewLog.txt"

2) If Task Successful ("My Documents\Automation Anywhere\Automation Anywhere\My Tasks\temp.atmn") Then

3) Log to File: Task completed successfully. in "C:\VIewLog.txt"

4) End If

5) Log to File: End Time- $Hour$:$Minute$:$Second$ in "C:\VIewLog.txt"

. Automation Anywhere Server is a multi user solution to centralize and collaborate the entire automation system. It centralizes the setup and maintenance of automation tasks, ensuring consistency, eliminating setup time and maintenance effort whether it be a single machine or multiple machines across a division or enterprise. A centralized server console allows for better control and lower costs. Easily edit automated tasks at any workstation, then upload or download the automated task onto multiple locations to seamlessly run automated tasks across users.
You can download the free version of Automation Anywhere Server from, Automation Anywhere Server - The Leading Automation Software

. Yes, unfortunately, Windows 2008 does not support Auto login feature provided by Automation Anywhere. However, you can set auto-login if you don't want to type the password every time computer restarts.

To set auto-login ON:

1) Click Start, click Run, and type control userpasswords2.
2) Uncheck the "Users must enter a username and password to use this computer" check box. Click Apply.
3) Enter the user name and password you wish to automatically log on with, and then click OK. Click OK again and you're all done.

Hope that helps.

[TylerMitton]

Another great reply, thanks again!

If anyone else (not employed by AA) has comments feel free to post.

[ColinJD]

Tyler I have been datascraping with AA for about 5 months now
10 different sources, in very different formats.
I consider myself an experienced developer in a few other fields.
Yes, AA seems to do everything, although occasionally you must use VB-Script or other little tricks.
And often as in many IT projects there are multiple solutions.
The major bonus I have found with AA, is Mr forumsupport,
If you post a sensible problem on this msg board, you will get a timely and useful answer, with quite often a demo atmn.

[TylerMitton]

Hi Colin, thanks for your message, I really appreciate that!

I'm glad to hear that you are able to create workaround's using VB-Script, I was starting to think that would be required for some specific things... though I haven't done it yet.

I come from mostly a database background and have done a lot of web development as well.

forumsupport has been great to me so far and looking around the forum they seem to do a great job helping people out and providing good working examples.

Assuming I get the approval to use this software for my client, I will be very active on this forum and hope to help others out as I learn more.

Cheers!

[trentonponder]

A little late to this chat, but I agree that AA can do most any task if you set your mind to it. Combining multiple tasks with macros, SQL and other scripts really makes everything possible. You'll find yourself creating workarounds to workarounds that you wouldn't normally try with standard programming languages but with AA it just makes sense.

I highly recommend running tasks on virtual machines such as VMware rather then directly on a server or standalone PC. That way the task will not interfere with ordinary server/computer usage.

[TylerMitton]

Hey Trent, thanks for the message... I'm excited to hear about your idea of running things in VMware... I'll post another message about that shortly.

After using the software for almost a month I would definitely agree to the comment about workaround to workarounds... I've had to do plenty already.

Here are some other comments I have so far for others that are reading this:

- performance can be a major issue if you have to run things more often that once a minute... it takes a while to open IE, navigate to pages... more time time than if you did it through code alone... that is the price you pay for having it replicate what a human would do. I have some tasks that I have done with code in the past and can be completed in just 2 seconds... with AA it is taking 45 seconds... which is not acceptable for my use right now

- the main reason I am using this software is because it means the scrapes I create will be easier to maintain by other non-developers... if you are a software company and have lots of developers, you could build your scrapes using code and things will run faster and be more customizable... but if you are doing it for a company that is not full of developers, then AA is a good option

- I have found some other tasks that can take advantage of this software that I didn't predict before... for example, I will be using it to track all websites and online services to make sure they are up and running... alerting me if they are down... there are other services for this, but this allows me to track things that other services can't

That's it for now!

[TylerMitton]

Trent, I'd love to hear more about your VMware use... that sounds like a great idea. One major drawback (which I should have put in my earlier post) is that when you have a bunch of tasks running on a machine, it makes that machine useless... you have to dedicate it to just running tasks. You can't create or test the tasks on that machine because all the other tasks are running and doing stuff all over the place... which makes it a real pain to do anything cause it shifts focus to different windows all the time.

So VMware would allow you to get around that and still be able to do things on the machine.

Some questions if you have a sec Trent:

- is your virtual machine running windows or linux?

- do you have your vm connected to your network and on a windows domain... I don't have much experience with that... my tasks connect to other machines and databases on other machines... so they require access to the windows network

- have you used VirtualBox for this or just VMware? I've used VirtualBox in the past but not VMware

If you have anything else interesting you can share about your setup I'd love to hear.

Thanks again for taking the time to write!

Tyler