A Full Data Scraping Solution?

[trentonponder]

I basically have a few dedicated servers running multiple instances of VMware Server with numerous Virtual Machines dedicated to AA tasks. The Virtual Machines are part of their own Windows Workgroup thats on the same local network as the databases they need to interact with.

There's no need to have them actually join the domain (unless there is a permission issue with your server). The individual VMs run Windows XP as AA won't work well with anything else. I have the VMs segregated by task-type and schedule. There are some VMs that only run one task because it could be triggered at any moment, while others can have dozens all set to go on a specific schedule.

Depending on your needs I would recommend at least one dedicated AA machine to host the VMs with a substantial amount of RAM. The more RAM you have, the more VMs you can create.

As for performance, that can be an issue but with many things traditional code is not a viable option.

Running the same task every minute can be excessive. Instead, try creating one task with a loop that does the same thing repeatedly. This could be based on a SQL query or some other condition. That way you're not opening and closing AA repeatedly.

[TylerMitton]

Very cool, I'm going to consider doing that Trenton (sorry, I'm guessing it's Trenton, not Trent)... I like the idea of using VMs.

Also, the method of looping within a task instead of setting a task to run over and over is a great idea... a good portion of the "slowness" I'm experiencing is just starting up AA... and also starting up IE (for my scrapes). I'm going to look at leaving IE open as well and then just refreshing the page to eliminate the IE startup time as well. It should be a lot easier on the machine to just leave IE open rather opening it and closing it every 30 seconds!