Interesting Problem

the_Grinch

So we have a customer who switched Antiviruses to our managed solution (Panda Endpoint Protection, don't get me started). Everything went smoothly, but suddenly their backup started going slow. When I say slow, I mean really slow. It starts at 6 PM and by 8 AM it is still running. We did varies troubleshooting and determined it was the antivirus. So we made a script to stop the antivirus, start the backup, restart antivirus after backup completes (or fails). This worked for a period of time and then we updated the antivirus. Now besides not stopping properly, we are still seeing the slow speeds. The kicker (it's crazy) is it is only the local job that is being affected. They have two jobs and two tape drives on this server: 1 backs up the local D drive and 1 backs up two servers. The job that backs up the remote servers has not been slow at all and completes in a normal time. Also, the remote servers have the same antivirus installed. Any ideas?

I should warn you that this customer is using Veritas 10D, so no support from Symantec....

RobertKaucher

Can you exclude the backup program and it's backup files from the AV's scan? Server apps like backup and databases (SQL, etc) need to be excluded.

Everyone

RobertKaucher wrote: »

Can you exclude the backup program and it's backup files from the AV's scan? Server apps like backup and databases (SQL, etc) need to be excluded.

I don't think that helps if the AV is doing an "on-access" type of scan. The performance hit comes when the backup software accesses a file, the AV thinks it needs to scan it, so the backup process and the AV process sort of fight each other. It isn't the backup process that is being scanned, it's the file it is currently trying to backup that is.

Tackle

Are you stopping ALL of the services? We use Total Defense and etrust, there are half a dozen services for both of them.

the_Grinch

We did exclude the directories, but that did not help. At first (as LucasMN stated) we had only stopped one service, which initially appeared to work. But after the update that stopped working, so we then ended all the services associated with the Endpoint. We then noted that it was somehow starting again and found out our remote agent was restarting the services due to a scan job being setup in the monitoring console. So we disabled the monitoring software, disabled the antivirus, and stopped all the related processes in task manager, but yet the issue continues. I did note last night that we have another customer with the same setup (Panda and Symantec 10D) and they haven't had any issues. Perhaps we are barking up the wrong tree with the antivirus, but keep the ideas/suggestions coming and thanks for the help thus far!!!

Tackle

For troubleshooting purposes it may be worth it to uninstall the AV for a day and see if the backup runs fine. As you've said, it could be something else...uninstalling may eliminate the AV as a possibility all together.

If the backup runs fine, it might be time to give Panda a call. They might have some suggestions.

RobertKaucher

Everyone wrote: »

I don't think that helps if the AV is doing an "on-access" type of scan. The performance hit comes when the backup software accesses a file, the AV thinks it needs to scan it, so the backup process and the AV process sort of fight each other. It isn't the backup process that is being scanned, it's the file it is currently trying to backup that is.

My point was in excluding the backup files (not sure if it is going to tape or what) - i.e. excluding the directories the app is writing to.

I understand the app is not being scanned. But the app's directories should be excluded.

it_consultant

the_Grinch wrote: »

We did exclude the directories, but that did not help. At first (as LucasMN stated) we had only stopped one service, which initially appeared to work. But after the update that stopped working, so we then ended all the services associated with the Endpoint. We then noted that it was somehow starting again and found out our remote agent was restarting the services due to a scan job being setup in the monitoring console. So we disabled the monitoring software, disabled the antivirus, and stopped all the related processes in task manager, but yet the issue continues. I did note last night that we have another customer with the same setup (Panda and Symantec 10D) and they haven't had any issues. Perhaps we are barking up the wrong tree with the antivirus, but keep the ideas/suggestions coming and thanks for the help thus far!!!

Make up a bullcrap excuse and get rid of Veritas 10D. Its just wrong.

terryfera

Had a similar issue with McAfee SaaS AV and BackupExec. After chasing it for weeks and having random days where it would be fast and thinking I found the solution with shutting down the AV it ended up being the media being written to *facepalm*.

Are you seeing the AV processes eating up more CPU cycles than usual while the backup is running?

the_Grinch

Have to look if the AV process is eating up a lot of cycles. The kicker is the customer is having a network and server refresh (with Backup Exec 2010) next week so this may be over either way. Fact is at this point I want to know what is causing the issue, would feel defeated if we don't find out! Thanks again for the suggestions, I'll be looking into it tonight...

RobertKaucher

the_Grinch wrote: »

Have to look if the AV process is eating up a lot of cycles. The kicker is the customer is having a network and server refresh (with Backup Exec 2010) next week so this may be over either way. Fact is at this point I want to know what is causing the issue, would feel defeated if we don't find out! Thanks again for the suggestions, I'll be looking into it tonight...

I'd not worry about it unless the issue continued after the upgrade. I understand your quest for knowledge, but you have to be practical about it. It will save you time and gray hairs.

the_Grinch

Only advantage to being mostly bald at a young age, fewer hairs to get grey! But you're right, can't solve every problem and the network upgrade will probably solve the issue. Thanks again everyone!

Devilsbane

Is it the antivirus actively scanning that causes the issues or just that it happens to be running?

the_Grinch

It just happens to be running, I don't believe any active scans are going on. Now I did some further research and was looking at the time line of events, things just aren't adding up. Here's how it goes (keep in mind the customer is saying that since we installed Endpoint, everything has been slow network, computers, and servers):

8/25 - Endpoint deployment is finished (62 nodes get it installed)
10/04 - First slowness issue is reported (for an old PC in a conference room)
10/05 - Second report of slowness, customer reports everyone has complained about slowness for WEEKS
10/13 - The backup starts on the 12th, takes 10 hours to complete (instead of 6)

Now, from that time line I see a number of issues. First, the customer complains everything was slow since we installed, but the first report that we get doesn't come until the 4th of October. Are we to believe that for almost two months everything has been slow, your users have complained, and you never contacted us? Second, the backup didn't start going slow until the 13th. Again almost two months from the install date and as best I can surmise (we don't have a ticket for some reason) we upgraded the Endpoint on or about the 5th. So for a week the backup was not affected by Endpoint. Third, there are two jobs that run on this server that backup to two different tape drives. The second job has not had any increase in time to complete during any of this.

I haven't been in this business a long time, but I know end users rarely don't complain about something that bothers them. I find it very hard to believe that for two months everyone in two offices had slowness issues and it was never brought to our attention. This customer hasn't upgraded their infrastructure in 10 years (yeah, 10 years) and have only a few new PCs. Even the customer said, "I know our PC's are old, but even the new ones are slow". Now anything could be slowing those PC's down and while I don't love our Endpoint, no other customers have had issues (that we hadn't fixed and then cleared for everyone else). At this point I think it was just a fluke that the job sped up when we stopped the antivirus and that while it might have contributed to the issue, there is a lot more wrong with the server, tape drive, or drive it is backing up. Thanks for all the help everyone!

Everyone

RobertKaucher wrote: »

My point was in excluding the backup files (not sure if it is going to tape or what) - i.e. excluding the directories the app is writing to.

I understand the app is not being scanned. But the app's directories should be excluded.

Excluding the backup destination won't help issues with an "on-access" scan. The files on the destination wouldn't be scanned until after they are written anyway, so that shouldn't be a problem. It is the file being read that is the problem. Say you're trying to backup C:\Windows to T:\Backups, every file the backup software touches in C:\Windows directory will trigger an "on-access" scan (if enabled) for most AV software. The AV sees the file being accessed and says "Hey wait, let me scan this first". If you had T:\Backups excluded, it shouldn't scan anything going into that location, but it will still scan everything from the source as it is read.