Moving thousands of files

arwes

I've got an interesting predicament. The insurance software we use stores attachments in a directory structure like "..\Attachments\Images\2010\07\21". Well, come to find out, if the thing has a hiccup it stores the files in "..\Attachments\Other\2010\07\21" instead. When the person goes to pull the file up, it says it can't be located. I can manually move the file to the correct location and the user can then view the attachment with no problems. It's incredibly random.

We just found out about this problem a week or so ago, and we've been using this system for nearly two years now. So yeah, we've got several thousand files that need to be relocated. I obviously don't have the time or the manpower to go through and manually move all this stuff. How should I go about doing this? A batch file? Would robocopy work? Our vendor says the beta release we're installing next week will fix the issue, but does nothing about the files that are currently in the wrong location.

N2IT

arwes wrote: »

I've got an interesting predicament. The insurance software we use stores attachments in a directory structure like "..\Attachments\Images\2010\07\21". Well, come to find out, if the thing has a hiccup it stores the files in "..\Attachments\Other\2010\07\21" instead. When the person goes to pull the file up, it says it can't be located. I can manually move the file to the correct location and the user can then view the attachment with no problems. It's incredibly random.

We just found out about this problem a week or so ago, and we've been using this system for nearly two years now. So yeah, we've got several thousand files that need to be relocated. I obviously don't have the time or the manpower to go through and manually move all this stuff. How should I go about doing this? A batch file? Would robocopy work? Our vendor says the beta release we're installing next week will fix the issue, but does nothing about the files that are currently in the wrong location.

I've used Robocopy and XCopy in a similiar situation.

>XCOPY "C:\Attachments\Other\2010\07"\21 "H:\\Attachments\Images\2010\07"\21 /e

It's been a while, I think this would copy all the folders subfolders and content.

I just listed C: and H: because I ran this to copy my windows profile to my network drive. It's just a suggestion.

Devilsbane

N2IT wrote: »

I've used Robocopy and XCopy in a similiar situation.

You could very easily use these in a batch file. Use some variables for the numbers and script it. Copy anything (*) in the wrong folder to the correct folder, and use variables so that you can set a loop.

tiersten

Good excuse to start learning PowerShell if you don't already know it...

N2IT

Note***


Not a system adminstrator. I deal with SAP all day long so it's been a while.

Michael.J.Palmer

I'd see if you could copy them by using xcopy in the cmd and see how that works. xcopy should copy that folder and all subfolders/files within it to the destination of choice.

arwes

Thanks for the suggestions guys. I'll test some things out this afternoon and see what happens.

it_consultant

Try MS synctoy, its incredibly easy to use with a full GUI interface.

N2IT

arwes wrote: »

Thanks for the suggestions guys. I'll test some things out this afternoon and see what happens.

Let us know.

arwes

N2IT wrote: »

Let us know.

Got it figured out. I copied over a few of the directories created over the past few days to my workstation and this worked flawlessly:

robocopy c:\TEMP\Attachments\Other /s c:\TEMP\Attachments\Images *.pdf /mov /mir

I'll do a bit more testing to make sure it's completely safe, and probably schedule it to run this evening.

Claymoore

Careful with that /mir switch. That will delete folders on the destination that do not exist on the source. Any script that has the potential to do a mass delete makes me nervous.

Having run some large robocopy jobs during SAN and disk migrations, and there are some other switches that I recommend:
Robocopy Syntax, Command Line Switches and Examples My Digital Life

/Rx for failure retries. Default is 1,000,000 - you want something more like 5

/Wx seconds to wait between retries. Default is 30, 2 is probably better. Leaving both R and W at their defaults means robocopy will retry a copy for 30 million seconds before giving up and moving to the next file.

/LOG:file file for logging. Someone always asks if everything was copied. Let them look through the log and they won't ask again.

/E copies all subdirectories including empties. This is like the /MIR but without the /PURGE deletion side effects.

robocopy source destination /move /r:5 /w:2 /log:robo.log

You may not need the R and W with the MOVE option, but I usually copy files and had have had files with long filenames mess up an entire evening's worth of work.

Of course, Step 1 is to do a backup before you start moving files around.

Devilsbane

Claymoore wrote: »

Leaving both R and W at their defaults means robocopy will retry a copy for 30 million seconds before giving up and moving to the next file.

Whose bright idea was that? That is more than 347 days of trying to move a single file... I'd give you $50 if you could give me a single instance when it would be a good idea to continue trying to move the same file for an entire year.

arwes

^^^^ I think Clay was a bit off on the seconds. The link he provided says the default is 30 seconds, not 30 million seconds. The # of failure retries was right though, 30 million lol.

Claymoore wrote: »

Careful with that /mir switch. That will delete folders on the destination that do not exist on the source. Any script that has the potential to do a mass delete makes me nervous.

Having run some large robocopy jobs during SAN and disk migrations, and there are some other switches that I recommend:
Robocopy Syntax, Command Line Switches and Examples My Digital Life

/Rx for failure retries. Default is 1,000,000 - you want something more like 5

/Wx seconds to wait between retries. Default is 30, 2 is probably better. Leaving both R and W at their defaults means robocopy will retry a copy for 30 million seconds before giving up and moving to the next file.

/LOG:file file for logging. Someone always asks if everything was copied. Let them look through the log and they won't ask again.

/E copies all subdirectories including empties. This is like the /MIR but without the /PURGE deletion side effects.

robocopy source destination /move /r:5 /w:2 /log:robo.log

You may not need the R and W with the MOVE option, but I usually copy files and had have had files with long filenames mess up an entire evening's worth of work.

Of course, Step 1 is to do a backup before you start moving files around.

Thanks for the warning. The destination & source directories SHOULD be identical. The system creates a new folder with the first attachment of the day. The only way it wouldn't be is if someone attached a file on a weekend and it only went in the "other" folder (unlikely, we're lucky to get people to work during the week). In our case, it sometimes got confused about PDF's and plopped them in the folder where it also keeps EML, MSG, XLS, PPT, TXT, WAV & HTM files. It's my understanding that if I used the /move command instead of /mov it would delete the source folder as well.

I was mainly using /mir for the mirroring of the directory structure. So I can just use /e instead of /s and it'll do the same thing, minus accidentally purging anything right? And yeah, I've got a good backup so I'm okay there.

arwes

Well that seems to have worked flawlessly! Only took about 6 minutes for a couple thousand files (figured it was more than that). Thanks for all the help!