Ramblings From The Litter Tray of Life

Data Replication Script

Posted by graycat on 20 August 2008

If, like me, you’re concerned about your data and like to have it multiple places then I’m sure you’ll have looked into methods of copying, replicating and synchronising this data about the shop. Now there are many methods to get this data a’movin and as many different cost levels as you can think of.

At my place of work we use an application by Veritas called Replication Exec and whilst it does the job, it does have some limitations IMO….. one of which I ran into today.

[fade out]

Admiral: Ok, son. We’ve got a major problem and we’re fresh out of ideas! WE need to replicate the critical data from the Eastern domain to the Central domain and we need it to work now!!

Capt Graycat: No problem, sir. Nothing I can’t accomplish with enough firepower.

Admiral: I admire your guts, Captain, but all our normal methods have failed. Replication Exec is useless and dead in the water. Surely you don’t think you can beat a powerhouse like that at it’s own game?!

Capt Graycat: Let me at it, sir! And don’t call me Shirley.

Capt Graycat ripples of a heroic salute and steps out into the night to take on the challenge. Over his shoulder he calls “smoke me a kipper, I’ll be back for breakfast” and disappears into the glum

Admiral’s sexy PA: What a guy! *swoon*

[fade back into real life]

Get all of that? Reread it if needed as there might be questions later 🙂

So what to do? Primary objective, sorry I meant the main thing is to get the data copied and that’ll mean a differential copy. Problems we’re going to run into are two separate domains that don’t trust each other properly (don’t ask), a lot of data to copy and get some value added features out of it because we’re that good.

My plan is to get the script to go like this:

  1. Make a working directory for the logs
  2. replicate the data using robocopy and various settings and log it into the “working” directory
  3. move the log to a dated folder and rename it to today’s date
  4. email it to various people
  5. clean up afterwards

Easy, huh?

ok, I’ll not discuss the making of folders or moving files about bit as that’s been done to death elsewhere.

Oh sod it, here’s the code –


REM   DR_Data_Replication.bat
REM
REM   Mirrors Server1 share contents from SOURCE_SERVER to TARGET_SERVER.
REM   Creates monthly directory and daily log then e-mails to address(es)
REM
REM   Written by - Graycat
REM

REM Creating Log Folder
md c:\#Scripts\Logs\Working\

REM Starting Replication …..

REM Replicating “Server1” into “Data_Share” on TARGET_SERVER
Robocopy e:\Server1 \\TARGET_SERVER\Data_Share /MIR /XN /XF AUTORUN.INF /LOG+:c:\#Scripts\Logs\Working\Working.txt /R:3 /rh:2200-0400 /pf

REM Move and rename log
move c:\#Scripts\Logs\Working\Working.txt c:\#Scripts\Logs\%date:~-4,4%\%date:~-7,2%\%date:~-4,4%%date:~-7,2%%date:~0,2%.txt

REM E-mailing Log file
echo DR data replication from SOURCE_SERVER to TARGET_SERVER logs attached. > body.txt
c:\#scripts\mpack.exe -s “DR Replication” -d body.txt -c application/exe -o body.msg c:\#Scripts\Logs\%date:~-4,4%\%date:~-7,2%\%date:~-4,4%%date:~-7,2%%date:~0,2%.txt
c:\#Scripts\bmail.exe -s 192.1.1.245 -f Robocopy@mydomain.com -t me@mydomain.com -h -m body.msg

REM Clean up
del c:\#Scripts\body.msg
del c:\#Scripts\body.txt

Exit

hopefully most of that follows on from what I was after in the script flow. Robocopy is in the Windows resource kit I believe but easily downloaded from MS. The mpack is for encoding MIME e-mail messages so that they can be sent with an attachment. It’s a cracking little free application and can be found over here along with bmail which is what was used to actually send the e-mail.

Ok, the robocopy switches might be of interest so here goes –

  • MIR – mirror directory
  • /XN – exclude newer files
  • /XF AUTORUN.INF – exclude files called autorun.inf
  • /LOG+:c:\#Scripts\Logs\Working\Working.txt – log to the end of working.txt log file
  • /R:3 – retry three times if it fails
  • /rh:2200-0400 – allow to run between 2200 and 0400
  • /pf – check the allowed run time for each file as it’s done. this’ll stop it running after 0400

on test runs this worked great …. apart from robocopy would pause rather than terminate when it hit the end of its allowed run time.
To get round this, I added another script to run at 0400 that used Taskkill /IM robocopy /f to kill any process called robocopy. This would then allow the rest of the script to run through nicely.

Right, that’s me for today. Time to go home. I’ll report back in a few days as to how it went.

Advertisements

2 Responses to “Data Replication Script”

  1. Adam Vero said

    Out of interest, why are you using /XN? Are you only making a copy of files from Source if they are newer than Target (and excluding them if Target is newer)? Does this not leave two versions of a file which are now different? How do you reconcile them back again?

    What happens when the mirroring does not complete by 04:00?

    You might also consider using the switch /A-A to turn off the archive attribute on copied files – this would make it easier to identify any which have been modified on the target server since the last mirror (if you were backing up this second server for example)

  2. graycat said

    Hey, Adam
    I’m using the /XN switch in case they modify a file over here during the first few weeks. Once everything has settled, that’ll be removed.

    when it gets to 0400 the robocopy will pause after finishing off the current file. Because it pauses rather than ends, I’ve used a separate script to kill any robocopy instance at 0405 so the rest of the script will kick off.

    Cheers for the tip on the /A switch! I’ll have a look into incorporating it in the base.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: