Tuesday, March 20, 2012

rsync with CDN and purge

I recently had to setup a new CDN for, using Edgecast,

They support rsync for content and also have an API, so I started with cron'd full rsync, but this was taking too long, so I moved to rsync only recently changed files, and used the output to do a flush of the cache for updated files. Edgecast provide a really good CDN solution, and also provide EU based rsync endpoints for getting your logs and pushing your content

set -e
rm -rf /tmp/rsync.ec.out
rm -rf /tmp/changedfiles

#move to directory so path names are good in outputs
cd /opt/www/htdocs/

#find recently modified files
find . -mmin -120 > /tmp/changedfiles

#run rsync for these changed files, only changing existing files and logging output
/usr/bin/rsync --itemize-changes -az --existing -e "ssh -i /home/rsync/.ssh/my_special_id_rsa -p8022 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no" --files-from=/tmp/changedfiles /opt/www/htdocs/ user@domain@rsync.ams.edgecastcdn.net:/content/mysubdir/ > /tmp/rsync.ec.out

#loop through output so we can purge changed files
for filetopurge in `grep "^<f" /tmp/rsync.ec.out | cut -d " " -f2`; do
if [ -f $filetopurge ]
echo '<MediaContentPurge xmlns="
http://www.whitecdn.com/schemas/apiservices/"><MediaPath>http://wpc.use-cust-number-here.edgecastcdn.net/use-cust-number-here/mysubdir/'$filetopurge'</MediaPath><MediaType>3</MediaType></MediaContentPurge>' | curl --insecure -X PUT -H "Authorization: TOK:use-your-token-here" -H 'Content-type: text/xml' -d @- https://api.edgecast.com/v2/mcc/customers/use-cust-number-here/edge/purge
echo -e "\nPurging ${filetopurge}"

#now run rsync for remainder of files
/usr/bin/rsync -azv --ignore-existing -e "ssh -i /home/rsync/.ssh/my_special_id_rsa -p8022 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no" --files-from=/tmp/changedfiles /opt/www/htdocs/ user@domain@rsync.ams.edgecastcdn.net:/content/mysubdir/