rsync with CDN and purge

I recently had to setup a new CDN for, using Edgecast,

They support rsync for content and also have an API, so I started with cron'd full rsync, but this was taking too long, so I moved to rsync only recently changed files, and used the output to do a flush of the cache for updated files. Edgecast provide a really good CDN solution, and also provide EU based rsync endpoints for getting your logs and pushing your content




#!/bin/sh
set -e
rm -rf /tmp/rsync.ec.out
rm -rf /tmp/changedfiles

#move to directory so path names are good in outputs
cd /opt/www/htdocs/

#find recently modified files
find . -mmin -120 > /tmp/changedfiles

#run rsync for these changed files, only changing existing files and logging output
/usr/bin/rsync --itemize-changes -az --existing -e "ssh -i /home/rsync/.ssh/my_special_id_rsa -p8022 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no" --files-from=/tmp/changedfiles /opt/www/htdocs/ user@domain@rsync.ams.edgecastcdn.net:/content/mysubdir/ > /tmp/rsync.ec.out

#loop through output so we can purge changed files
for filetopurge in `grep "^<f" /tmp/rsync.ec.out | cut -d " " -f2`; do
if [ -f $filetopurge ]
then
echo '<MediaContentPurge xmlns="
http://www.whitecdn.com/schemas/apiservices/"><MediaPath>http://wpc.use-cust-number-here.edgecastcdn.net/use-cust-number-here/mysubdir/'$filetopurge'</MediaPath><MediaType>3</MediaType></MediaContentPurge>' | curl --insecure -X PUT -H "Authorization: TOK:use-your-token-here" -H 'Content-type: text/xml' -d @- https://api.edgecast.com/v2/mcc/customers/use-cust-number-here/edge/purge
echo -e "\nPurging ${filetopurge}"
fi
done

#now run rsync for remainder of files
/usr/bin/rsync -azv --ignore-existing -e "ssh -i /home/rsync/.ssh/my_special_id_rsa -p8022 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no" --files-from=/tmp/changedfiles /opt/www/htdocs/ user@domain@rsync.ams.edgecastcdn.net:/content/mysubdir/

Comments

Popular posts from this blog

Baileys liquor Chocolate Chip and Cream desert

nginx decode base64 url for use with imgproxy

using t1n1wall, opnsense or pfsense on Google Compute Engine GCE