Monday, August 29, 2016

I recently faced dealing with some badbots and scrapers,  it was in a LAMP stack with varnish at the edge.  I decided to deal with it in varnish, as I always try handle as many tasks at the edge as I can, and leave apache to serve php.

So, I thought about the problem a bit, and decided to use a token bucket, nothing unusual about that.  However I went a bit further and decided that different pages are 'worth' more than others, i.e. they are more sensitive.  For example, accessing the homepage vs accessing account pages.  This required a patch of the throttle mod to allow you to pass the 'cost' of a page, so more than 1 token is removed from the bucket.  For now it just logs, but I intend to send a user that is exceeding the request rate to a different backend server that will give them fake data to devalue their scraping.

you could detect user agent strings or other patterns and use as a multiplier , so bad user agent will multiply the tokens to be removed by say 5.  you could do the same with cookies too.

sample config below

vcl 4.0;
import var;
import vsthrottle;
import std;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "";
    .port = "8080";

sub vcl_recv {

        # set weights on pages using regex patterns

var.set_int("sensitivity", 1);
        if (req.url ~ "^/browse/?") {
                var.set_int("sensitivity", 10);
        } elsif (req.url ~ "^/stats/?") {
                var.set_int("sensitivity", 20);
        } elsif (req.url ~ "^/account/?") {
                var.set_int("sensitivity", 30);

        # now, lets see if they have enough credit in their token bucket to ask for this page
        # token bucket is set to 150 tokens, and is measured for 10 seconds

        if (vsthrottle.is_denied(client.identity, var.get_int("sensitivity") , 150, 10s)) {

          # Client has exceeded credit limit, lets do things like;
# set req.backend = fakedataserver;
# maybe set a http header into the get request to add to apache logs ?
std.syslog(180, "RECV: " + + req.url+ client.identity);
return (synth(429, "Too Many Requests"));


sub vcl_backend_response {

sub vcl_deliver {

Tuesday, January 13, 2015

BIND 9.8 and ipv6 reverse records

The $GENERATE directive in a zone file includes nibble mode, which means you can do things like this

; reverse IPV6 zone file for
$TTL 2d    ; default TTL for zone 172800 secs
@         IN      SOA (
                        2003080800 ; sn = serial number
                        12h         ; refresh = refresh
                        15m        ; retry = update retry
                        3w         ; expiry = expiry
                        2h         ; min = minimum
; name servers Resource Recordsfor the domain
          IN      NS
; the second name servers is 
; external to this zone (domain).
          IN      NS
; PTR RR maps a IPv6 address to a host name
; hosts in subnet ID 1
$GENERATE 0-4294967295  ${0,15,n}. PTR 2001-0db8-0000-0001-0000-0000-${0,15,n}

After adding the zone of course !

zone "" {
        type master;
        file "/etc/bind/ipv6.rev.local";

Tuesday, March 18, 2014

Getting Google IP ranges

Google puts them into SPF records ?!

dig +short txt | sed 's/"//g; s/ip4://g;' | tr ' ' '\n' | grep include | cut -d ':' -f2 | xargs dig +short txt  | sed 's/"//g; s/ip4://g;' | tr ' ' '\n' | grep '/'

responds with

Wednesday, January 8, 2014

IPFilter 5 and compiled access lists

I've been back to Freebsd lately, and using freebsd10 rc2.

I noticed an option to compile access lists into the kernel module for ipfilter.  I got it to work, but thought I'd note down how ...

firstly the kernel doesn't know the option to compile access lists so add this instead


or you could add to /etc/make.conf

now, take your ruleset in a file and create ip_rules.c and ip_rules.h  by doing this

/sbin/ipf -n -cc -f <filename>

copy them into 


edit ip_rules.c and take out the if statement for NetBSD by deleting

#if (__NetBSD_Version__ >= 399000000)

then, if you have in and out rules, there is a bug in generating the ip_rules.c , work around it by adding

frentry_t *ipf_rules_out_[1] = {
        (frentry_t *)&out_rule__0

and adjust [1] to the amount of rules you have.

then compile the kernel as normal and install it, load the module and you should see something in dmesg like

IP Filter: v5.1.2 initialized.  Default = pass all, Logging = enabled (COMPILED)

Wednesday, May 29, 2013

Centos in a proxy enviroment

I use centos in work, behind a proxy that uses NTLM auth.

some tips;

  1. install CNTLM, which access a proxy to NTLM proxies :-) configure it so it listens for localhost connections
    • /etc/cntlm.conf
      • Username        username
      • Domain          domain
      • Password        password
      • Proxy           upstream.proxy.addr:8080
      • NoProxy         localhost, 172.18.32.*, 127.0.0.*, 10.*, 192.168.*
      • Listen          3128
      • Gateway yes
      • Allow 
      • Deny            0/0
  2. edit yum.conf
    • echo "proxy=" >> /etc/yum.conf
  3. edit maven settings.xml
    • /usr/local/apache-maven-3.0.5/conf/settings.xml
                    1. setup git
                      • git config --global http.proxy
                    2. setup your shell (wget etc)
                      • http_proxy=""
                      • export http_proxy

                    Tuesday, May 28, 2013

                    Cloudstack 4.0.2 with vsphere integration and netscaler integration

                    Based on a clean install of CentOS-6.4-x86_64-minimal.iso , this step builds the RPM's and shares them via apache as a repo to install.
                    /etc/init.d/iptables stop
                    yum groupinstall "Development Tools"
                    yum install unzip createrepo ws-commons-util wget java-1.6.0-openjdk-devel.x86_64 ant ant-jdepend genisoimage mysql mysql-server ws-common-utils MySQL-python tomcat6 httpd.x86_64
                    tar -zxvf apache-maven-3.0.5-bin.tar.gz
                    mv apache-maven-3.0.5 /usr/local/
                    export PATH=/usr/local/apache-maven-3.0.5/bin:$PATH
                    bunzip2 apache-cloudstack-4.0.2-src.tar.bz2
                    tar -xvf apache-cloudstack-4.0.2-src.tar
                    cd apache-cloudstack-4.0.2-src/deps
                    #now we are ready, lets build
                    cd ../vmware-base/
                    mvn install 
                    cd ..
                    mvn -D nonoss -P deps
                    cp cloud.spec
                    #Build RPM's and share as repo
                    ./waf rpm
                    cd artifacts/rpmbuild/RPMS/x86_64
                    createrepo ./
                    mkdir /var/www/html/cloudstack
                    cp -R * /var/www/html/cloudstack/
                    apachectl start

                    Now the RPM's are built and shared, time to setup the actual server

                    vi /etc/yum.repos.d/cloudstack.repo
                    # put in the repo info as
                    yum update
                    yum install cloud-server cloud-client mysql-server
                    #edit /etc/my.cnf as per docs
                    setenforce permissive
                    cloud-setup-databases cloud:secretpassword@localhost --deploy-as=root:password

                    and open in your browser http://ip:8080/client

                    installing cloudstack with ubuntu server

                    after a fresh install of  ubuntu 12.04


                    works out like this

                    1. echo "deb precise 4.0" > /etc/apt/sources.list.d/cloudstack.list
                    2. wget -O -|apt-key add -
                    3. apt-get update
                    4. apt-get install cloud-client-ui
                    5. apt-get install mysql-server nfs-kernel-server
                    6. cloud-setup-databases cloud:secret --deploy-as=root:password
                    7. cloud-setup-management

                    now go to  http://server:8080/client/ and login as admin/password

                    you should read the manual and do nfs etc like this
                    1. mkdir -p /export/primary
                    2. mkdir -p /export/secondary
                    3. echo "/export *(rw,async,no_root_squash)" >> /etc/exports
                    4. exportfs -a
                    5. put this into /etc/default/nfs-kernel-server
                    1.  /etc/init.d/nfs-kernel-server restart
                    2. mkdir -p /mnt/primary /mnt/secondary
                    3. mount -v -t nfs ubuntu:/export/secondary /mnt/secondary
                    4. mount -v -t nfs ubuntu:/export/primary /mnt/primary
                    5. /usr/lib/cloud/common/scripts/storage/secondary/cloud-install-sys-tmplt  -m /mnt/secondary -u -h kvm -F
                        you can run kvm on the box too if you want

                        sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
                        optionally sudo modprobe kvm
                        optionally sudo adduser `id -un` libvirtd