Monday, June 12, 2017

Debugging with Curl

For years I used --header "host: foo.bar" to make curl do the right thing when testing.

Turns out there is a better way

curl -svo /dev/null https://www.foo.bar --resolve www.foo.bar:443:127.0.0.1

Monday, August 29, 2016

I recently faced dealing with some badbots and scrapers,  it was in a LAMP stack with varnish at the edge.  I decided to deal with it in varnish, as I always try handle as many tasks at the edge as I can, and leave apache to serve php.

So, I thought about the problem a bit, and decided to use a token bucket, nothing unusual about that.  However I went a bit further and decided that different pages are 'worth' more than others, i.e. they are more sensitive.  For example, accessing the homepage vs accessing account pages.  This required a patch of the throttle mod to allow you to pass the 'cost' of a page, so more than 1 token is removed from the bucket.  For now it just logs, but I intend to send a user that is exceeding the request rate to a different backend server that will give them fake data to devalue their scraping.

you could detect user agent strings or other patterns and use as a multiplier , so bad user agent will multiply the tokens to be removed by say 5.  you could do the same with cookies too.

sample config below

vcl 4.0;
import var;
import vsthrottle;
import std;


# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {

        # set weights on pages using regex patterns

var.set_int("sensitivity", 1);
       
        if (req.url ~ "^/browse/?") {
                var.set_int("sensitivity", 10);
        } elsif (req.url ~ "^/stats/?") {
                var.set_int("sensitivity", 20);
        } elsif (req.url ~ "^/account/?") {
                var.set_int("sensitivity", 30);
        }

        # now, lets see if they have enough credit in their token bucket to ask for this page
        # token bucket is set to 150 tokens, and is measured for 10 seconds

        if (vsthrottle.is_denied(client.identity, var.get_int("sensitivity") , 150, 10s)) {

          # Client has exceeded credit limit, lets do things like;
# set req.backend = fakedataserver;
# maybe set a http header into the get request to add to apache logs ?
std.syslog(180, "RECV: " + req.http.host + req.url+ client.identity);
return (synth(429, "Too Many Requests"));
        }

}

sub vcl_backend_response {
}

sub vcl_deliver {
}

Tuesday, January 13, 2015

BIND 9.8 and ipv6 reverse records

The $GENERATE directive in a zone file includes nibble mode, which means you can do things like this



; reverse IPV6 zone file for example.com
$TTL 2d    ; default TTL for zone 172800 secs
$ORIGIN 0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa.
@         IN      SOA   ns1.example.com. hostmaster.example.com. (
                        2003080800 ; sn = serial number
                        12h         ; refresh = refresh
                        15m        ; retry = update retry
                        3w         ; expiry = expiry
                        2h         ; min = minimum
                        )
; name servers Resource Recordsfor the domain
          IN      NS      ns1.example.com.
; the second name servers is 
; external to this zone (domain).
          IN      NS      ns2.example.net.
; PTR RR maps a IPv6 address to a host name
; hosts in subnet ID 1
$GENERATE 0-4294967295  ${0,15,n}.0.0.0.0.1.0.0.0 PTR 2001-0db8-0000-0001-0000-0000-${0,15,n}.example.com.


After adding the zone of course !


zone "0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa" {
        type master;
        file "/etc/bind/ipv6.rev.local";
};

Tuesday, March 18, 2014

Getting Google IP ranges

Google puts them into SPF records ?!

dig +short txt _cloud-netblocks.googleusercontent.com | sed 's/"//g; s/ip4://g;' | tr ' ' '\n' | grep include | cut -d ':' -f2 | xargs dig +short txt  | sed 's/"//g; s/ip4://g;' | tr ' ' '\n' | grep '/'


responds with

8.34.208.0/20
8.35.192.0/21
8.35.200.0/23
108.59.80.0/20
108.170.192.0/20
108.170.208.0/21
108.170.216.0/22
108.170.220.0/23
108.170.222.0/24
162.216.148.0/22
162.222.176.0/21
173.255.112.0/20
192.158.28.0/22
199.192.112.0/22
199.223.232.0/22
199.223.236.0/23
23.236.48.0/20
23.251.128.0/19
107.167.160.0/19
107.178.192.0/18

Wednesday, January 8, 2014

IPFilter 5 and compiled access lists

I've been back to Freebsd lately, and using freebsd10 rc2.

I noticed an option to compile access lists into the kernel module for ipfilter.  I got it to work, but thought I'd note down how ...

firstly the kernel doesn't know the option to compile access lists so add this instead

makeoptions     CFLAGS+=-DIPFILTER_COMPILED

or you could add to /etc/make.conf

now, take your ruleset in a file and create ip_rules.c and ip_rules.h  by doing this

/sbin/ipf -n -cc -f <filename>

copy them into 

/usr/src/sys/contrib/ipfilter/netinet

edit ip_rules.c and take out the if statement for NetBSD by deleting

#if (__NetBSD_Version__ >= 399000000)
#else

then, if you have in and out rules, there is a bug in generating the ip_rules.c , work around it by adding

frentry_t *ipf_rules_out_[1] = {
        (frentry_t *)&out_rule__0
};

and adjust [1] to the amount of rules you have.

then compile the kernel as normal and install it, load the module and you should see something in dmesg like

IP Filter: v5.1.2 initialized.  Default = pass all, Logging = enabled (COMPILED)


Wednesday, May 29, 2013

Centos in a proxy enviroment

I use centos in work, behind a proxy that uses NTLM auth.

some tips;


  1. install CNTLM, which access a proxy to NTLM proxies :-) configure it so it listens for localhost connections
    • /etc/cntlm.conf
      • Username        username
      • Domain          domain
      • Password        password
      • Proxy           upstream.proxy.addr:8080
      • NoProxy         localhost, 172.18.32.*, 127.0.0.*, 10.*, 192.168.*
      • Listen          3128
      • Gateway yes
      • Allow           127.0.0.1
      • Deny            0/0
  2. edit yum.conf
    • echo "proxy=http://127.0.0.1:3128" >> /etc/yum.conf
  3. edit maven settings.xml
    • /usr/local/apache-maven-3.0.5/conf/settings.xml
  <proxies>
    <proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>127.0.0.1</host>
      <port>3128</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
  </proxies>
                    1. setup git
                      • git config --global http.proxy http://127.0.0.1:3128
                    2. setup your shell (wget etc)
                      • http_proxy="http://127.0.0.1:3128"
                      • export http_proxy


                    Tuesday, May 28, 2013

                    Cloudstack 4.0.2 with vsphere integration and netscaler integration



                    Based on a clean install of CentOS-6.4-x86_64-minimal.iso , this step builds the RPM's and shares them via apache as a repo to install.
                    ---
                    /etc/init.d/iptables stop
                    yum groupinstall "Development Tools"
                    yum install unzip createrepo ws-commons-util wget java-1.6.0-openjdk-devel.x86_64 ant ant-jdepend genisoimage mysql mysql-server ws-common-utils MySQL-python tomcat6 httpd.x86_64
                    wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
                    tar -zxvf apache-maven-3.0.5-bin.tar.gz
                    mv apache-maven-3.0.5 /usr/local/
                    export PATH=/usr/local/apache-maven-3.0.5/bin:$PATH
                    wget http://www.us.apache.org/dist/cloudstack/4.0.2/apache-cloudstack-4.0.2-src.tar.bz2
                    bunzip2 apache-cloudstack-4.0.2-src.tar.bz2
                    tar -xvf apache-cloudstack-4.0.2-src.tar
                    cd apache-cloudstack-4.0.2-src/deps
                    wget http://zooi.widodh.nl/cloudstack/build-dep/cloud-iControl.jar
                    wget http://zooi.widodh.nl/cloudstack/build-dep/cloud-manageontap.jar
                    wget http://zooi.widodh.nl/cloudstack/build-dep/vmware-vim.jar
                    wget http://zooi.widodh.nl/cloudstack/build-dep/vmware-vim25.jar
                    wget http://zooi.widodh.nl/cloudstack/build-dep/vmware-apputils.jar
                    wget http://community.citrix.com/download/attachments/37847122/cloud-netscaler-jars.zip
                    unzip cloud-netscaler-jars.zip
                    #now we are ready, lets build
                    ./install-non-oss.sh
                    cd ../vmware-base/
                    mvn install 
                    cd ..
                    mvn -D nonoss -P deps
                    wget http://people.apache.org/~jzb/cloudstack/dist/releases/4.0.2/nonoss.cloud.spec
                    cp nonoss.cloud.spec cloud.spec
                    #Build RPM's and share as repo
                    ./waf rpm
                    cd artifacts/rpmbuild/RPMS/x86_64
                    createrepo ./
                    mkdir /var/www/html/cloudstack
                    cp -R * /var/www/html/cloudstack/
                    apachectl start
                    ---

                    Now the RPM's are built and shared, time to setup the actual server

                    vi /etc/yum.repos.d/cloudstack.repo
                    # put in the repo info as http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.2/html/Installation_Guide/configure-package-repository.html#configure-package-repository-rpm
                    yum update
                    yum install cloud-server cloud-client mysql-server
                    #edit /etc/my.cnf as per docs
                    mysql_secure_installation
                    setenforce permissive
                    cloud-setup-databases cloud:secretpassword@localhost --deploy-as=root:password
                    cloud-setup-management


                    and open in your browser http://ip:8080/client