Wednesday, January 31, 2018

FreeBSD in GCE

Yes there are supported images of amd64 freebsd available in GCE to boot, but I needed an i386 variant so here are the steps I took.

I downloaded the mini memstick image, converted it to a disk format virtualbox recognised

VBoxManage.exe convertfromraw FreeBSD-11.1-RELEASE-amd64-mini-memstick.img FreeBSD-11.1-RELEASE-amd64-mini-memstick.vdi  --format vdi

Then made a virtual machine with that as it's disk, booted and chose the shell option, then remounted as read/write, and put this into /boot/loader.conf

mount -o rw /dev/ufs/FreeBSD_install /


then shutdown the machine (gracefully!)

now I have an image that supports install via a serial port, just need to make it the right format for GCE to consider as a boot disk

VBoxManage.exe clonehd FreeBSD-11.1-RELEASE-amd64-mini-memstick.vdi disk.raw --format raw

tar -Sczf /tmp/compressed-image.tar.gz disk.raw

now put compressed-image.tar.gz in a bucket and create an image from it.

create a virtual machine with this as the primary disk and a 20GB second disk. Boot , and connect serial port , and install to the second disk.  before finishing the install make sure you update fstab on the second disk, so it can be used as a primary disk.

lastly fixup your nic MTU

ifconfig_vtnet0="SYNCDHCP mtu 1460"

Thursday, November 30, 2017

Controlling an LG soundbar

As part of my home automation configuration, I wanted to figure out how to control my soundbar.

Seems it's not documented, there isn't an API, but there is an Android App.  So I installed Genymotion, set the nic to be bridged and used wireshark to sniff what the app was doing.  I saw udp packets for each key, so copied the data, and used it like this....

Mute: 080312002a0808011204081a1002 followed by 080312002a0808011204081a1001 
Vol down: 080312002a080801120408191002 then 080312002a080801120408191001
Vol up: 080312002a080801120408181002 then 080312002a080801120408181001
Power off: 080312002a080801120408011002 then 080312002a080801120408011001 
switching inputs (optical/bluetooth/home):  080312002a0808011204084d1002 then 080312002a0808011204084d1001

So, to send this udp data to a sound bar at address 

echo "080312002a080801120408011002" | xxd -r -p > /dev/udp/ 

netcat seemed quite slow for some reason

Tuesday, November 21, 2017

caching a webcam

I Recently tried to use a webcam on the internet and it was unusable because too many people were using it.

Taking a peek, it was clear it was simple jpgs being refreshed by client side JS , using the built in webui in the camera itself, the hardware just couldn't handle too many users.

So I setup some urls and cloudfront to cache, and then an nginx proxy which would do the right thing with headers to make the images cache friendly.  Then reached out to the webmaster and talked them through what to do

Next step is to consider changing the client JS to not just increment a number in the url, which is ignored by the CDN, but maybe find some other way to cache to you get better than one frame a second.

server {
listen  8888;

access_log      /var/log/nginx/cam1.access.log;
error_log       /var/log/nginx/cam1.error.log;

location / {

    add_header Pragma public;
    add_header Cache-Control "public,max-age=1,s-max-age=1";

    proxy_set_header Host;


server {
listen  8888;

access_log      /var/log/nginx/cam2.access.log;
error_log       /var/log/nginx/cam2.error.log;

location / {

    add_header max-age 1;
    add_header s-max-age 1;
    add_header Pragma public;
    add_header Cache-Control "public";
    proxy_set_header Host;


server {
listen  8888;

access_log      /var/log/nginx/cam3.access.log;
error_log       /var/log/nginx/cam3.error.log;

location / {

    add_header max-age 1;
    add_header s-max-age 1;
    add_header Pragma public;
    add_header Cache-Control "public";
    proxy_set_header Host;

Sunday, November 12, 2017

home automation, alexa and an LG lm660 series tv, cctv and more

I thought I'd try alexa, and so it began ....

Alexa (unlike google home) talks over your LAN to philips HUE devices, so you can simply setup a fake hue bridge to interface with alexa.  this is much easier than the official lambda foo you have to do in aws.

I also had domoticz setup, tracking my solar usage, electrical usage, and Nest data. 

So, using habridge I ended up connecting habridge to domoticz, but will probably remove that integration, and just do habridge direct to my scripts.

in domoticz I setup virtual switches that ran scripts, so habridge saw these switches and presented them like hue lights to alexa, so I can say things like 'alexa turn on sky tv' , or 'alexa turn on netflix'

My two interesting problems were 1) how to turn on my tv, as it doesn't have WOL, and 2) how to handle CCTV (not really alexa related)

My tv is an LG lm660t , which doesn't have WOL, so looked at solutions to use infra red to turn it on, but it all felt pretty flaky.  I got a raspberry PI , and hooked it up to a hdmi port, and turned on simpllink in the LG, so I could now run a commandline tool on the pi, to tell the tv to wake up via the hdmi port.  This tool became a script behind a virtual switch on domoticz.

This was pretty cool, but as my tv had netflix and medialink / plex apps installed, I wanted to be able to make them run too, I extended to send those commands following the dev docs

now I could turn on the tv via the pi, then signal the tv to launch and app, or turn off the tv.

I also hooked up sky tv via this tool , so I could control it via IP 

that left me with one last job , hooking up cctv, I wanted to be able to view my annke streams which are 1080p mp4 rtsp streams, and thought about plex, but ended up just using the pi, as all it would take was switching the input, no lag or boot times etc.

I endedup using this to do it, but had to override some settings to make it work just right on my tv.  I got lines and sync problems and all kinds of weird things until I put these lines in /boot/config.txt



then it was all good

using t1n1wall, opnsense or pfsense on Google Compute Engine GCE

I have been doing some work in GCP recently, both appengine , and GCE.  We wanted to make sure all our instances were on private ips and only the LBs had internet ips.

This gave us the problem of how to allow our instances reach the internet for updates or for api calls outside.

In AWS you had a NAT gateway, but in GCP this doesn't exist. So I set about looking at the easiest way to do NAT from a private IP subnet to a Public address.  I am very familiar with m0n0wall and t1n1wall and a tiny bit with pfsense and less with opnsense.

The tl;dr is that all these distributions ship an image which within the distribution there is a disk image.  Taking this internal image, renaming it and re-compressing it is all you need to do to get it working in GCE. you can configure it via the serial port using these instructions

 There are guides out there around doing things with Linux and stuff, but you can skip that step.

I took the latest version of t1n1wall 2.11 , got the simple amd64 serial version and 

1) decompressed it
2) renamed image.bin to disk.raw
3) recompressed it to a tar.gz  (see here)
4) Copy to a google bucket
5) in GCE -> Images, create a new image from that file in a bucket
6) create an instance from that new image , configure networks, boot and configure via serial port

For pfsense, it's a bit more complex, as the image you boot is an installer image, that expects to install to a disk, so when making the instance, add a second disk, then install to it, then make an image of that disk

For opnsense, use the nano image as it's a live image, not an installer

Voila !  I prefer t1n1wall for this as it's so simple, and runs on cheaper virts better.

Monday, June 12, 2017

Debugging with Curl

For years I used --header "host:" to make curl do the right thing when testing.

Turns out there is a better way

curl -svo /dev/null --resolve

Monday, August 29, 2016

I recently faced dealing with some badbots and scrapers,  it was in a LAMP stack with varnish at the edge.  I decided to deal with it in varnish, as I always try handle as many tasks at the edge as I can, and leave apache to serve php.

So, I thought about the problem a bit, and decided to use a token bucket, nothing unusual about that. (I had to modify the source to allow passing values instead of defaulting to 1 token).  However I went a bit further and decided that different pages are 'worth' more than others, i.e. they are more sensitive.  For example, accessing the homepage vs accessing account pages.  This required a patch of the throttle mod to allow you to pass the 'cost' of a page, so more than 1 token is removed from the bucket.  For now it just logs, but I intend to send a user that is exceeding the request rate to a different backend server that will give them fake data to devalue their scraping.

you could detect user agent strings or other patterns and use as a multiplier , so bad user agent will multiply the tokens to be removed by say 5.  you could do the same with cookies too.

sample config below

vcl 4.0;
import var;
import vsthrottle;
import std;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "";
    .port = "8080";

sub vcl_recv {

        # set weights on pages using regex patterns

var.set_int("sensitivity", 1);
        if (req.url ~ "^/browse/?") {
                var.set_int("sensitivity", 10);
        } elsif (req.url ~ "^/stats/?") {
                var.set_int("sensitivity", 20);
        } elsif (req.url ~ "^/account/?") {
                var.set_int("sensitivity", 30);

        # now, lets see if they have enough credit in their token bucket to ask for this page
        # token bucket is set to 150 tokens, and is measured for 10 seconds

        if (vsthrottle.is_denied(client.identity, var.get_int("sensitivity") , 150, 10s)) {

          # Client has exceeded credit limit, lets do things like;
# set req.backend = fakedataserver;
# maybe set a http header into the get request to add to apache logs ?
std.syslog(180, "RECV: " + + req.url+ client.identity);
return (synth(429, "Too Many Requests"));


sub vcl_backend_response {

sub vcl_deliver {

FreeBSD in GCE