Jason Punyon

Get Your Redis On on Windows

TL;DR: Want a virtual machine running redis in however long it takes you to download 400MB + a little completely automated install time? Follow the instructions here and you’ll be on your way.

Well, it only took me a year of having this blog for me to write up something even remotely technical. But here you are, and here I am…so let’s just tone down the celebration a little bit and get on with it already.

So…it’s hard running a dev environment sometimes. We at the Stack Exchange will use anything to get the job done, but on the programmer’s side we’re mainly a windows shop. One piece of software we’ve come to know and love is Redis though. We love it so much we’ve got antirez on speed dial. It’s really the greatest.

Here’s where it isn’t quite the greatest though (for us): it’s really meant to run on Linux. Some people have made mostly working windows builds in the past that were good enough for dev’ing on but had weird behavior when it came to background operations. They’re great and I appreciate the work they d(o|id), but they fall behind when redis bumps stable versions (it’s behind 1 stable version right now leaving out features like the Lua scripting engine). Microsoft went through the rigamarole of patching redis so that it will run on windows, but that patch isn’t getting merged to master…ever.

So what’s a girl to do? When I’ve been on a team of one and had this kind of problem I thought to myself, “Self! Get VMWare on here, spin up a one off VM with ubuntu and just run it there! Problem Solved!” and many internal high-fives were had. But when you’re on a team of 6 (the Careers team, plug: we’re hiring) that doesn’t really scale well. So what are my choices? Let’s go to the big board of options:

  • Just tell my teammates “Hey, spend a couple hours spinning up your own VM and hope the one you have and the one I have match up and behave exactly the same”. (HINT: No)
  • Check in a 10 gig VM into source control and push so the other members of the team can run it too? (HINT: NO. That’s an example of what we call the “I quit” check-in.)

So how do you solve this problem?

Enter the Hobo

So it turns out a bunch of other people have this problem too (WEIRD RITE?). A smart dude decided to solve it and created Vagrant. Vagrant is a super simple yet powerful way to create and manage a reproducible virtual dev environment. Check in a couple kB of config and you get a virtual machine (or a multiple machine environment) your whole team can run. Vagrant wraps around Virtual Box for it’s virtual machines and it’s not just for windows. It runs on Linux and Mac too. Let’s run it down.

Getting in Installed

Follow the startup guide here. It’s basically install VirtualBox and install Vagrant.

Creating a machine

To create the machine, the first thing we do is create your Vagrantfile. Don’t worry…it isn’t a driter fetish. It’s just a config file that outlines how your virtual machine is setup. It’s also just a bit of ruby. Here’s the one we’re using:

No Drifter Fetishes Here (Vagrantfile) download
1
2
3
4
5
6
7
8
9
10
11
Vagrant::Config.run do |config|
  config.vm.box = "ubuntu-12.04-amd64"

  # THE URL FROM WHERE THE 'CONFIG.VM.BOX' BOX WILL BE FETCHED IF IT
  # DOESN'T ALREADY EXIST ON THE USER'S SYSTEM.
  config.vm.box_url = "http://dl.dropbox.com/u/4031118/Vagrant/ubuntu-12.04.1-server-i686-virtual.box"
  config.vm.forward_port 6379, 6379
  config.vm.customize ["modifyvm", :id, "--memory", 1024]

  config.vm.provision :shell, :path => "init.sh"
end

So first we tell Vagrant which box to use. A box is essentially a map from a key to a file. Box names can be anything you want, in this case I just have a name telling me that it’s ubuntu’s latest 64-bit release.

Next we have a url that points to a file. As it says in the comment there, this url points to a box file that will be downloaded if the box with the name in config.vm.box doesn’t exist. This is nice because it means i send the file and when my teammate runs it it will go fetch everything it needs to create the virtual machine. Brilliant. A bunch of base boxes can be found at Vagrantbox.es. They have many different guest operating systems and versions and such to use. Very cool.

Next we have some port forwarding settings. Vagrant takes care of setting up the network for you, you just have to tell it what you need. So I’m just forwarding to port 6379 on the guest machine (the default port on which redis runs) from port 6379 on my host machine.

Next I customize the vm to have a gig of memory instead of whatever the base box has by default.

So that’s it for the configuration the box. The last line runs a provisioner which will setup the box once it’s running. There are a number of provisioners to choose from including Puppet, Chef and shell. This was the gotcha for me when I was doing it the first time. The docs list the provisioners in this order…

So I spent an hour or two trying to grok the chef and puppet docs and ended up getting frustrated. Those systems have a bunch of abstractions in them which probably make them great for doing sys-adminny type stuff but in my head I was screaming “AAAARRRGH. JUST LET ME RUN A FUCKING SHELL SCRIPT!”. Of course I go back to the vagrant docs after that, look an inch or two down and feel like an idiot. I do wish the bullet points there went in order of increasing complexity though.

Long story short, the provisioner just executes the specified shell script on the guest box after it boots up.

Shellack It

So once the machine boots up what do we want it to do? Well, this:

(init.sh) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash
mkdir /opt/redis
mkdir /opt/redis/bin

cd /opt/redis
wget http://redis.googlecode.com/files/redis-2.6.7.tar.gz
tar xzf redis-2.6.7.tar.gz

cd redis-2.6.7
make

cp src/redis-server /opt/redis/bin/redis-server
cp src/redis-cli /opt/redis/bin/redis-cli

cp /vagrant/redis.init.d /etc/init.d/redis
cp /vagrant/redis.conf /etc/redis.conf

useradd redis

/etc/init.d/redis start

So first we make the directories where redis will live. Then we go to the top level one, download and extract the code for the version of redis we’re interested in and build it. Then we copy the resulting executables to their final home in /opt/redis/bin.

Next we copy an init.d script to where it needs to be, then we copy the redis configuration to where it lives. Add a redis user, start redis and we’re all finished.

You might be asking “How did that init.d script and redis configuration get into the vagrant directory on the guest box?”

The way you run vagrant is by going to the directory where the Vagrantfile lives and typing vagrant up. That starts the whole ball rolling. When vagrant starts up your VM, it automatically shares the directory where the Vagrantfile is with the guest box at /vagrant on the guest box. It’s a magical default behavior.

So that’s pretty much it

Well, for now anyway. Vagrant can be used to setup multiple machine environments (which I might do next to test out an elasticsearch cluster for Careers). It has many more bells and whistles to keep your virtual dev environment running lean and mean. I’ve been super impressed with just how easy it is to work with (total home grown code to get my redis VM up was 31 lines, 15 of which were the shell script for installing redis) and bonus everyone on my team thinks I’m a hero. It’s that magical.

+1 Vagrant…+1.

Appendix

This is the init.d script I used which I cribbed from Ian Lewis.

(redis.init.d) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
#! /bin/sh
### BEGIN INIT INFO
# Provides:     redis-server
# Required-Start:   $syslog
# Required-Stop:    $syslog
# Should-Start:     $local_fs
# Should-Stop:      $local_fs
# Default-Start:    2 3 4 5
# Default-Stop:     0 1 6
# Short-Description:    redis-server - Persistent key-value db
# Description:      redis-server - Persistent key-value db
### END INIT INFO


PATH=/opt/redis/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=`which redis-server`
REDIS_CLI=`which redis-cli`
CONFIG_FILE=/etc/redis.conf
DAEMON_ARGS="$CONFIG_FILE"
NAME=redis-server
DESC=redis-server
PIDFILE=/var/run/redis.pid
LOGFILE=/var/log/redis.log

test -x $DAEMON || exit 0
test -x $DAEMONBOOTSTRAP || exit 0

set -e

case "$1" in
  start)
    echo -n "Starting $DESC: "
    touch $PIDFILE $LOGFILE
    chown redis:redis $PIDFILE $LOGFILE
    if start-stop-daemon --start --quiet --umask 007 --pidfile $PIDFILE --chuid redis:redis --exec $DAEMON -- $DAEMON_ARGS
    then
        echo "$NAME."
    else
        echo "failed"
    fi
    ;;
  stop)
    echo "Stopping $DESC"
    if [ ! -e "$PIDFILE" ]
    then
      echo "failed"
    else
      LISTENING_PORT=`grep -E "^ *port +([0-9]+) *$" "$CONFIG_FILE" | grep -Eo "[0-9]+"`
      $REDIS_CLI -p $LISTENING_PORT SHUTDOWN
      #rm -f $PIDFILE
    fi
    ;;

  restart|force-reload)
    ${0} stop
    ${0} start
    ;;
  *)
    echo "Usage: /etc/init.d/$NAME {start|stop|restart|force-reload}" >&2
    exit 1
    ;;
esac

exit 0

Comments