Jason Punyon

So You Want a Zillion Developers…

I work at Stack Overflow on Careers 2.0. In addition to our job board we have a candidate database where you can search for developers to hire. Our candidate database has 124K+ developers in it right now.

Customers frequently gawk at this number because they’ve looked at other products in the dev hiring space that offer millions of candidates in their databases. Sourcing.io claims to have “over 4 million developers” in their database. Gild offers “Over 6 Million Developers”. Entelo will give you access to “18+ million candidates indexed from 20+ social sites.”

Yeah man, your numbers stink

Hey. That hurts.

Let’s put those numbers in perspective. The vast majority of the developers “in” these other databases don’t even know they exist. The devs never signed up to be listed or even indicated that they were looking for work. There isn’t even a way to opt out. These databases are built by scraping APIs and data dumps from sites developers actually care about like Stack Overflow and GitHub.

On the other hand the only people you’ll find in the Careers 2.0 database are ones who made the affirmative choice to be found. They start by getting an invitation to create a profile. They build out a profile with their employment and education history, open source projects, books they read, peer reviewed answers on Stack Overflow, and so on. Then they can choose to be listed as either an active candidate (they’re in the market for a job right now) or a passive candidate (they’re happy where they are but are willing to hear your offer) or not to be listed at all. After a candidate gets hired they can delist themselves from the database so future employers don’t waste any time on them.

So the difference between us and them is that we give you a smaller number of candidates who are already interested in job offers and they give you a giant database filled with hope and built by skeez.

We have some data from Careers that tells us hope is not a recruiting strategy.

Our Experiment

Careers 2.0 experimented with the “index a bunch of people who don’t know they’re being indexed” model to see if it could possibly work. We created what we called “mini-profiles” which consisted exclusively of already public information available on Stack Overflow. We would add mini-profiles to the database if the Stack Overflow user provided a location in their profile and had a minimum number of answers with a minimum score. We showed these mini-profiles along with our “real” candidates in search results. If an employer wanted to contact one of the people behind a mini-profile Careers 2.0 would send them an e-mail asking if they want to open up a conversation with the employer. They also had the option to tell us to go away and never send them an e-mail again. If the candidate wanted to continue they could authorize us to share their contact information with the employer and they’d start working on their love connection.

Our Results

We track response rates to employer messages to look out for bad actors and generally make sure the messaging system is healthy. A candidate can respond to a message interested/not interested or they can not respond at all. Response rate is defined as Messages Responded To / Messages Sent. When we compared the response rates of messages to mini-profiles to the response rates of messages to “real” profiles the results were not good for mini-profiles. Messages to “real” profiles were 6.5x more likely to get a response than messages to mini-profiles. That was the last and only straw for mini-profiles. We retired the experiment earlier this year.

So what about the zillions of programmers?

All those services I named at the beginning of this post do what we did in our experiment, just a little more extensively by including devs from more places online. I have to believe that the response rates from their unqualified leads are similar to the ones we found in our experiment. I suppose the response rates from randodevs on GitHub or Bitbucket could be higher than that of randodevs on Stack Overflow thus invalidating our conclusion, but anecdotal evidence from our customers about those other services suggests not.

“Wait a sec there Jason,” you’re thinking, “if their databases are at least 6.5x larger than yours I’ll still get more responses to my messages right?” Absolutely! That’s called spam. You are totally allowed to go down the path of the spammer but let me hip you to the two problems there. The first problem with your plan is that devs hate recruiting spam more than they hate PHP, and they hate PHP a lot. The word will get out that you’re wasting everyone’s time. People will write about it. The second problem is that spam is supposed to be cheap. This isn’t cheap. In this case you’ll have to spend at least 6.5x the time wading through these zillions of devs identifying the ones that meet your hiring criteria, messaging them, and waiting for responses. So not only are you wasting their time, you’re wasting yours.

We aren’t going to build this business off hope and spam and wasting people’s time. If a smaller database is the price, so be it.

Commuting: A Perverse Incentive at Stack Exchange

So, we just went through comp review season here at the Stack Exchange. This is pretty much the only time of year we talk about money, because that’s the way we want it. We pay people enough to be happy and then shut up about it. You’ll probably only ever hear stuff about comp from me around September each year because that’s the only time it’s on my mind. The system works, and I’m generally happy about my financial situation, but we have a comp policy about remote work that subjects me to a bit of a perverse incentive when it comes to commuting.

The policy is that if you work out of the New York office, you get a 10% pay increase relative to what you’d be making if you worked remote. The reason for this has always been a little cloudy to me. I’ve heard cost of living adjustment. I’ve heard we want to incentivize people to be in the office because of “accidental” innovation from pick-up meetings and conversations in the hall. Regardless of the reason, that’s the policy.

I live in Stamford, CT and have been commuting to the New York Office 3 days a week (down from 5) since my daughter Elle was born in December. My total commute time averages just under 3 hours a day (10 min from my house to the Metro North, 55 minutes to Grand Central, 20 minutes from Grand Central down to the Office). So I end up commuting about 36 hours per month (down from 60).

On the Metro North getting a seat means cramming in next to one or two other people in side-by-side seats leaving little elbow room for typing (or living, FSM forbid they’re overweight), sitting in the seats that face each other and knee-knock with people who are drawn from a population with a mean height of 7 feet, or sitting on the floor in a vestibule near the doors. Some days the Metro North crawls because apparently they didn’t design this surface rail line to deal with even the slightest amount of rain. The subway is the subway, you get what you get. This commute stinks and it’d be my default position to forgo it.

Here’s where the perversion comes in. Let’s say I make $120K a year (I’m using this number because the math works out simply) out of the New York Office and decide to go remote. Every month I’ll make $1K less and get 36 hours of my life back. So Stack Exchange thinks my commute is worth $27.78 an hour. 4x minimum wage for no productive output is nice work if you can get it.

When done right, it makes people extremely productive. Private office? Check. Flexible hours? Check. Short commute? Check. I’ll let you in on a secret: most of our remote developers work longer hours than our in-office devs. It’s not required, and probably won’t always be the case, but when going to work is as simple as walking upstairs (pants optional, but recommended) people just tend to put in more hours and work more productively.

Going remote means a large portion of the 36 hours a month I spend commuting would go back to productive work (I won’t lie, some of it will be spent enjoying time with my daughter) so Stack Exchange is better off. I’d be happier because I get to skip the dreadful commute and work instead so I’d be better off. But I don’t make nearly enough that I can just drop 10% of my pay and not feel it.

Fun With RNGs: Calculating π

So, calculating π is a fun pastime for people it seems. There are many ways to do it, but this one is mine. It’s 12 lines of code, it wastes a lot of electricity and it takes forever to converge.

1
2
3
4
5
6
7
8
9
10
11
12
public double EstimatePi(int numberOfTrials)
{
  var r = new Random();
  
  return 4 * Enumerable.Range(1, numberOfTrials)
                       .Select(o => {
                                      var x = r.NextDouble();
                                      var y = r.NextDouble();
                                      return Math.Pow(x, 2) + Math.Pow(y, 2) < 1 ? 1 : 0;
                                    })
                       .Average();
}

What’s going on here? First we initialize our random number generator. Then for 1 to the number of trials we specify in the argument we do the following:

  1. Generate two random numbers between 0 and 1. We use one for the X coordinate and one for the Y coordinate of a point.
  2. We test if the point (X,Y) is inside the unit circle by using the formula for a circle (x2 + y2 = r2).
  3. If the point (X,Y) is inside the circle we return a 1 otherwise a zero.

Then we take the average of all those zeros and ones and multiply it by a magic number, 4. We have to multiply by four because the points we generate are all in the upper right quadrant of the xy-plane.

How bad is it? Here’s some output:

    Number Of Trials       Estimate of Pi
        10                  3.6
        100                 3.24
        1000                3.156
        10000               3.1856
        100000              3.14064
        1000000             3.139544
        10000000            3.1426372
        100000000           3.14183268
        1000000000          3.141593 (Took 2:23 to complete)

Things That, Were I to Unfortunately Wake Up Tomorrow as a Recruiter, I Would Never Do

I would never send e-mails that make potential candidates for a position think I’m not effective at finding potential candidates for a position. Giving candidates that impression just makes them think I stink at everything else too.

Subject: Barbara Nelson in Search of Javascript Expertise

Do you mean the Barbara Nelson?

Hello from Barbara!

What a great salutation! Not. Save that one for your next family newsletter.

I saw your profile either on github or on stackoverflow

Really? WOW! It sounds like you did a lot of research on me and moreover you’re the kind of go-getter who keeps the relevant information she needs at her fingertips at all times.

I am looking for several strong JavaScript Object-Oriented Engineers (not “just” web developers). These three openings have been especially challenging to fill…

Well alright let me click through and see what these jobs are about. Oh…no company names? The third one is really a C++ job? And you say you’re having trouble filling these positions?

Some JavaScript opportunities I am helping to fill are at solid funded start-ups, some are at start-ups already acquired by a well-known global company with solid benefits. We can make your relocation to the beautiful Bay Area happen if there’s a good fit.

That’s good I guess…I’m not really that interested in moving to the Bay Area.

Those who are interested in a brief discussion on the phone: please send a resume or an online profile that reflects your experience, a good time to talk, and a good phone number, and we’ll schedule a quick call.

Those who sent this e-mail should learn how to address the recipient directly and singularly instead of giving the impression that this is just another useless e-mail blast from a contingency recruiter.

If you never want to hear about career opportunities from me again, just let me know; reply and say so.

By the way you almost whited that out I’d almost think you didn’t want me to actually do that.

I love referrals.

I love how I almost don’t even get the feeling you’re trying to get me to do your job for you.

Wat?

Contrast

So let’s look an e-mail with a similar goal.

Subject: Facebook Engineering

Do you mean the Facebook? Let’s not be unfair to poor Barbara. Her subject line is much harder to get right than this one.

I hope all is well. I had the pleasure of stumbling upon your information online and saw that you have been working on some pretty neat stuff with Stack Overflow and various companies (it wasn’t disclosed on your resume) plus you have an awesome academic background from SUNY Geneseo to complement it.

This is much better than what Barbara had to say about me. Minimally Jeremy has read my public Careers 2.0 profile and noted my current position and where I went to school. He also called out the fact that I don’t list the companies I’ve worked at before on my profile (mainly so I can write about my experiences there when I want to without anyone getting bent out of shape). This e-mail is about me. It’s not a cattle call.

I am currently helping grow our engineering team in the NYC office and would love to chat with you about what you’ve been up to and perhaps put us on your radar; if nothing else we can have a friendly conversation. Let me know what works for you and we can schedule a time at your convenience. If this isn’t the right time, I completely understand and we can stay in touch based on your schedule – no rush. I look forward to hearing from you.

Great tone. Sounds like a human. He tells me what he’s after while being accomodating and not pushy. He makes me believe that if I respond, he’s going to respond back. Jeremy could’ve broken some of this down into paragraphs to make it less WALLOFTEXT but other than that it was a decent recruiting e-mail.

Get Your Redis On on Windows

TL;DR: Want a virtual machine running redis in however long it takes you to download 400MB + a little completely automated install time? Follow the instructions here and you’ll be on your way.

Well, it only took me a year of having this blog for me to write up something even remotely technical. But here you are, and here I am…so let’s just tone down the celebration a little bit and get on with it already.

So…it’s hard running a dev environment sometimes. We at the Stack Exchange will use anything to get the job done, but on the programmer’s side we’re mainly a windows shop. One piece of software we’ve come to know and love is Redis though. We love it so much we’ve got antirez on speed dial. It’s really the greatest.

Here’s where it isn’t quite the greatest though (for us): it’s really meant to run on Linux. Some people have made mostly working windows builds in the past that were good enough for dev’ing on but had weird behavior when it came to background operations. They’re great and I appreciate the work they d(o|id), but they fall behind when redis bumps stable versions (it’s behind 1 stable version right now leaving out features like the Lua scripting engine). Microsoft went through the rigamarole of patching redis so that it will run on windows, but that patch isn’t getting merged to master…ever.

So what’s a girl to do? When I’ve been on a team of one and had this kind of problem I thought to myself, “Self! Get VMWare on here, spin up a one off VM with ubuntu and just run it there! Problem Solved!” and many internal high-fives were had. But when you’re on a team of 6 (the Careers team, plug: we’re hiring) that doesn’t really scale well. So what are my choices? Let’s go to the big board of options:

  • Just tell my teammates “Hey, spend a couple hours spinning up your own VM and hope the one you have and the one I have match up and behave exactly the same”. (HINT: No)
  • Check in a 10 gig VM into source control and push so the other members of the team can run it too? (HINT: NO. That’s an example of what we call the “I quit” check-in.)

So how do you solve this problem?

Enter the Hobo

So it turns out a bunch of other people have this problem too (WEIRD RITE?). A smart dude decided to solve it and created Vagrant. Vagrant is a super simple yet powerful way to create and manage a reproducible virtual dev environment. Check in a couple kB of config and you get a virtual machine (or a multiple machine environment) your whole team can run. Vagrant wraps around Virtual Box for it’s virtual machines and it’s not just for windows. It runs on Linux and Mac too. Let’s run it down.

Getting in Installed

Follow the startup guide here. It’s basically install VirtualBox and install Vagrant.

Creating a machine

To create the machine, the first thing we do is create your Vagrantfile. Don’t worry…it isn’t a driter fetish. It’s just a config file that outlines how your virtual machine is setup. It’s also just a bit of ruby. Here’s the one we’re using:

No Drifter Fetishes Here (Vagrantfile) download
1
2
3
4
5
6
7
8
9
10
11
Vagrant::Config.run do |config|
  config.vm.box = "ubuntu-12.04-amd64"

  # THE URL FROM WHERE THE 'CONFIG.VM.BOX' BOX WILL BE FETCHED IF IT
  # DOESN'T ALREADY EXIST ON THE USER'S SYSTEM.
  config.vm.box_url = "http://dl.dropbox.com/u/4031118/Vagrant/ubuntu-12.04.1-server-i686-virtual.box"
  config.vm.forward_port 6379, 6379
  config.vm.customize ["modifyvm", :id, "--memory", 1024]

  config.vm.provision :shell, :path => "init.sh"
end

So first we tell Vagrant which box to use. A box is essentially a map from a key to a file. Box names can be anything you want, in this case I just have a name telling me that it’s ubuntu’s latest 64-bit release.

Next we have a url that points to a file. As it says in the comment there, this url points to a box file that will be downloaded if the box with the name in config.vm.box doesn’t exist. This is nice because it means i send the file and when my teammate runs it it will go fetch everything it needs to create the virtual machine. Brilliant. A bunch of base boxes can be found at Vagrantbox.es. They have many different guest operating systems and versions and such to use. Very cool.

Next we have some port forwarding settings. Vagrant takes care of setting up the network for you, you just have to tell it what you need. So I’m just forwarding to port 6379 on the guest machine (the default port on which redis runs) from port 6379 on my host machine.

Next I customize the vm to have a gig of memory instead of whatever the base box has by default.

So that’s it for the configuration the box. The last line runs a provisioner which will setup the box once it’s running. There are a number of provisioners to choose from including Puppet, Chef and shell. This was the gotcha for me when I was doing it the first time. The docs list the provisioners in this order…

So I spent an hour or two trying to grok the chef and puppet docs and ended up getting frustrated. Those systems have a bunch of abstractions in them which probably make them great for doing sys-adminny type stuff but in my head I was screaming “AAAARRRGH. JUST LET ME RUN A FUCKING SHELL SCRIPT!”. Of course I go back to the vagrant docs after that, look an inch or two down and feel like an idiot. I do wish the bullet points there went in order of increasing complexity though.

Long story short, the provisioner just executes the specified shell script on the guest box after it boots up.

Shellack It

So once the machine boots up what do we want it to do? Well, this:

(init.sh) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash
mkdir /opt/redis
mkdir /opt/redis/bin

cd /opt/redis
wget http://redis.googlecode.com/files/redis-2.6.7.tar.gz
tar xzf redis-2.6.7.tar.gz

cd redis-2.6.7
make

cp src/redis-server /opt/redis/bin/redis-server
cp src/redis-cli /opt/redis/bin/redis-cli

cp /vagrant/redis.init.d /etc/init.d/redis
cp /vagrant/redis.conf /etc/redis.conf

useradd redis

/etc/init.d/redis start

So first we make the directories where redis will live. Then we go to the top level one, download and extract the code for the version of redis we’re interested in and build it. Then we copy the resulting executables to their final home in /opt/redis/bin.

Next we copy an init.d script to where it needs to be, then we copy the redis configuration to where it lives. Add a redis user, start redis and we’re all finished.

You might be asking “How did that init.d script and redis configuration get into the vagrant directory on the guest box?”

The way you run vagrant is by going to the directory where the Vagrantfile lives and typing vagrant up. That starts the whole ball rolling. When vagrant starts up your VM, it automatically shares the directory where the Vagrantfile is with the guest box at /vagrant on the guest box. It’s a magical default behavior.

So that’s pretty much it

Well, for now anyway. Vagrant can be used to setup multiple machine environments (which I might do next to test out an elasticsearch cluster for Careers). It has many more bells and whistles to keep your virtual dev environment running lean and mean. I’ve been super impressed with just how easy it is to work with (total home grown code to get my redis VM up was 31 lines, 15 of which were the shell script for installing redis) and bonus everyone on my team thinks I’m a hero. It’s that magical.

+1 Vagrant…+1.

Appendix

This is the init.d script I used which I cribbed from Ian Lewis.

(redis.init.d) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
#! /bin/sh
### BEGIN INIT INFO
# Provides:     redis-server
# Required-Start:   $syslog
# Required-Stop:    $syslog
# Should-Start:     $local_fs
# Should-Stop:      $local_fs
# Default-Start:    2 3 4 5
# Default-Stop:     0 1 6
# Short-Description:    redis-server - Persistent key-value db
# Description:      redis-server - Persistent key-value db
### END INIT INFO


PATH=/opt/redis/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=`which redis-server`
REDIS_CLI=`which redis-cli`
CONFIG_FILE=/etc/redis.conf
DAEMON_ARGS="$CONFIG_FILE"
NAME=redis-server
DESC=redis-server
PIDFILE=/var/run/redis.pid
LOGFILE=/var/log/redis.log

test -x $DAEMON || exit 0
test -x $DAEMONBOOTSTRAP || exit 0

set -e

case "$1" in
  start)
    echo -n "Starting $DESC: "
    touch $PIDFILE $LOGFILE
    chown redis:redis $PIDFILE $LOGFILE
    if start-stop-daemon --start --quiet --umask 007 --pidfile $PIDFILE --chuid redis:redis --exec $DAEMON -- $DAEMON_ARGS
    then
        echo "$NAME."
    else
        echo "failed"
    fi
    ;;
  stop)
    echo "Stopping $DESC"
    if [ ! -e "$PIDFILE" ]
    then
      echo "failed"
    else
      LISTENING_PORT=`grep -E "^ *port +([0-9]+) *$" "$CONFIG_FILE" | grep -Eo "[0-9]+"`
      $REDIS_CLI -p $LISTENING_PORT SHUTDOWN
      #rm -f $PIDFILE
    fi
    ;;

  restart|force-reload)
    ${0} stop
    ${0} start
    ;;
  *)
    echo "Usage: /etc/init.d/$NAME {start|stop|restart|force-reload}" >&2
    exit 1
    ;;
esac

exit 0

Geography’s the Fuck.

Raganwald poses an interesting question. Why do some of the best minds in our industry spend time figuring out how to make people click more on ads? Aren’t there more interesting problems for these bright up-and-comers to spend their valuable time and insight on?

One simple answer to the question is Geography.

How do you know this?

I’ve spent a little time figuring out how to make people click more on ads :)

I work at Stack Exchange on our Careers 2.0. We try to make it easier for programmers to get better jobs. We have two main ways of getting people to the Careers 2.0 website. Our users evangelize on our behalf by inviting their programmer friends to show off their accomplishments on their own Careers profiles (like mine here).

The second way looks like this

Me and the team spent a bit of time trying to figure out how to get programmers to click on this ad. It (and it’s smaller variant) shows up on the vast majority of the pages of Stack Overflow.

The number one way to make people click on this ad is to show them a job from a place near them. It’s simple as that. Showing the user a job near them outperforms every other way of constructing the ad we could come up with by a factor of between 2 and 5.

Location is still incredibly important to job seekers (and employers). If our ad analysis is to be believed, they are more interested in having a job near them than having a job that matches their skills. Or a job that is interesting. Or a job where they work on “super important” problems.

Personally…

I know this to be true from my own experience. I spent the summers of my college years working in Berkeley, CA for Lawrence Livermore National Laboratory. We did low energy nuclear physics experiments measuring certain reaction cross sections on radioactive nuclei. Some of this work gets used downstream by the Stockpile Stewardship program. Stockpile Stewardship is the program responsible for maintaining the reliability of the United States nuclear arsenal. The United States doesn’t test nuclear weapons anymore and this program ensures that the ones we have continue to work, and ascertain the failure modes of the ones that aren’t going to work. I think this would qualify as an important problem to Raganwald (and most other people). Most of the work I did was on software for simulating our particle and gamma ray detectors.

After college was over I ended up back in New York because that’s where my life was. I worked at a management consulting company and then a couple hedge funds before I got my head out of my ass and realized I wanted to work on something that mattered a little bit more than the continued aggrandizement of the uber-rich. By then I’d been married for a few years. My wife had a job in New York, my family was in New York (and elsewhere on the East coast). If I was going to work on something that mattered, it was going to have to be something that mattered…in New York.

Wait…haven’t we solved this?

But, what about technology? Hasn’t technology solved the geography problem? Don’t we have Skype, company chat, Google hangouts and shudder the lowly telephone to connect people remotely? All these media come with their own problems. You need people who are exceptionally good at communicating through electronic media for these solutions to work. It is REALLY HARD. Stack Exchange was founded remotely. We built our own chat system because others were inadequate. We hire a bunch of remote devs, sys admins and others. I do internal support for people in 3 timezones (that’s more than any of my international hedge fund jobs). A lot of people aren’t cut out for it. I’ll admit it, sometimes I’m not.

Long story short it’s really easy to say “Hey we’ll just hire the best people remotely.” but much harder to do in practice. Your culture has to be just so.

Star-crossed

Unfortunately you and the “right job for you” (for some criteria of right) are star-crossed. At all times there’s an ordered list of jobs that are a best match for you based on all factors. Rarely anyone has the Juliet at the top of their list. You and your Romeo pass eachother silently (and not so silently) in the night for myriad reasons. Transaction costs for switching jobs are high. People need certainty. I’m harping on geography here, but it’s merely one of the reasons people aren’t working “the right” job or fixing “the right” problems.

Geography is important to programmers. It’s probably at the top of the list of factors that goes into deciding whether a job is “right”. If the company working on “important” problems doesn’t jive with your geography, you’re probably going to leave it by the wayside.

Thanks to Matt Jibson for an edit.

A Guy Walks Into an Apple Store…

So me and my wife had a babby recently. Unfortunately my wife had some preterm labor around week 33 and we had to spend about 10 days in the hospital. Don’t worry, everything turned out all right (see perfection below), but I was pretty burnt out after the 10 days in the hospital. My wife was happy with my performance during our mini-crisis and she told me to go indulge a little. I’d been super excited about the iPhone 5 release, so I went and picked one up.

"Perfection"

Fast forward a few weeks and a couple things started going wrong with the phone. The first thing you can probably see from Elle’s picture. Let me blow it up for you…

That’s not a birthmark, something was wrong with the camera. It showed up the week my wife was due. I had slightly more important things happening so it just went on the ever growing pile of stuff I needed to do later.

Then right after the babby was born another problem popped up. I was talking to my sister on the phone at CVS and the receiver just cut out. Tried calling back, no dice. The speakerphone worked and if I used the headphones I got sound but the receiver was completely borked. But then a day later I picked up the phone and called someone and voila the receiver was working again. Weird. Cut to a few days later and it was out again. So the problem was intermittent.

I’d never had a problem with an Apple product before, so I had no idea how their customer service was in situations like this. I assumed the worst. I worried they were going to tell me it was my fault and they had never seen problems like this and if I want to get it fixed it’d cost ${hefty sum}. So as I usually do when I think I’m going in against a company that’s going to try to screw me I went about building my case.

I started searching around for problems like mine. I found a couple threads talking about the earpiece problem. So it wasn’t just me or my phone. I fired up my iPad and loaded up those bookmarks in safari and headed to the store. I didn’t have a plan for the camera discussion, but the earpiece case seemed pretty strong. I figured I’d lead with that, then throw the camera problem in later.

I drove to the store, steeling myself to deal with crappy customer service and a maybe dim/maybe dumb rep. Went over the plan in my head a couple times and thought about how to deal with refutations. I arrived, walked up to the genius bar and made an appointment. They told me it’d be an hour so I cruised around the mall for a while and came back. I bellied up to the bar to meet my genius with my first rehearsed line on the tip of my tongue ready to start sparring.

She asked me what the problem was and I demonstrated it for her by pressing play on one of my voicemails and holding the phone up to her ear. “See, no sound.” I said. Then before I was able to say anything else she whisked my phone away into the back and came back a minute later. “Yep. I confirmed there was no sound. We’re going to go ahead and replace your phone.”

But…but…I had all my plans to deal with a crappy customer service rep and get outraged about some ridiculous policy. I practiced arguments. I didn’t even get to show you my links!

I had dealing-with-bad-customer-service-blue-balls. I was so ready for an argument that just never materialized. So what do you do when you get what you want? YOU SHUT UP and let them give it to you. I almost made it but broke while she was setting up the new phone. I tossed out a “Is this kind of thing common? I read about it on blah blah blah” nonchalantly, half making conversation but mostly just needing to get out some of that pent up preparation frustration. I picked up my brand new working phone and headed out happy as a school girl.

Compare and Contrast

A few days later I picked up Influence: The Psychology of Persuasion. This isn’t usually the kind of thing I read, but I was really impressed with a guy I learned about named Apollo Robbins. It was reported he was enamored with the book so I grabbed it on my kindle and started reading.

Chapter 1 of this book talks about a bunch of stuff, but the thing that caught my eye was the contrast principle.

There is a principle in human perception, the contrast principle, that affects the way we see the difference between two things that are presented one after another. Simply put, if the second item is fairly different from the first, we will tend to see it as more different than it actually is. So if we lift a light object first and then lift a heavy object, we will estimate the second object to be heavier than if we had lifted it without first trying the light one.

Robert B. Cialdini Influence: The Psychology of Persuasion

So What’s The Point?

OK, so I’ve had you reading this pedestrian story about increasing the planet’s population, things working out splendidly at the Apple store, a cool video on the internet and a 6 year old book for 13 paragraphs now. Here’s the point:

Having good customer service is even more powerful than you think.

I know…that came out of nowhere, right?

Things go wrong with company’s products. This makes customers worry. Not just because the thing they have is broken, but because there’s so much bad customer service out there and they now have to go interact with it. Cable companies, utility companies, these guys, the MTA, and the DMV are all out there creating bad customer service experiences.

When something goes wrong with your product your customer gets worried. They’re primed by all the bad customer service experiences they’ve ever had and they call you up or walk into your store. At this point you have a tremendous opportunity to take advantage of the contrast principle. They’ve presented themselves with all the bad service they’ve ever gotten worrying if you’ll be the same. They’re aiding and abetting your success before they even talk to you. Present them with just adequate service and because of the amplifying effect of the contrast principle they’ll think you’re the greatest thing since that last thing that was so great.

Now imagine how over the moon your customers will be when you empower your customer service team to go the extra mile and literally do everything it takes to make them happy…

Rock Stars Went Where?

So I read this article today about how the tech industry is too elitist and thinks everyone who’s good is already rich and can follow their bliss until the cows come home waiting for the olympian software companies of the world to come hire them. I just wanted to provide a data point from Stack Exchange. I’m not gonna bloviate that we’re all rock stars at Stack Exchange (I wouldn’t even describe myself like that) but here’s the list of schools attended by all the programmers and sysadmins at Stack Exchange:

Georgia State University
Arcadia University
Metro State College of Denver
Technische Universität Clausthal
Duke
Rice
University of Texas
University of Exeter
Boston University
University of New South Wales
Rensselaer Polytechnic Institute
State University of New York at Geneseo (Plug: The physics department is awesome.)
University of Pittsburgh
Dartmouth
Colorado State University
New England Conservatory of Music
Cleveland Conservatory of Music
North Carolina State University (2)
Washington University, St Louis.
Montgomery County Community College
Northern Territory University

You might notice a significant dearth of Ivy there. (And the one Ivy Leaguer we do have has been promoted to management so he doesn’t even code anymore. We still love him though :))

An underlying premise of Stack Overflow and Stack Exchange is that there’s tremendous amounts of knowledge tied up in “ordinary” people, and given an easy way to show it off and a little incentive they can make this knowledge available and help as many people as possible. Given Stack Exchange’s belief in the common (wo)?man it’s not surprising that the educational background of our programmers and system administrators is pretty…well, ordinary.

Locke setup up a straw man straw man by invoking Joel. I know he said the words “Ivy League” but Joel wasn’t really talking about the Ivy League. He was talking about selectivity. Specifically that as a sorting criteria a resume showed that the associated applicant had successfully navigated some process that was highly selective:

Selectivity. Another thing we look for on resumes is evidence that someone has gone through some highly selective process in the past. Not everyone at Ivy League schools is worth hiring, and not everyone at community college is worth avoiding, but getting into a very selective school does at least mean that someone, somewhere judged you using some kind of selection process and decided that you were pretty smart. Our company criterion for selectivity is usually getting into a school or program that accepts less than 30% of its applicants (there are about 60 schools in the US that meet this standard), or working for a company which is known to have a difficult application process, like a whole day of interviews. Highly selective branches of the military like officer’s training or pilot’s courses, or even just getting into the Marines indicates someone that has made it through some kind of difficult application/selection procedure and all in all this is a positive sign.

Tech companies are looking outside the Ivy League and the Dan Shippers of the world for their “rock star” developers. I could list the previous positions of all the Stack Exchange programmers and sysadmins but I’ll save you the suspense: none of us were paying our bills running our own startups just waiting for something awesome like Stack Exchange to come along. In my case I sought them out and quit my cushy yet crappy hedge fund job.

Complaining about how it’s not fair because they went to an Ivy League school and you didn’t or they started a company and you didn’t and these things makes their resume look better than yours isn’t going to get you very far, though. You have to make it easy for employers to choose you from the pile of resumes. You know there’s competition out there for these jobs. If you want to work at a start up you have know you’re going up against some of the best in the industry and you have to make yourself stand out.

Your resume (or your Careers 2.0 profile) has the single purpose of showing them you’re worth it. It has to make them believe you’re worth calling for an interview. Provide some evidence. Don’t have any? You’re a programmer…manufacture some! Make a blog. Write a ruby gem. Answer some questions on Stack Overflow. Write some code for a charity. Make a website for your band. This stuff won’t take forever to do. It isn’t like the old days where you had to apprentice with a master furniture maker for a decade to get some cred. The medium you work in allows you to make things in days, not years. It doesn’t have to change the world, it just has to give the person looking at your resume a reason to choose you instead of the other g(al|uy).

Just in case you’re in the market and looking to stand out…we built Careers 2.0 to help you stand out by enabling you to show off your programmery stuff. It’s invitation only so if you’d like an invite just tweet me @JasonPunyon and I’ll hook you up. You may not have gone to an Ivy League school but if you can show off some evidence that you’ve got skills you rank up there with the best of the Yalees and Browners. You’ve got a section for open source projects hosted at Github, Bitbucket, Code Plex or Source Forge. There’s a section for Apps you’ve written. Write a blog or read any good books lately? Choose your favorites and show them off. When employers search the Careers database for candidates with particular skills we take all of this stuff into account and sort you accordingly.

Also! We’re always looking for great developers to come work with us at Stack Exchange. Don’t let what school you went to keep you from applying today (we’ll even take you Ivy Leaguers).

Oh and again, if you’re into physics…GO TO GENESEO…if only to watch Dr. Fletcher’s valiant yearly attempts to demonstrate quantum tunneling by running full speed towards the classroom wall. He hasn’t succeeded yet, but if he tries long enough probability says…

Don’t Be Represented by Contingency Recruiters

Before I faded a crazy counter offer and got my dream job I worked in finance. I was going on interviews in 2009 having been laid off from my second job after the 2008 collapse. This was back in the days when I was still using contingency recruiters to find jobs. They’re a natural fit for finding finance jobs because banks and hedge funds have money they’re willing to spend on recruiters when they aren’t swimming in it (artists rendering on the right), or spinning it into thread for their golden parachute side business.

A Counter…

I went on an interview and it turned out they liked me, a dubious honor in retrospect, so we continued through the process. One day my recruiter, let’s call her Nancy, phoned me up and had exciting news. Nancy had always been pulling for me. She was on my side. We’d gone through interviews at a couple of companies together already. This one liked me enough that they wanted to make me an offer at $X a year. “Great!” I thought.

I took a pause and rewound to what happened the last time I was made an offer. I didn’t have a particular reason for doing so, but after I heard the offer of $Y I said “It’d be really nice if you could make that $Y+5K a year.” The guy I was talking to was the one who would go on to hire me and without a thought he said “Sure, no problem.” So going off my statistically significant set of one data point I said the same thing to Nancy.

I was expecting “Hey, let me check on that for you” or even a hedged “I’m not sure where they are on the range, but I’ll find out”. Instead Nancy’s tone dropped from celebratory to agitated. “Jason, I don’t know what you’re thinking…they aren’t going to move on this. You need to take it.” This was not the Nancy who was on my side. This was Nasty Nancy, a voice and a force not to be trifled with. The exact same question provoked a totally different response. Why would she have turned on me so quickly for my inquiry, when the last person I asked had no problem with it?

Incentives…

There could be a million reasons, but the incentives for contingency recruiters best explain the evidence. Contingency recruiters are paid on commission when they successfully place a candidate in a job and the candidate stays for a honeymoon period of 3-6 months. The payment is a handsome ransom at 25%-35% of the candidate’s first year salary. So for a $100K job they might see $25,000 (No doubt a kingly sum for what is essentially an introduction, but that’s neither here nor there. I don’t understand it, but the market pays what the market pays).

So my attempt at a counteroffer put a little wrinkle in her reward. She felt like she had a done deal and I was jeopardizing her jackpot. “The company made an offer! It’s so close and this dunderhead is screwing it up!”. She had $ChunkOfChange in her hand. If the company was fine with the counteroffer she stood to make an extra ~$1K. If they weren’t, she could end up making nothing. I’m sure it didn’t help that business was down because everyone was getting laid off after the collapse in 2008 and there was less hiring than ever going on. Faced with this situation Nancy attempted to “Close the deal” with me. She took the hard line to try and get me to just accept the offer so she could book the commission.

A very similar situation comes up when you use a real estate agent to sell your house. They may not take the hard line like Nancy, but they will try to persuade you to take the deal if offerred. Real estate agents are paid the same way as recruiters, commissions for done deals. The percentages are lower (around 3% per agent) which magnifies the effect of them wanting to close the deal. The difference between $475K for your house and $500K for your house is $25K to you, an amount not to be sneezed at. The agent’s commision only goes up a mere $750 bucks though, from $14,250 to $15,000. Of course they’d much rather you take the deal than try and bargain your way up to a better price if it means they could lose everything.

Turns out this was a bad idea from the start…

When you’re the candidate and a recruiter is involved, my experience has been that you don’t talk to someone from the company very much aside from interviews. If you get rejected, you get rejected by the recruiter. When you get an offer you get it from the recruiter. So when it comes to a counter offer, most likely the recruiter is going to be in the middle. The problem with having the recruiter in the middle if you’re going to make a counter offer is Dual Agency.

Dual Agency is a term that comes from real estate. One situation where dual agency comes up is when one entity (a single agent or two agents that work at the same broker) represents both the seller and buyer of a house in a transaction. The problem is that a real estate agent has a fiduciary responsibility to the party they represent. They can’t fulfill this obligation to two parties on opposite sides of the same transaction. One side wants to sell high, the other wants to buy low, the agent is stuck in the middle unable to fulfill their duty.

Nancy doesn’t have a fiduciary obligation to anyone written in law like real estate agents do but the essence of the dual agency problem still exists. When Nancy’s services were employed by the company, Nancy became their agent. By trying to route a counter offer through her I was trying to engage her in dual agency. The employer wants to buy low, I want to sell high and Nancy’s stuck in the middle unable to effectively represent both sides. It was a bad idea from the outset. Should you not heed my advice (really…you should) and use a contingency recruiter anyway, make sure you don’t try to make a counter offer through them.

I’m taking my ball…

Because of the incentives at play and the problems of dual agency the goals of a contingency recruiter (close the deal at any price) simply aren’t aligned with those of a candidate seeking a job (sell my labor for the highest price). I don’t have any data to support it (I’d love to see some if you’ve got it) but I wouldn’t be surprised if candidates who choose (yes…it’s your choice) to be represented to companies by contingency recruiters end up with lower salaries than they could’ve had because of this misalignment (not to mention that giant fee the company pays).

Ultimately I don’t think Nancy’s a bad person who was nefariously trying to screw me out of anything, or that she wanted to be mean to me. Getting a stack of money dangled in front of you and being threatened with getting it yanked away at the last minute is going to have an effect on someone’s behavior.

Incentives matter and I don’t agree with the incentive scheme that’s been setup, so I’m just not going to play ever again.

Don’t be represented by contingency recruiters.

On Counter Offers

Last I left you, dear readers, I had just been given an offer to work my dream job at Stack Exchange by Joel. I accepted the offer on the spot, and went home walking on air. What do you do when you get the offer for your dream job? You go home, crack open a bottle of wine and write your resignation letter.

Achieving a dream is really liberating. I’d written a resignation letter before, but it wasn’t like this. Last time I was leaving one garbage job for one with promise but there was ambivalence instead of the uncontainable excitement I was feeling. I knew things were getting “better”, but still didn’t know what to expect. This time I was typing with purpose and a carefree attitude. Things were gonna be great! I know this sounds silly because resignation letters are generally a one line affair. “This is my notice that I will be leaving the company effective [notice period] from today.” I printed out three copies, signed ‘em, enveloped ‘em and I was ready to go in in the morning and start my new adventure.

I had two bosses. So naturally I printed out three copies. One for each of my bosses and one for the #2 guy at the 1,000 person company that needed one according to the employee handbook (this would be my first interaction with him). I went in to work and boss #1 was already there. I checked the error log and made some quick fix commits while waiting for boss #2 to arrive.

Boss #2 finally got there and I called them both into boss #2’s office. We sat down and I broke the news to them and the inevitable happened. They started bargaining.

Boss #1: “Well, is there anything we can do to keep you around?”

Me: “No. This is my dream job.”

Boss #2: “What about $[2X my current salary]”

BOOM. The counter offer and a doozy at that. The employer’s last resort in the face of losing a good employee.

Boss #1’s eyes went a little wide, remembering that we went through bonus and raise season a few months earlier where he and I had it out over a pittance relative to the current offer on the table. B&R season at hedge funds is a harrowing time for everyone at the company. Almost all productivity stops for a month while secret incantations are spoken in back rooms by big wigs and pointy-headed bosses. The results are never transparent and rarely equitable and good people end up unhappy.

This counter offer reeked of skeeze. He was trying to buy my dreams from me. When someone tells you “I’ve been given the opportunity to live my dreams, I’m leaving” the decent human thing to do is to congratulate them and send them on their way. (With a security escort out the building, of course. You don’t know what these dream heads are gonna do on the way out.) What you don’t do is tell them you think so little of them, their lives and their goals that you think can buy them off.

The only thing the counter offer does is open up the possibility that the employee might take it. In my case I had a really positive reason about another company for leaving. It usually goes the other way though. The employee can have a really negative feeling about their current job, which led them to seek out new employment. Either way, the worst thing that can happen at this point is for the employee to take the counter offer.

If I were to accept the completely out of whack offer on the table, the company would have successfully bought my dreams…for a little while. Eventually, the hard truth of “Holy shit, did I really just sell me and my dreams down the river for a couple extra benjamins?” would get the better of me, and I’d end up really really unhappy. Bad for me. Over time I’d become less and less productive and more and more depressed at work and now I cost them minimum twice as much. Bad for the company.

The other case, where the employee is leaving because the company she’s at sucks is pretty much just as bad. The employee is so unhappy with the company that they’ve gone to the trouble of updating their resume, getting in the candidate pipeline at at least a few companies, phoners, in person interviews, second in person interviews. They’ve invested an inordinate amount of time and effort which should tell you, the company, one simple thing. It’s not them, it’s you. Again, money might paper things over for a while but the ultimate truth is, they’re not going to be the awesome employee you want. They’re going to end up just as unhappy and unproductive after the money wears off as I would’ve been had I taken it.

The other thing employees have to think about with an outsize counter offer is the fact that it may just simply be a pipe dream. Companies don’t just allow their managers to get slapped around and make outrageous counter offers to people. Budgets and HR minions can’t stand that. The manager gets dinged by the higher ups because she’s “lost control”. So some of these offers are just plain “keep them in the building while we find a replacement who’ll do the job at the price we were paying.” As soon as they do that, you’ll find yourself on your ass without the cushion of the comfy offer you declined to take earlier.

I didn’t miss a beat when I heard the offer, told Boss #2 “No,” and got up and left the room.

There was no way I was going to let this skeezy jerk buy my dreams.