How Artsy Hires Engineers

By Ash Furrow, Lily Pace, Steve Hicks

Interviewing is hard. Interviewers want to make sure they're hiring the person who will add the most value to their team; candidates want to make sure they're joining a company that aligns with their goals and perspectives.

Recent trends in hiring are white-boarding sessions, trivia questions, and hours of take-home assignments. At Artsy, we don't use any of these. We often get asked why not - and how we assess technical skill without them.

We think our interview process at Artsy is unique, but we also think our interview process is great. We'd love to see the tech community examine its hiring practices, and hopefully to adopt some of what's made our hiring process successful. Focusing on knowledge and facts that are already acquired is one way to approach hiring; we prefer to look at how a person can fill a gap in our team and help us grow.

Artsy's process of hiring new engineers was created and is maintained by our current engineers. It has evolved over time as we learn new lessons and new perspectives join our team. Our process has always been driven by a top-down culture of respect for candidates, which aligns with our company values. Our team currently has 36 engineers, and we refreshed our hiring practices last year to support our team's growth; we hired a dozen engineers in 2018. We don't use recruiters (though we did to hire our recent VP of Engineering).

Our former Director of Web Engineering has a blog post where he describes Artsy's hiring process. Though some specifics have since changed, the foundations remain the same.

If Artsy has a secret sauce, it is how it hires. All else falls from the assumption that they have hired the best people who want to work together to achieve Artsy’s mission.

Our hiring process starts with an informational, where candidates are met for a coffee or over a teleconference call. We have public documentation so candidates can know what to expect. We do a lot of these and move candidates who we think would succeed at Artsy on to in-person interviews. The interviews last 3 hours and are split across four 45-minute behavioral interviews, conducted by engineers and other colleagues, ranging from gallery liaisons to product managers to editorial writers. Artsy generally, and Engineering specifically, have both significantly invested in helping interviewers be effective and consistent; this includes documentation, question banks, and unconscious bias training.

Each interviewer is given key areas to focus on, based on the candidate's background. We have documentation specifying how to evaluate each of these areas, including example questions. These areas include, but aren't limited to:

  • Comprehension of Artsy
  • Artsy company values alignment
  • Ability to communicate complex ideas
  • Learning and adaptation
  • Self-learning and drive
  • Independence and teamwork
  • Systems development
  • Product knowledge

After the interview, feedback is written up as quickly as possible. To limit bias, interviewers can't see each other's feedback until after they write up their own. The write-up includes a recommendation: do you think we should move on to reference checks? Answers are either "strong yes", "yes", "no", or "strong no"; after everyone has completed their write-ups, the interviewers debrief and reflect on how to do a better job next time. Their feedback is used by the hiring manager to decide whether to move on to reference checks.

Quoting again from our former Director of Web's blog post:

Artsy believes that 'references are not a defense against hiring poorly, they are a way to hire great people'.

Artsy's reference checks are in-depth and deserve their own blog post; they are key to our hiring process. We know that job interviewers only evaluate how good someone is at interviewing, so we put a larger emphasis than most companies on references. The most accurate predictor of future job performance is past job performance, not how well someone can perform in an interview.

If we decide to hire the candidate, we make them a job offer. Artsy offers what we think is a fair wage based on the local market and the candidate; we do not low-ball candidates and we don't negotiate on compensation.

What's wrong with typical hiring practices?

There are many tactics for assessing a candidate's technical abilities, but we've found that many are unfair to the candidate. Some strategies put unnecessary pressure on the candidate. Some select against qualified candidates who have competing responsibilities outside work. Some unwittingly weed out underrepresented applicants, even at a time when companies are trying to diversify their teams.

In-person coding challenges

The intention of in-person coding challenges is to verify that the engineer can "actually write code." This strategy puts excessive pressure on the candidate to perform in front of an audience. This is usually not a good reflection of what the candidate would be doing if they were hired. Sometimes it is a reflection of the stressful conditions on the team, and the act of applying pressure to the candidate is intentional, to measure their ability to handle it. In either case, we don't feel like this is how we want to measure engineers; it just doesn't reflect reality.

Whiteboard interviews

One intention of whiteboard interviews is to reduce the stress on the candidate, because they don't have to worry about code syntax while under a microscope. These types of interviews still lead to stressful conditions, though, and they don't provide a good measure of what makes a great teammate or even a great developer. Again, sometimes the pressure is intentional, to see how the candidate reacts.

It can be very difficult to find a problem that is succinct enough for a whiteboard exercise but still reflective of the work the candidate will actually be doing on the job. The ability to write an algorithm to search a binary tree might be reflective of whether a candidate has a traditional Computer Science degree, but doesn't necessarily speak to their ability to build complex interfaces or streamline performance. More importantly, whether or not they can write a binary search tree from scratch on a whiteboard doesn't even necessarily speak to their ability to use search trees in day-to-day work. Questions like this can eliminate excellent developers who took a non-traditional approach to their knowledge building but are still highly capable.

Sample code

Sometimes a company will request a code sample from candidates - after all, what shows off their ability to code better than their actual code? The downfall of this strategy is that it eliminates developers who don't have code they can share. Many great engineers work for closed-source companies; many great engineers have family responsibilities that prevent them from contributing to open-source at night.

It is also important to consider the insularity and biases that exist in the open source community that can make contributing more difficult for developers from underrepresented groups. A study published in the PeerJ Computer Science journal found that women’s contributions to open source projects were accepted more frequently than men’s contributions when the gender of the contributor was unknown. However, when the gender of the contributor was apparent, men's contributions were accepted more frequently than women's.

Take-home challenges

The most recent trend in hiring is the take-home exercise. The goal is honorable - have the candidate produce code on their own time, so they aren't overwhelmed with the pressure of an audience. We’ve found that requiring this early in the process is unfair, and including it later in the process is uninformative; by the time a take-home challenge would be appropriate, we have already evaluated the candidate's technical skills to our satisfaction (more on that later).

This strategy also assumes the candidate has time to work on homework. Many single parents do not for example, nor do engineers who care for family members. There can also be misalignment on the expected time to complete a take-home challenge. While the exercise might take a current engineer at the company 2 hours to complete, that doesn't consider several factors: (1) a candidate might not be familiar with all technologies requested, and can easily lose time to research and learning; (2) the candidate wants to look good, so they're likely to work longer than you expect; and (3) the candidate might be interviewing for several companies at once, and have multiple competing assignments to work on.

Many companies use take-home challenges early in the hiring process to shift the burden of evaluation from the company on to the candidates themselves. This unfairly excludes lots of potentially amazing colleagues.

What we do instead

In addition to the above strategies not being fair, we've found that they measure things that are secondary to what we're looking for.

Artsy is more complicated than FizzBuzz. Too complicated for any one engineer to build, in fact. Individual engineers working alone can’t build the software Artsy needs to succeed – they must work together. So the skills we evaluate for are things like empathy, communication, and kindness. Not that technical skills aren’t important, but the ability to communicate and learn is more important.

Engineers who excel at empathy, communication, and kindness can pick up the technical stuff once they're hired; personal and interpersonal skills are harder to teach. Adding a colleague to the team who lacks these skills could harm the culture we've built.

When you interview with Artsy as an engineer, you won't just meet other engineers and a manager. You'll meet with people from other departments too. If you're hired as an Artsy engineer, you're going to work with folks from all across the company - we want to make sure you can communicate with them because that's something we do every day.

But we still evaluate technical aptitude

Technical aptitude is less important to us than interpersonal skills, but it is still important. Note that we said "aptitude," not "skills": we don't expect our engineers to already know everything about the tech stack we're using. Instead, we expect them to have a strong ability to learn our stack and use it effectively once they have. (This is touched on in our docs on what we look for in junior engineers.)

So if we skip all the usual tactics for evaluating technical aptitude, how do we do it? By talking to people.

We learn a lot about candidates in their interviews. We'll have a conversation with them about technology. Instead of white-boarding, we ask them to describe what they like about their favorite library, or what they wish they could change. We ask them to describe some legacy code they’ve worked with, and ask them how they think it got that way. We’re looking for a mix of technical skills as well as empathy and an ability to communicate nuanced ideas.

References are important to us

We also learn a lot through reference checks. Our reference checks aren't simply validation of your employment history - they are a 30 minute-long conversation with each of your three references that go into detail about your work history and career growth. It's quite an in-depth conversation, with questions structured to dig into specifics about the candidate's behavior.

An Artsy reference call might include the following structured questions:

In your capacity as [relationship to the candidate], how many people have you worked with in the candidate's role?

Okay, in just terms of job performance, how you rank the candidate out of that [X] many people?

Okay, finally, what's the difference between [the candidate's rank] and number one? How would the candidate need to grow to get to number one?

The first question establishes the context for the reference. The second question primes the reference to use that context when answering the next question. The third question is what we're actually interested in. These aren't easy or comfortable questions, but they give us an insight into the candidate's career, history, and areas to grow.

Fully half of our decision to make an offer or not is based on our reference checks. Artsy Engineering candidates go through the same reference check process as anyone applying for a job at Artsy, with Engineers sitting in on the call with Artsy's hiring staff.

But seriously, we really care about the personal side

We also make sure every interview ends amicably. No candidate should feel bad after interviewing with Artsy, even if we don't give them an offer. This seems self-evident to us, given our values, but it makes a lot of business sense to maintain our reputation as an engineering team.

Our hiring practice philosophy

One of our core values at Artsy is that People Are Paramount. We like to think that our interview process was built to reflect this.

We see the interview process as an opportunity to build a relationship with a candidate. We talk to them to find out if they're a good fit for Artsy, and we help them decide if Artsy is a good fit for them. Our hiring process focuses more on human skills than most processes do. It's not perfect, but it has served us well.

Our hiring process will never be "finished" because we're always improving on it. Some recent improvements are inward-facing to help us get better, like:

  • Starting a #dev-ersity Slack channel for talking about how to diversify our team and the industry at large.
  • Integrating hiring updates into our weekly standup.
  • Creating a Slack bot for engineers to monitor our hiring pipeline.
  • Periodically rotating hiring managers to spread institutional knowledge and get new perspectives.
  • Many, many docs written on guiding the process.

Artsy engineers, guided by our company values, created the hiring process for new engineers. Combined with an iterative process and a desire to constantly improve, we've created a hiring process that is fair, effective, and respectful. This kind of engineering-led approach is gaining popularity; for example, Microsoft recently revamped its hiring process with this approach.

We hope this catches on.

So what can you do? A great first step is to send this post to your HR rep. Another great step is to open source your hiring documentation; you'd be surprised how motivating this can be, and it's a great opportunity to get feedback from other companies. Leave a comment below, let's brainstorm on other ways to improve the state of hiring in software engineering!

And remember: while you might be motivated based on what feels "right", businesses are motivated by bottom lines. Fortunately for us, the evidence is on our side: this is a better way to hire, for everyone.