It was 95 days ago, and I was sitting before of my computer in NYC. I loaded my terminal, opened TapTalk on my phone, tapped on my colleague Ash’s avatar and held my finger there. I paused, this moment was a long time, it was worth waiting. I switched the camera from my face to the screen with the terminal open.

This moment was probably the most nervous I had been in years. It showed git push origin master. I said “it’s happening.” I hit return. A new era in the Artsy mobile team had started. A few minutes later, I wrote this tweet:

Let’s go over the process we went through to get to that point.

Read on →

Obtaining, reissuing and installing SSL certificates across various pieces of Artsy infrastructure (read about our tech stack here) is a mind-numbing excercise dreaded by every platform engineer. Few people really understand what’s going on, installing SSL certificates is right up there in the opaque world of user-unfriendliness, along with onboarding a new iOS developer in an organization with Xcode.

Still, it can be done - below is our cookbook.

Read on →

We recently picked up a Rails application that was a few features away from completion. This application allows our Genome Team to classify multiple artworks based on visual and art historical characteristics. These characteristics, or “genes”, can be added, removed, and changed for any of the artworks on the panel.

Our genomers are masters of efficiency, and over the years we have worked closely with them to tailor a dynamic interface to their specific workflow.

When we started working on the app, the back-end was organized, modular, and interfaced seamlessly with the Artsy API, but there were still a few front-end features we needed to implement before it could be used by the Genome Team. The app did not use a front-end framework, and as our features scaled it was difficult to keep track of UI state with pure CoffeeScript and global event listeners. Eventually, we decided to stop trying to patch our leaky roof and build a new one.

Read on →

Fair compensation is an important part of cultivating and growing engineers, and determining how much money someone is going to make is often more art than science, especially in young start-ups. We wanted this to be a collaborative process and not an adversarial one, and as our engineering team grew past 20 members it became important to think about compensation more systematically.

In this post I will share our individual contributor engineering ladder and our performance evaluation and compensation process.

Read on →

Recently, the Artsy Mobile team had the pleasure of working together on an article for objc.io, an online publication dedicated to in-depth articles about iOS and OS X development. Each monthly issue consists of several articles centered around a theme—past issues include Testing, Animations and Swift. objc.io’s latest issue, iOS at Scale, features articles by a handful of developers about how they approach iOS development at their respective companies. Our contribution to the issue is a collaboratively-written article by and about Artsy’s Mobile team from the angle of Artsy’s focus on the core value of openness that is foundational to how we (and the greater Artsy team) work.

As our article puts it:

Our team intrinsically believes in the ideas of open source. Though our individual motivations vary — from a conviction in the power of education to a determination to help others see from giants’ shoulders — we all maintain the value of open source.

iOS at Artsy is characterized by a culture of open source stewardship. Although our team is small, our contributions to the Cocoa developer community are significant. We hope our article sheds some light on our development and testing practices while also providing a high-level look into our team’s philosophy. If you’d like to learn more, feel free to reach out to us at mobile@artsy.net or @ArtsyOpenSource.

Artsy has now grown past 100 team members and our Engineering organization is now 20 strong. For a brief overview of what the company has accomplished in the last two years, check out our 2013 and 2014 reviews.

This is a good opportunity to describe our updated technology stack. Last time we did this was when Artsy launched publicly in 2012.

Three years ago Artsy was a classic Ruby-on-Rails monolith with a handful of adjacent processes and tools. We’ve since broken it up into many independent services, and continue to heavily be a Ruby and JavaScript shop, using Rails where appropriate, with native code on mobile devices and some JVM-based experiments in micro-services.

Read on →

We built CocoaPods-Keys as a way to remove API keys and secrets from our source code. We use it to open up our code to the public, without leaking private details. One of the cool things about doing it this way was that we could all use different API keys stashed away in each developers Keychain.

To ensure we could run CI on our apps we came up with two different ways to use keys on CI. This post explains them both.

Read on →

The website is finally back up after crashing hard for 4 hours straight.

Recently AWS decided to reboot a few of your servers for a critical update. It didn’t seem like it was going to be a big deal, except that the schedule was only accommodating if you were in the Pacific Northwest. The first reboot took out a secondary replica of our MongoDB database. Unfortunately the driver handled that poorly and spent the first 400ms of every subsequent HTTP request trying to reconnect to the missing instance. That server came back up, but failed to find its storage volumes because of a human mistake in a past migration and the alerts were mistakenly silenced by someone monitoring the system. A few hours later the primary was being stepped down and rebooted, sending the driver into panic over another bug. The site went down.

None of this was obvious while it was happening as the rate of automated alerts grew. Engineers communicated to the team that they are actively focusing on bringing the systems back up. This helped to fend off a large amount of instant messages, e-mails, texts and phone calls from various people on the team that were in the middle of demoing something to a very important prospective customer on the other side of the planet. It was also the middle of the night in New York.

Now that all the systems are back up, lets write a detailed outage post-mortem.

Read on →