Artsy Engineering

Artsy Mobile 2025 Wrapped

Fri, 09 Jan 2026 00:00:00 +0000

The past year has been an exciting one for our mobile apps (Artsy, Folio and Palette Mobile) and we started to finally get closer to where we want to be: High-performing and developer-friendly React Native applications, enabling rapid feature iteration and a superior user experience.

We would like to share with you some of the improvements we made that might also be applicable to your react-native app.

Infra

Expo!

At the end of 2024, after RFC: Trial Expo in Energy and/or Palette-Mobile, Spike on Eigen Risks, Rewards and Effort, we decided that we are giving Expo a try. Quickly afterwards, we added Expo with Prebuild on 2 out of 3 of our Apps: The CMS App, named Folio and our design system app, named Palette-Mobile.

For our main app, Eigen, adding expo sdk happened faster than we thought after Microsoft decided to retire VS App Center. We needed an alternative for code-push, and we settled for Expo over-the-air updates.

Eigen still has a lot of native code blocking us from fully migrating to Continue Native Generation (CNG) and sometimes conflicting with Expo. However, our experience from Energy and Palette-mobile has been positive so far and we could imagine CNG in Eigen! We will definitely share more about this if it happens.

The new architecture

Yes! We did it, all our apps are on the new architecture now. Expect a blog post about this!

Tech Debt

Eigen is an old repo, it’s probably one of the oldest react-native apps out there. This comes with a price though. The industry changed a lot in the past years and a lot of the legacy code can now be rewritten using modern patterns more easily (and efficiently).

To address that, we identified some weaknesses we had and prioritised them and decided to address them. Some notable mentions here include:

Not long ago, it was hard for us to imagine our navigation infra all handled in react-navigation. But with more and more screens being rewritten in react-native and RN becoming more performant, it actually finally became possible! And we were so happy to do it. This refactor helped us make it easier to add routes, reduced our navigate helper method latency from 320ms to 70ms, and most importantly consolidated our Android and iOS navigation infra.

Our city guide is now fully built with React Native - and is available for Android

The city guide is an old RN component with lots of logic we had in the native side. Although we rarely had issues with it, it was pretty tricky to make any changes there. This migration makes it easier for our product teams to extend the city guide without sacrificing UX.

Refactor our push notifications

Our push notifications handling logic differed between iOS and Android, making it tricky to keep the same logic across both platforms because you would need to implement everything twice - especially tracking.

While migrating to the new architecture, we were able to consolidate both platforms thanks to Notifee - PR here.

Better Keyboard handling

This is a favourite for me because proper keyboard handling in mobile apps is an important factor in mobile apps UX, and it was never easy. Until react-native-keyboard-controller came to the rescue!

If you haven’t already, I recommend you check out this talk from the author Kiryl Ziusko on youtube: https://www.youtube.com/watch?v=W_Y0Y18aFV8

DX

Rozenite

You hear a lot of good stuff about RN, but debugging was never a strong point until the last few versions when the new RN debugger was announced over at React Universe Conf 2024. It was something everyone in the community welcomed and it enabled other additions like Rozenite, that helped us do a lot more from the debugger and bring us “closer” to the web debug experience.

So far, we have plugins to debug navigation events, to inspect our bundle size, networks and monitor performance. See PRs here and here and watch the announcement talk from this year’s React Universe Conf

Mise

This is not specific to our mobile apps, but it’s something we did in all our repos after: RFC: Migrate from asdf to mise If you haven’t tried it already, you are missing out on a lot. It just works, no drama!

Yarn doctor and Yarn repair

The most pain we had over the years in our major repo (Eigen), is when devs who don’t work often in the repo get back to it. Quite often, dependency drift happened and fixing the environment isn’t trivial. It’s why we added these commands to help with some of the failures.

Better betas overview

One annoyance we always had when working on branches that require creating multiple betas, is getting the build number. Since we moved our build process to happen on Github Actions, it was only natural to us that while we are at it, to comment the build number on Github.

Performance

Last year, we did a lot of work to improve our mobile vitals. The most clear one was TTI. That was so fun to work on, and we think the changes we brought can be applied anywhere.

Queries optimization

What we did was pretty simple:

Fetch only what you need (minimal payload)
Fetch before you need it (speculative prefetching)
Fetch things separately when possible (query splitting)

Example:

Let’s think of how we can optimise the fair screen query: This is a query that fetches the fair metadata, overview, exhibitors, and artworks.

Define a query that fetches only the artist image and metadata
Prefetch that query whenever there is an artist visible - before the user taps it!
Define the other tabs queries and apply the same concept to the tabs queries

Example: To render the exhibitors tab, we don’t need just one query. In our case, because getting the artwork for each exhibitor was taking a long time, we had multiple queries, one to get the exhibitor metadata, and another query to get the artworks, that is lazily loaded when the exhibitor rail is close to the viewport

Bonus: Because the first query result is the same to all users (the artist name and image is always the same) - we can actually cache that query in our Cloudflare CDN and return it immediately.

We applied the same concept to most of our major screens and in many cases, we were able to bring the TTI to under 100ms thanks to our Cloudflare CDN cache hits, making the navigation in the app feel seamless.

FPS optimizations

Here, the tips from Callstack’s Master React Native Performance Optimization were very useful to us and led to noticeable improvements in multiple surfaces.

However, in some cases, they were not enough and we had to use some of the native debugging tools to investigate high CPU usage causing frame drops. What helped us was Xcode View Hierarchy, which revealed that we were always loading skeletons behind our images, even after the image was loaded.

CI

Migrating to Github Actions

Thanks to being an open source project, Eigen was eligible to use Github Actions for free! Thanks to having modular scripts, and Github Actions and Circle CI having a more or less similar language/concept, this was a trivial migration for us that saved us a good amount of CircleCI tokens.

Reduce Android build time by 75%

Sometimes, you can make a major improvement with one line of code. This was the case here. Thanks to Speeding up your Build phase we were able to reduce our build time by 75%. The idea is simple here, instead of building all 4 ABIs (armeabi-v7a, arm64-v8a, x86 & x86_64), we build only the one that is relevant for us: arm64-v8a

E2E Tests

Reliable E2E testing in react-native is tricky. Over the years, we tried multiple tools but we never added them to our CI pipeline. 2025 was different, we added E2E test coverage to some of our critical user flows to facilitate QA. We settled for Maestro and we are looking forward to expanding our coverage in 2026.

On Android, along with Flashlight, this enabled us to also run performance tests, which were very useful when upgrading to the new architecture.

Other mentions

We migrated to a more secure secrets management library - react-native-keys
Significant Android quality increase - shout out to mobile practice and deciding to prefer Android screenshots and videos in PRs - we started the year with a 28-day average rating of 4.1 stars, ended with 4.8 stars and a default Google Play rating of 4.4 stars.
Established Performance SLAs (crash rate, query latency…)

What’s next?

We’re excited for what’s ahead in 2026! The React Native ecosystem continues to mature, and we’re looking forward to more stable releases that’ll make our lives easier 🤞

One area we’re particularly excited about is CNG and improving our developer experience with AI tooling. We’ve started experimenting with Claude and leveraging Skills to make AI assistants more effective in our repos. The idea is simple: instead of the AI having to figure out our conventions every time, we can document our patterns, architecture decisions, and common tasks in a way that makes collaboration seamless. Early results have been promising, and we’re excited to see how this evolves.

If you’re interested in any of the topics we covered here, feel free to reach out! We will be happy to share more details about it.

Two years of Next.js at Artsy: A Retrospective

Thu, 07 Mar 2024 00:00:00 +0000

Where to begin! Where to begin…

Lets start with a good measure of success, one that I think most engineers could agree on: a from-scratch rebuild of a complex, internal tools app. Success with rebuilds is often fleeting, as many can attest. There’s risk involved. Extending the challenge further, lets assign (most) of the task to a team of platform engineers who aren’t too familiar with modern front-end technology – and then say that we succeeded.

Is Artsy crazy? Or is Next.js (along with our team 🙂) just that good? Well, it’s a little of both. And it’s a little bittersweet, in a way. Here’s our tale.

Beginnings

Our first formal use of Next came from a project spun up during Hackathon, two years ago. Hackathon’s are great opportunities to push technology forward, and Artsy has had a lot of success with them. We’d been talking about Next.js for a while, and experimenting here and there, but we were never able to find the proper intersection of product and purpose to take things further. With this in mind, one of our Engineers (Roop 👋) spun up a Next.js POC that allowed us to deduplicate artist records. On the surface, the effort was non-trivial. It involved authentication, DB communication, and more. Yet when presented during Hackathon, all of this was done and it also looked beautiful.

How could so much product work get completed in such a short period of time? A large part of that is due to Next.js’s framework design, and how it simply gets out of the way and allows one to start building product features. The defaults made sense; to create a new route, simply create a new folder or file in the pages directory, and Next takes care of the rest. And what about compilation, TypeScript, linting, and all of the other extraordinarily confusing JavaScript toolchain details that folks typically struggle over? (Most) of the setup was taken care of at the framework level, and the API side of things was light and easy to understand, thanks to great documentation.

There are omitted details of course, and there were things we needed to figure out (such as auth, via next-auth, and passing runtime ENV secrets to our Dockerized container), but in any case this gave us a lot of confidence to start discussing what it might take to seriously consider rebuilding our internal tools app. A few meetings later it was decided, and our platform team agreed to take it (with loose backup support from a couple other engineers). And a few short month’s later much of the core functionality was complete, executed by a team working amidst unfamiliar terrain.

That’s the definition of success, and the measure. With our internal tools app complete it was natural to start looking around for other opportunities – but first, a quick detour.

Enter: Next 13 (aka, The `app` Router)

Many of the limitations of Next 12 are well known, with the most notable ones being the inability to (naturally) do page layouts along with quirks around getServerSideProps and _app and _document request fetching not being as flexible as one would like. These issues, however, were not deal-breakers; and in fact, we hardly even noticed them. Our page layouts simply rendered a shared <Layout> component that accepted children; and per-route data fetching requested data per-route, with globally shared data going in _app. It got the job done.

With these things in mind (and a few things more), many of us were extremely excited when Next released its Layouts RFC, outlining the next version of Next.js, which would be built on top of another long-anticipated React.js feature, React Server Components.

In short, the Layouts RFC outlined what looked to be a beautifully obvious, yet performant architecture. Using Next’s preference for file-system based configuration, a typical “page” could soon look like this:

- app/
---- layout.tsx
---- page.tsx
- app/artist
---- layout.tsx
---- page.tsx
---- middleware.tsx

And so on. “Global” SSR data could be fetched right there in layout or page and shared with its subtree; and likewise, for sub-sub-tree’s we could do the same in each individual app layout or page component.

Additionally, with React Server Components, we would no longer need to use many Next.js-specific APIs. To fetch data on the server, you simply await a promise and pass it right to your component as props. Next’s already minimal API footprint would diminish even further, and it became possible to glean a vision of “just vanilla JS” all the way down, and within that vision the possibility of true simplicity.

Little did we know, it wouldn’t take much to turn this beautiful vision into something of a dilemma, but that part of the story comes a bit later.

Expanding Next.js at Artsy

Coming off of our success with the internal tools app rebuild, we wanted more. And we didn’t need to look far: right around the corner was a ~10 year old external CMS app that our partners use to manage their inventory.

We decided the path forward was Next, and like our internal tools app rebuild, for the most part it has been a success. We again went with the pages router (as the new app router wasn’t yet released) and so far there’s been minimal confusion from the team. And buisness-wise, its been refreshing to defer framework design decisions, lib upgrades and more to Next, versus having to maintain all of these things in-house.

It’s also worth mentioning that there have been a few significant challenges involved (such as setting up performant SSR patterns for using Relay, our GraphQL client – thats another blog post), but on the whole Next has served our needs well. Team performance was unlocked, and we’ve been able to quickly get to building and rebuilding CMS pages in this new application. And our engineers have loved working in it.

Back to Next 13

In the meantime, Next 13 was released. Imagine our excitement! Just as this new app is spinning up we receive a little gift from the stars. Carlos and I are the first ones to bite; lets see what migrating our work over to the new framework might look like, what kind of effort.

From the start, it was immediately obvious that the Next.js team released an alpha-quality (or less) product, marked as stable. Not a beta, not an RC to peruse and experiment with, but rather an

npm install next

package that comes with an application scaffold generator that suggests using the app router over pages – marked as “Recommended”. In other words, highly polished. And what’s the first thing one experiences? Hot Reloading is broken.

And what’s the next thing? Styles are broken. It turns out that RSC (React Server Components) doesn’t fully support pre-existing CSS-in-JS patterns. Or rather, they do, kind of, but they can only be used inside use client components (which, in Next.js, actually means a server-side rendered component environment that’s separate from a RSC rendered “server-only” environment – aka the old pages router model). And we certainly weren’t about to throw out our Design System component library Palette, which has been nothing but a runaway success (and a highly portable one at that).

With this limitation, our ability to use React Server Components had been severely hampered. Excluding the root-most level of our component tree, we were now required to prepend use client on the top of every component, lest we receive ambiguous errors about rendering a client component (which used to be server-side render safe) on the “RSC server”.

Things can be taught, however. So lets proceed from the assumption that through some kind of tooling / linting layer, use client is added to every new component. It should behave at that point just like the old Next and now we get the best of both worlds. Nope: turns out that even with the CSS-in-JS setup instructions described in the the next docs above, we still run into issues. There are bugs.

(These are the two main red flags, but there are many others as well.)

At this point, we wisely back out. It’s only next@13.0.0, and what they’re doing here is to a certain extent revolutionary. It’s a new way of thinking about React, yet an old way of thinking about page rendering. It’s like… PHP, or so they say. RSC is interesting, there’s something to it. Lets give them the benefit of the doubt and return to things in a few months, after a few minor version bumps; there are, after all, countless eyes on the project.

Many Months Later

We run npx create-next-app@latest (this is around the time they release 13.4) and then add these two components inside the newly-created vanilla project:

// app/HelloClient.tsx
'use client'

export const HelloClient = () => {
  return <div>Does this hot reload</div>
}

// app/layout.tsx
import { HelloClient } from './HelloClient'

export default Layout() {
  return <HelloClient />
}

Everything renders. And then

export const HelloClient = () => {
  return <div>Does this hot reload... nope :(</div>
}

In the most basic project setup, the most obvious Next.js selling point – Developer Experience – failed to deliver. Vercel is really forcing us to question things. But we’re flexible, and we like to investigate at Artsy, so even though this definitely-required feature doesn’t quite work, maybe it will once we’re done with our migration spike, and maybe we can still take advantage of everything else that RSC has to offer.

So again, we start refactoring the project. Stuff from the pages directory starts getting copied over to app. We update configuration. We setup styling (it seems to work better). Things are almost there. But then the obscure framework errors start to arrive, and CSS still doesn’t quite work: it turns out that refactoring across RSC-use client boundaries is harder than one thought. I.e., if any piece of “client” (remember, ‘use client’ actually means SSR-safe) code anywhere in the dependency tree happens to intersect an RSC boundary, the whole thing will fail. And this includes any use of React’s createContext – because React Contexts aren’t supported. Given an app of any reasonable size, you’re likely to rely on a context somewhere, as contexts are so critical within the react hooks model of behavior. Said contexts might come from within your app, and if not there they’ll certainly come from a 3rd party library.

One would expect the errors to be helpful in tracking this down – Next.js is all about DX – but no. Confusion reigns.

We’re experts though, and we eventually do find the source of the violation, and we make sure to create a “safety wrapper” around the offender so that it doesn’t happen again. But it does happens again – and again, any time any piece of any complexity is added in the new RSC-intersected route. It’s rather unavoidable. And each time solvable, but at a great cost to the developer. Thankfully we know what we’re doing!

Another trivial yet annoying issue (thankfully fixed with some custom eslint config) is accidentally importing the useRouter hook from the app router location, or redirect, or any number of other new app router features, because all of these things don’t work in pages, and will error out. The errors here are slightly less opaque, but what if you’re a backend dev who knows nothing about any of this? Googling “useRouter next” now yields two sets of docs. Figure it out.

At this point, we make a judgement call: this simply isn’t going to work at Artsy. We’re here to empower folks and unlock productivity. Remember the team of DevOps engineers on Platform who rebuilt a CMS in record time? In the new Next 13 model, that would be unfathomable, impossible even. Paper cuts would kill motivation, and dishearten the already skeptical. And the front end already has a bad rap, for good reason: historically, everything that seems like it should be easy is hard and confusing for those who aren’t experts. And everything is always changing. And the tooling is always breaking. And everybody always has a bright new idea, one that will finally end this madness for good.

A certain amount of sadness is appropriate here, because Next’s pages router was very, very close to being the silver bullet for web applications that we’ve all been looking for. Even though the pages router has its flaws, it showed us that it’s possible to get something out the door very quickly with little prior knowledge of Front End development. This is no small thing. Next 13’s fatal error is that its destiny, being coupled to RSC, now requires experts. And by ‘expert’ I mean: those with many years of experience dealing with JavaScript’s whims, its complex eco-system, its changeover, as well as its problems. In short, folks who have become numb to it all. This is no way to work. Thankfully the community is finally responding.

A Quick Note on Performance

It’s worth remembering that Next 12 was industry-leading in terms of performance and pioneered many innovative solutions. Let me say it again: Next’s pages router IS fast. Next 13 combined with RSC is faster, but at what point does an obsession with performance start negating other crucial factors? What’s good for the 90%? And what’s required for the other 10%? Most companies just need something that’s fast enough – and easy enough – to move fast. And not much more.

Back At Artsy…

With all of this in mind, and with the uncertainty around long-term support for Next.js pages (amongst other things), we recently decided to hit pause on future development in our new external CMS app rebuild. Weighing a few different factors (many entirely unrelated to Next), including a team reorg that allowed us to look more closely (and fix) the DX in our old external CMS app, we took a step back and recognized that our needs are actually quite minimal:

Instant hot reloading
Fast, SPA-like UX performance
Simple, convention-based file organization

With these things covered, the “web framework” layer looks something like the following, minus a few underlying router lib details:

const Router = createRouter({
  history: 'browser',
  routes: [
    {
      path: '/foo',
      getComponent: React.lazy(() => import('./Foo'));
    },
    {
      path: '/bar',
      getComponent: React.lazy(() => import('./Bar'));
    }
  ]
})

We’re now required to manage our compiler config, but that layer isn’t too complicated once its setup, and it works great. (If you’re using something like Vite, it could be even simpler.)

Final Thoughts

Next 13 and React Server Components are very intriguing; it’s a new model of thinking that folks are still trying to work out. Like other revolutionary technologies released by Meta, sometimes it takes a few years to catch on, and maybe RSC is firmly in that bucket. Vercel, however, would do well to remember Next’s original fundamental insight – that all good things follow from developer experience. Improvements there tend to improve things everywhere.

In addition to fixing some of the obvious rough edges in the new app router, it would be helpful if Next could provide an official response to the question of long-term Pages support. There’s been quite a backlash in the community against Next 13, and that should give all developers pause. It’d also be great to get word on whether there will be any further development on the pages router – perhaps some of the features from the new app router can be migrated to pages as well? – or if the pages router is officially deprecated and locked. All of this is currently ambiguous.

Another area where Next could improve is their willingness to ship buggy features, and to rely on patched versions of React in order to achieve certain ends. Even though Vercel employs many members of the React Core team, by releasing Next versions that rely on patched and augmented canary builds of React, Vercel is effectively compromising some of React’s integrity, and forcing their hand. Once a neo-React feature is added to Next, it makes it hard to say no; Next has captured too much of the market-share.

All of this calls for sobriety and hesitation on the part of developers working with – and building companies on top of – Vercel’s products. Next is Open Source, yes, but it’s also a wildcard. Artsy has had some real success with Next, but sometimes that’s just not enough to avoid hitting pause, and looking at the bigger picture. Inclusivity should always win.

The Checklist for Deploying a Scary Change

Wed, 13 Sep 2023 00:00:00 +0000

Lately, I’ve been getting involved with some sketchy stuff. You know what I’m talking about–data migrations.

I’ve been rolling out changes that have a significant risk of breaking our production environment for mission-critical services. It’s been exciting work (keep your eyes out for more posts on the exact project, coming soon™️), but I’ve definitely caused a couple incidents along the way.

After accidentally taking down a key service for a couple hours, I decided I needed to have a better pre-deploy process for these changes. I did some thinking and came up with a short checklist to run through before I press the shiny green button.

Here’s the checklist I came up with:

With this checklist in hand, I’m deploying more confidently and causing fewer incidents along the way.

Do you have something similar? Are there things you think this checklist should include? Let me know in the comments!

TypeScript magic

Wed, 01 Mar 2023 00:00:00 +0000

At Artsy, we love TypeScript. We use it in most of our node/web/mobile repos. Today, I want to talk about a specific case we found while trying to make our types more strict on palette-mobile, which is our Design System for React Native.

Check this out:

const welp: "hello" | "world" | string // `welp` is of type `string`.

Like the comment says, even though we have two specific strings, the fact that we do a union with string, makes welp have a type of just string. This is because both "hello" and "world" are strings, and the union tends to go to the type that includes the most.

Think of set theory and bubbles.

"hello" is a type by itself, and "world" is a type by itself. Unioning them together gives us a new type, which is a bubble that contains both "hello" and "world". In that "hello" | "world" union bubble, we see both "hello" and "world" types as subsets.

The string bubble contains all strings, so it contains "hello" and "world" and "hello" | "world", so the union of them with string is string.

That is usually ok, but for our case, it didn’t work. Here is what we wanted to do.

The problem

In our Design System, we have certain color, named like black100, black80, blue100, red150 etc. We can have a type like

type ColorDSValue = "black100" | "black80" | "blue100" | "red150" // | etc

and that works great. We get to have autocomplete, typechecking, all the good stuff that TypeScript brings.

But we also want to support any other string, like "#000000", "#000", "rgb(0,0,0)", "rgba(0,0,0,0.5)", "hsl(0,0%,0%)", "hsla(0,0%,0%,0.5)". Ok, you might say, just make more types like

type ColorHexValue = `#${string}`
type ColorRGBValue = `rgb(${number},${number},${number})`
type ColorRGBAValue = `rgba(${number},${number},${number},${number})`
type ColorHSLValue = `hsl(${number},${number}%,${number}%)`

and so on. That’s great. So far, so good.

We also want to make sure CSS color names are accepted. So then we add something like

type ColorCSSString = "red" | "blue" | "hotpink" // | etc

and now we have a type with all the values. That seemed ok, but it also felt a bit too much. If CSS names change, we need to update. Also what we wanted to do is actually have autocomplete and typechecking for our DS values, and just leave it loose for all the rest.

So we tried

type ColorDSValue = "black100" | "black80" | "blue100" | "red150" // | etc
type ColorOtherString = string

type Color = ColorDSValue | ColorOtherString

but we ended up with Color being just string, which automatically means no autocomplete and no typechecking.

Now check this out!

const wow: ("hello" | "world") | (string & {}) // `wow` is of type `"hello"` or `"world"` or `string`.

This weird-looking intersection of string & {} makes it so that the specific strings "hello" and "world" are distinguished from string as a whole type.

The way this works is this:

the intersection of string and {} (which is string & {}), is essentially the same as string, but it is a new type, different from string.
the union of "hello" and "world" is "hello" | "world", which is a new type, different from "hello" and "world". It contains both.
the union of "hello" | "world" and string expands the type to string, since that is the common type. "hello", "world", and "hello" | "world", all inherit from string.
the union of "hello" | "world" and string & {} is "hello" | "world" | (string & {}), which is a new type, different from just string. This is because "hello" and "world" DO NOT inherit from string & {}, so they are distinguished from string & {} as a whole type.

With this type trick, essentially we can tell the type system that we want specific string, but also any other string.

Here is a complete view of the sets.

It seems pretty funky that string and string & {} are same in a way, but different in another way. They both tell the type system that any string is accepted. But one is inherited by every type that is a string (like type Hi = "hello"), where as the other is not inherited, so they are distinguished from each other.

That is so cool to me! I wanted to do this and didn’t even have the words to describe it, I didn’t know how to google it. We kind of found it accidentally.

This is so so useful for types or props where you want the general type for support (string), but you also want the specific type for autocomplete ("black100"). It made my whole week when I figured that out and made that color type.

Here is the final type:

type ColorDSValue = "black100" | "black80" | "blue100" | "red150" // | etc
type ColorOtherString = string & {}

type Color = ColorDSValue | ColorOtherString

Now we have autocomplete and typechecking.

Final thoughts

This is such a useful little TypeScript trick. Thanks to Sultan for finding this. He found it in a TypeScript issue. Then we tried it and figured out how to work with this, and how to make our type exactly what we wanted, for the best DX we can get.

Link to palette-mobile, where we use this type: link

Link to a TypeScript playground with the examples: link

Are you using the right Mongo geospatial query?

Fri, 10 Feb 2023 00:00:00 +0000

We recently got a report from one of our galleries in the Los Angeles area that they weren’t showing up on our Los Angeles exhibition listings.

I fielded the report and right away confirmed: when we asked our core API for /shows?near=<los angeles coordinates>, sure enough this gallery partner didn’t make the cut.

Turns out they are based in Santa Monica, a separate and neighboring municipality. They must not be within the 25km radius that we use by default for these sorts of queries.

Case closed. Or so I thought.

After some back and forth with our partner I decided to investigate more thoroughly, this time using some tricks of the trade from my other life outside of Artsy.

Casting a wider net

If there was something wrong with our 25km radius query, I wanted to start by casting a wider net and visualizing the results.

I consulted our Rails application code to find the query logic in question, and then issued the same query directly to MongoDB. Something like the following query (simplified for clarity):

// a $geoWithin $center query

db.events.find({
  coordinates: {
    $geoWithin: {
      $center: [[-118.24, 34.05], 25 / 111.32],
    },
  },
})

In the query above we are asking MongoDB to give us all events within a 25km radius around the point 34.05°N, 118.24°W which we have designated as a central point within Los Angeles. For our purposes in this post we can consider “events” ≈ “exhibitions” ≈ “shows.”

We are not asking for the 25km radius directly, but rather converting it into an equivalent amount of geographic degrees by using a conversion factor of 1° ≈ 111.32 kilometers, a factor which is true at enough the equator.

I modified the above query to cast a 50km net in order to see if there were some edges cases that needed scrutiny. Taking the resulting JSON response, I fired up Placemark, my favorite new tool for wrangling geospatial data.

(Incidentally I recommend reading Tom Macwright’s recent reflection on creating Placemark as a bootstrapped indie developer.)

All shows within a 50km radius

Ball of confusion

One nice feature of Placemark is that it lets us place geodesic circles on the map, that is, circles that represent a constant radius around a point, as plotted on a globe.

When I placed a 25km radius circle on the map, something stood out immediately.

25km radius superimposed. Ball of Confusion, that's what the world was that day.

The partner in question is highlighted in pink — and is clearly within the 25km radius. What gives?

By spot-checking a few points on the map against our current geo query I found that edge cases near the top or bottom of the circle were likely to be evaluated correctly, while edges cases at the left and right were being incorrectly omitted, as our partner gallery was.

A fuller visualization of that finding would look like this:

Results of a $geoWithin $center query around Los Angeles, evaluated against a grid of test points.

A distinctly non-circular circle — that rung a bell.

More than one way to draw a circle on the Earth

It was at this point that I recalled the specific form of the geospatial query our code was performing, and consulted the MongoDB docs for the $geoWithin query.

Turns out that you can invoke this as a radius query in one of two ways, by specifying $center or $centerSphere.

Per the docs for $center, this query…

calculates distances using flat (planar) geometry

Let us pause for a moment to note that while only some maps are deceitful, all maps are untruths. In the sense that they flatten three dimensions down to two, and inevitably distort the world in the process.

The surface of a three-dimensional globe cannot be flattened down to a two-dimensional plane without some stretching or tearing, any more than an orange peel can be. The mathematical algorithms for turning those three dimensions into two are what we know as map projections. (Ah, the good old days when “dimensionality reduction” meant from three to two.)

If you do your distance calculations in such a flattened, projected coordinate system — as the $geoWithin $center query does — then you are accepting whatever distortions are inherent to that projection.

That’s the situation we were in. We thought we were catching everything inside the green circle, but in fact we were only catching everything inside the red egg:

Actual $geoWithin $center results vs. expected results.

And our unfortunate partner was just outside the egg, thus being incorrectly omitted.

Fixing the query

Luckily the solution was simple.

As noted above MongoDB supports a second variant for radius queries using a $centerSphere operator instead of the $center that we were using.

Per the docs for $centerSphere, this version…

defines a circle for a geospatial query that uses spherical geometry

In other words, this query effectively draws our circle on the three-dimensional globe rather than on the stretched and distorted two-dimensional map.

We just need to rewrite our query as follows:

// a $geoWithin $centerSphere query

db.events.find({
  coordinates: {
    $geoWithin: {
      // $center: [[-118.24, 34.05], 25 / 111.32],    /* BEFORE */
      $centerSphere: [[-118.24, 34.05], 25 / 6378.1] /* AFTER */,
    },
  },
})

There is a new conversion factor in play here, this time denoting that the radius of the earth is approximately 6378.1 km. In this $centerSphere flavor of the query we are working in unprojected spherical coordinates, measured in radians. Thus we need to account for the size of the sphere that we are calculating upon.

We can re-run our test with this version of the query, and now we see that the results are finally in line with what we were expecting:

Results of a $geoWithin $centerSphere query around Los Angeles, evaluated against a grid of test points.

Having updated our API to use this variant of the query, we solved the problem and returned our partner gallery to its rightful place in our listings, as shown by the pink highlight above.

That was the happy ending we were looking for.

A postscript on map distortion

But if you’re curious to learn a little more about map distortion, let’s dig a bit deeper into the nature of the problem that we were encountering.

Returning to Placemark’s ability to draw different kinds of circles on the map, let’s now place a geographic circle on the map rather than a geodesic one. This one is computed in the simplest possible map projection — a geographic projection where we simply treat the longitude as the X coordinate and the latitude as the Y coordinate. (This projection goes by many names, such as “geographic”, “equirectangular”, “Plate Carrée” or even “unprojected”, which is not quite accurate.)

This corresponds to what you get when you use MongoDB’s $geoWithin $center query on geospatial data:

Animation showing the distortion a geographic circle encounters at various latitudes.

As you can see, we get: a moderately oblong egg at the latitude of Los Angeles; a nice circle as we get close to the equator; and a very oblong egg as we approach the poles.

If you are wondering why you should trust my claims about the egginess of this two-dimensional image after I just said that all such maps are lies — good question!

It just so happens that nearly all web-based interactive maps, including Placemark, use a variation of the Mercator projection, the one you might remember from schoolroom maps. Mercator is a so-called “conformal” projection, meaning that its particular lie is to sacrifice area in favor of shape.

A shape drawn on a globe will be correctly maintained in a Mercator map, but the scale will vary across the map: true at the equator and very incorrect towards the poles. This is the reason for the common complaint that Mercator maps show Greenland as about the same size as Africa, when in fact Africa is about 14 times larger.

The amount and nature of the distortion introduced by map projections is such an important topic that cartographers have long relied on a clever technique for communicating this distortion visually, known as “Tissot’s indicatrix.”

To give you a sense of the kind of distortion we encountered with the $center query, here is what Tissot’s indicatrix looks like for the geographic projection. This shows essentially the inverse of the animation above — what does a true circle plotted on the globe look like at various locations on this map projection?

Tissot's indicatrix for equirectangular projection. Credit: Justin Kunimune, CC BY-SA 4.0, via Wikimedia Commons

Now, imagine the inverse operation of this ⤴︎, drawing true circles on this planar space, in order to get a sense of how distorted your query results on a globe would be.

Finally, here is Tissot’s indicatrix for the Mercator projection, demonstrating its ability to preserve shapes at the expense of sizes.

Tissot's indicatrix for Mercator projection. Credit: Eric Gaba, CC BY-SA 4.0, via Wikimedia Commons

An interesting recent development is that the web’s reliance on Mercator is changing, but only very slowly. Google began to make this change a few years ago, and Mapbox has written about their approach as well.

Hopefully this digression into the display of geospatial data has been illuminating. There is much more to say on this topic, since geospatial is more or less one asterisk after another. For example, we haven’t mentioned that the Mercator projection above is incapable of depicting the north or south poles at all! Nor have we touched on MongoDB’s various geospatial data formats, indexes, or the $near query and its spherical sibling the $nearSphere query, all worthy topics.

But we hope that understanding this crucial distinction between planar ($center) and spherical ($centerSphere) calculations will help you make the right choice when devising your own radius queries with MongoDB or other geospatial engines.

Parallelizing Jest and Cypress.io Tests on CircleCI

Wed, 07 Sep 2022 00:00:00 +0000

At Artsy, exploring ways to improve the developer experience is part of our makeup. Whether it’s implementing hot-swapping for Express.js or integrating the Rust-based SWC compiler into our front-end build pipeline, we’re always trying to reduce the amount of time it takes for a code cycle to take place. CI is no exception. When a developer opens a PR, we want to ensure they get timely feedback. Do their unit tests pass? Does the app build correctly? And how about smoke tests? Each of these jobs are complex processes that take time, and the more one can parallelize said tasks the less devs will need to wait. Scaled out to a whole engineering org, minor improvements to CI can be radical.

In this regard, two things came across our radar recently that we’d like to share: sharding via Jest, and a (free) way to parallelize Cypress.io integration tests via CircleCI’s split command.

Sharding in Jest

“What is sharding?” Good question! In short, it means “a small part of a whole”. The database community has employed sharding techniques for decades, where a large database is split up into smaller, more manageable chunks, usually to improve performance at scale. The same idea can be applied to any process or task involving a lot of data, including tests.

Think about it like this. Imagine an app that has thousands of tests. One can open up their terminal and run yarn test and execute all of the tests at once in a single process, or one can open two terminal tabs and run yarn test src/utils and yarn test src/routes, and have both processes allocate a pool of memory to complete each (smaller) subset of tasks. Because each process has its own memory pool the performance characteristics are generally better, and thus the overall time required to run our tests is reduced / decreased. Running each of these commands scoped to a particular folder is easy enough, but in a CI environment this is somewhat cumbersome; we’d need to define two new jobs and then the conditions in which they run, increasing the scope and complexity of our configuration file.

This is where Jest’s new sharding feature comes into play, which taps nicely into most modern CI runners. Using a hypothetical app containing 100 tests, here’s a quick example of how it works:

$ yarn jest --shard 1/5

What this says is: take the total number of tests (100), divide them into five buckets (containing 20 tests each), and execute the test runner against the first bucket (the first 20 tests). Continuing:

$ yarn jest --shard 2/5

Now take the second bucket and execute the next 20 tests – and so on. Simple enough.

Taking this further, we could turn this into a bash loop, including an & symbol to run things in parallel and automating some of the redundancy away:

BUCKETS=5

for i in {1..${BUCKETS}}
do
   yarn jest --shard $i/$BUCKETS &
done

For many the above snippet should be sufficient to speed up your test suite, but who wants to write bash loops? Thankfully, most modern CI task runners contain the ability to split jobs into separate processes programatically and so this kind of logic is unnecessary.

Here’s how to do this in Circle CI:

test:
  parallelism: 5
  steps:
    - run: yarn test --shard=$(expr $CIRCLE_NODE_INDEX + 1)/$CIRCLE_NODE_TOTAL

Set a parallelism value, and drop the jest command into a cool one-liner. The variable CIRCLE_NODE_INDEX refers to which container index the job is running on, and CIRCLE_NODE_TOTAL points to the value of parallelism.

On Artsy.net, we’ve been able to reduce the average time it takes to run our unit tests from around ~10 minutes per PR to just above 2m. A 4-5x performance improvement.

Parallelizing Cypress.io Integration Tests (For Free)

For those who want robust integration test coverage, Cypress.io has been a game-changer due to its reliability and ease of use. Here at Artsy we use it in a number of apps, most notably Integrity. One complaint, however, is just how slow it is. This is reasonable; Cypress is simulating a user browsing your website and sometimes a user needs to do x and y (such as logging in) before they can do z. At scale this can really slow things down and lead to bottlenecks, especially if deploys are dependent on all of your integration tests passing.

The Cypress.io team has recognized this bottleneck and released the Cypress Dashboard, a paid product which includes the ability to unlock parallelized tests on your CI. For those willing to pay for another SAAS product this will get the job done well, but for those with leaner budgets there’s another way to accomplish this for free, and on CircleCI it’s very easy to setup via the CircleCI CLI command split.

You can check out the full example here, but in short:

integration:
  parallelism: 5
  run: |
    TESTS=$(circleci tests glob "cypress/integration" | circleci tests split | paste -sd ',')
    cypress run --spec $TESTS

We use the circleci tests glob command to gather all of our tests, and then pipe that into the circleci tests split which will divide our tests into buckets, similar to how Jest’s --shard command works up above. We then assign that to a $TESTS variable and pass it into cypress run --spec $TESTS. CircleCI sees the parallelism prop in the config and automatically divides our tests into 5 separate containers, each running a small subset of our integration tests in parallel.

On Artsy.net, our smoke tests times have gone from around ~7m on average down to ~3m. A huge reduction for only a few lines of config!

Hacking Around Safari's 7-day Cookie Limit

Tue, 23 Aug 2022 00:00:00 +0000

Amongst the many, many things that organizations have to contend with around cookie consent laws is Apple’s very own browser, Safari. Did you know that Safari will only retain a client-side cookie for 7 days? This is in support of Apple’s Intelligent Tracking Prevention (ITP) feature, designed to protect a user’s privacy.

These privacy efforts are great but, in hand with laws like GDPR and CCPA, their rollout often creates a UX nightmare for users without some extra care. Here at Artsy, we’ve landed on a way to make things slightly less bad and want to share our approach.

Scenario: Imagine that as a EU resident you visit artsy.net for the first time. A banner appears asking you to Accept or Deny tracking cookies from our site. You don’t like tracking cookies, so you click the “Deny” button and the banner disappears. All good, right? Nope! You visit Artsy a week later and again, a banner appears asking you to choose your preferences. This happens again and again until you switch browsers and realize that what you were experiencing was Apple’s ITP feature in action. After choosing your preferences, the cookie we use to store them is erased after 7 days, necessitating another interaction.

We thrashed around in this vicious cycle for months until we found a simple, elegant solution thanks to a WebKit engineer’s prompt (during Apple’s open lab calls at WWDC – which you too can schedule!) She mentioned that the 7-day cookie limitation only applies to client-side cookies and that same-domain, secure, server-side cookies are not limited to these constraints.

This got us thinking. Our third-party cookie consent management service sets a client-side cookie, not a server-side cookie. Could we perhaps overwrite the client-side cookie with a server-side cookie of the same name and trick Safari into persisting the user preferences beyond the 7-day limit?

We gave it a try and… Yes. We. Can! And this means that you can too (and it’s also real easy to implement).

First, define an API endpoint server-side:

const app = express()

app.get("/set-tracking-preferences", (req, res) => {
  const { trackingPreferences } = req.query

  const cookieConfig = {
    maxAge: 10000000,
    httpOnly: false,
    secure: true, // important!
  }

  if (trackingPreferences !== "undefined") {
    // Overwrite client-side cookie with cloned, secure server-side version
    res.cookie("trackingPreferences", trackingPreferences, cookieConfig)
  }

  res.send("Tracking preferences set.")
})

Next, call the endpoint from the client when your app boots or when preferences have been set by the user:

import cookies from "cookies-js"

const App = () => {
  useEffect(() => {
    const trackingPreferences = encodeURIComponent(cookies.get("trackingPreferences"))

    fetch(
      `/set-tracking-preferences?trackingPreferences=${trackingPreferences}`
    )
  }, [])

  return <CookieConsentBanner onClick={...} />
}

And that’s it. I’m not sure of the exact reason why we can overwrite cookies in this way, but I suspect it has to do with Safari’s assumption that a server we control will always take precedence and thus assumes things are safe. And in 7 days, when Safari would otherwise erase the cookie, it will see that it’s now secure and ignore it, preserving the user’s preferences.

Third Time's the Charm: Deprecating KAWS

Mon, 09 May 2022 00:00:00 +0000

How do you tell when it’s time to deprecate a system? If something mostly works OK, is it worth spending time and effort replacing its functionality?

At Artsy, we realized several years ago that we needed to be able to group a bunch of artworks together. If we wanted to have a page with all of the ceramics by Lucio Fontana, or contemporary prints from the IFPDA Fair Sprint 2022, or a gift guide curated by Antwaun Sargent, we needed to have a way to make that happen.

We decided to call these things “collections,” a reasonable name for a collection of artworks. In order to create them, we developed a service called KAWS, named after the artist (whose works we wanted to put in several of these collections).

Now, 4 years later, we’ve taken down the service and folded its functionality into Artsy’s main database and API, Gravity.

Let’s talk about why and what we learned along the way.

A little context

KAWS has a somewhat unusual design. It’s a server, a Node.js app with its own Mongo database, and it serves up a GraphQL API. It also relies on TypeORM to map TypeScript models to database records and TypeGraphQL to keep TS and GraphQL types in sync.

This makes it a bit different from most services Artsy maintains. Most of our APIs are Ruby on Rails apps, and we don’t have any other uses of TypeORM and TypeGraphQL. When it was created, it was a cool experiment with a possible direction for future APIs - one we decided not to pursue for the time being.

KAWS doesn’t store any artworks - or even any artwork IDs. Instead, it just stores a “query”, a set of Elasticsearch criteria. This could be a list of artist IDs or gene IDs, a tag ID, a keyword, or any combination of those things.

In other words, a KAWS artwork collection could be defined in human-readable terms as something like “artworks with the ‘Black and White’ gene by the artist Bridget Riley” or “artworks by Auguste Rodin with the keyword ‘bronze’ and the gene ‘Sculpture’.”

This approach results in a somewhat odd flow for resolving collection requests. Let’s say a user requests the page KAWS: Toys. Our frontend, Force, sends a request to Metaphysics, our GraphQL API.

Metaphysics knows to route that request to KAWS - it asks KAWS “hey, what’s the Elasticsearch query for the collection with slug kaws-toys? Also, give me the title, the category, the header image, the description text, and [any other pieces of metadata about the collection itself].” But it can’t ask KAWS for the artworks!

Instead, it receives the query from KAWS and then turns around and sends another request to Gravity, which stores all of our artworks and indexes them in Elasticsearch so that we can filter them. “Hey Gravity, I got this query from KAWS (the service, not the artist): { artists: 'KAWS', genes: 'Sculpture' }. Can you return me all of the artworks that match that query?”

Gravity takes those parameters and passes them to its Elasticsearch cluster, fetching all artworks that match the criteria (and sorting/filtering them according to what the user has selected on the page).

Gravity returns those artworks to Metaphysics, which then packages them up with all of the collection metadata it received from KAWS and returns to the client that requested it. Whew!

Here’s a diagram showing what that looks like:

Why did we decide to deprecate it?

Idiosyncratic stack

As mentioned above, KAWS is an unusual app by Artsy standards. It’s neither a database-backed Rails/Ruby app, nor an API-consuming JavaScript/Node app. It was missing the typical Rails niceties (a dev console, background jobs, rake tasks, etc.), and inclusion of elements like TypeORM and TypeGraphQL that don’t exist in our other apps meant there was a bit of a learning curve for working with it.

While it’s not terrible to have some variety in our systems, it does make them harder to work in and keep up-to-date. And that also resulted in a high…

Lottery factor

There are still people at Artsy who’ve worked in KAWS, but the top 4 contributors have moved on to other companies. That meant that any projects involving KAWS required extra effort to familiarize devs with the system.

Lack of relationship between collections and artworks

Any KAWS-backed artwork grid (e.g. any /collection/:slug page) is the ephemeral result of a canned search query. This has some advantages: collections stay evergreen & current, so long as artworks with appropriate metadata continue to land on the platform. But it also has some downsides: there is no modeled relationship between a collection and its artworks, so given an artwork, you can’t tell what collections it’s in.

Extra hops on each request

Every /collection/:slug request has to go to → MP → Kaws→ MP → Gravity → ES. The diagram above illustrates this flow. And it’s worth noting that KAWS really doesn’t store that much. It’s a whole service that only stores a single data type.

Lack of admin UI

Collections in KAWS were first created via a CSV import, then eventually a Google Sheets-driven workflow was added. To create or update a collection, an admin creates/modifies a row in the spreadsheet, then goes to our Jenkins UI and kicks off an “update collections” job, which causes KAWS to pull the new data from the Google Sheet.

This is a pretty rough workflow, and it means it’s very easy to accidentally modify or delete information from a collection (e.g. overwriting the text in the wrong cell).

How did we go about planning it?

This actually wasn’t the first time we attempted to deprecate KAWS. It became something of a running joke: every October for 3 years running (2019, 2020, 2021), someone would say “hey we should probably deprecate this thing” and write a tech plan.

At least we're self-aware

This time, there were three key differences that allowed us to get it done:

There was a business need to do it. We want to work on improving collections, bringing new features to them and making it easier for our admins to manage and use collections for important marketing campaigns. We took a look at collections as they existed and said “deprecating KAWS and moving its functionality into Gravity is a prerequisite for these projects.” We could have eventually completed them, but it would have been slower and more difficult - we were confident that investing time into this refactor now would save a lot of time and headaches later.
We kept the plan very tightly scoped, focusing only on “maintain existing functionality and move it to Gravity.” Previous iterations of the tech plan had proposed broader changes, such as rethinking collections modeling entirely or breaking with our existing GraphQL schema. Any changes to fields currently being consumed by our frontend clients significantly increases scope, especially since our GraphQL schema is “baked in” to our app. That means that if we removed fields the app was currently relying on, some users who never upgrade their apps would see that functionality break.

To be clear, the authors of those tech plans weren’t wrong to propose rethinking and improving how collections are modeled and served! We may very well end up implementing some of their ideas and suggestions. The learning this time around was mostly “we’re never going to get this done unless we really rein in the scope.”

Our goal was 100% compatibility: all of the currently-used fields would still be available and could be successfully resolved by Gravity instead of KAWS. Even if we changed the type of those fields or updated how they were resolved, they still needed to exist. We did end up getting rid of lots of fields that were not being used by frontend clients (which meant a lot of spelunking through our code and trying to find out if fields like showOnEditorial were still being used), so the schema is still significantly simpler and cleaner than it was.
Roop laid a really nice foundation by writing up a thorough document about what KAWS is, how it works, and the steps we might need to take in order to deprecate it. Much of this blog post comes almost verbatim from that document! It’s a testament to the power that well-organized documentation can have: being able to see and understand the problem gave us confidence to tackle it.

Here’s the diagram he created that sketches out how we might go about deprecating KAWS:

What did we learn?

It’s easy for scope to expand without you noticing

I mentioned that our goal was JUST moving KAWS functionality to Gravity, and that we wanted to avoid adding new functionality to keep the project focused. We mostly did that - but we did get off track in one important way.

We’ve discussed how KAWS doesn’t include direct relationships between artworks and collections. As we worked on moving collections to Gravity, we had a baked-in assumption that there would be a join model between Artwork and MarketingCollection (the name of the model in Gravity). This was largely informed by the approach we took to moving ArtistSeries from KAWS to Gravity, a project we previously completed.

In that case, we created an explicit link between the ArtistSeries model and the Artwork model, and we updated those relationships with a daily recurring job. We would essentially run the query defined by the ArtistSeries, see if there were any new artworks that matched, and if so, add them to the Series in question.

When we tried this same approach for MarketingCollections, we noticed there was a problem: ArtistSeries cap out at a few hundred works, but MarketingCollections could be as many as several hundred thousand for collections like “Contemporary” that are based on very general criteria. The jobs to associate artworks with MarketingCollections were taking so long that they ate up all of our Sidekiq resources and prevented other jobs from being executed in a timely manner.

When we took a step back and looked at the problem, we realized that we were getting ahead of ourselves. Yes, we will likely want to directly associate artworks with collections - but there actually wasn’t a clear business reason for doing so. We were just doing it because we were “pretty sure we might need it at some point.”

So to unblock the project, we removed the join models and instead kept a similar approach to what KAWS used initially: fetching artworks at request time using the Elasticsearch query stored on the MarketingCollection.

There’s a reason we don’t name our projects after artists!

To quote a post about how we name our projects written by our Senior Director of Engineering, Joey:

Choose a code name scheme that isn’t directly related to your technology or business. A flower business using flower names is cute, but breaks down when you want to build a feature that actually is about tulips.

Most of our systems at Artsy are named after scientific concepts. Force, Eigen, Fresnel. This can seem confusing at first, but it means there are not many opportunities for mistaking a system with its data. With a project like this one named after an artist, discussions get confusing quickly! “Yeah, we need to make sure we’re fetching all of the artworks from KAWS. Sorry, not the artworks by KAWS, I actually meant the Warhol collection. But KAWS is the service. Right. Anyway…”

Feature flagging a GraphQL service with a schema is hard

Our plan for launching this change was pretty simple: introduce a new environment variable to Metaphysics and toggle it when we wanted to QA or launch. Toggling the variable to true and restarting Metaphysics’ Kubernetes pods would result in a few updates to the schema (removal of all of the unused types that we got rid of) and would cause requests for collections to be routed to Gravity instead of KAWS.

We figured that we would flip this variable on staging, QA, resolve any issues, and release. We wouldn’t need to flip it off unless we had broken something big by accident; we wouldn’t be in danger of shipping these changes to production since they would exist only in our staging environment and were not checked in to version control.

However, we had forgotten something crucial: Metaphysics has a post-merge step in CI that creates PRs with any schema updates in client services like Force and Eigen, and the schema it ships to those services is pulled directly from staging Metaphysics. It’s not the version of the schema that’s stored in GitHub (which we had carefully avoided changing by not flipping the feature flag), it’s the schema that exists in the staging environment.

In other words, Force (for example) received an updated MarketingCollection type in its schema, which didn’t exist (and wouldn’t exist until we finished QA and decided to launch) in production Metaphysics schema. Oops.

This also meant we accidentally blocked deploys for a while. Services like Force have a check before they can be deployed to production: “does my copy of Metaphysics’ schema have changes that are not present in production Metaphysics? If so, that probably means deploying me would break something, so go deploy Metaphysics first.”

Because the flag was still set to false on production, the schema updates would never move past staging. We eventually figured out the problem and flipped the flag to “off”, re-ran the “update client schemas” job, and deployed Force without the updated schema. But it made for a pretty confusing few hours, and it could have been bad if we needed to ship an urgent bugfix. Lesson learned.

We ended up creating a Force review app that pointed at a Metaphysics review app with the flag set to true to avoid causing more issues. This is definitely what we would do next time.

Conclusion

Overall, we’re feeling pretty happy about how this project went. We hit a few snags along the way, but we launched the change earlier this week and haven’t seen any noticeable bugs or problems.

Of course, there’s still an important question: was it worth it? What did we gain by completing this refactor?

The answer will mostly depend on what happens next. We’re meeting with stakeholders next week to talk about what kind of features we can add or modifications we might want to make. But for now, we’re excited to have one less application to maintain and to have made it easier for future Artsy engineers to understand and iterate on our collections infrastructure.

Stepping Out Of The Knowledge Zone

Wed, 13 Apr 2022 00:00:00 +0000

As I am writing my first blog post for Artsy, here is a short introduction on who I am: My name is Kaja and I am an engineer in our Berlin entity. As a Ruby-born programmer I am calling myself a backend engineer, but that also may change and is more about emotional identification and less about what I actually do (as you will see in this blog post). My background is not in engineering at all. In university I graduated in philosophy and historical linguistics, both the most impractical but most beautifully theoretical subjects I can think of. I really love being in the world of ideas and thought experiments that challenge the current status of what is, as opposed to what is thinkable.

At Artsy I am currently working in the PX (Partner Experience) Team and since I joined 1 year ago, most of my time in the PX team has been dedicated to the implementation of the Artsy Shipping feature. In the backend I helped tying the ties between the external ARTA-API (a white glove shipping company) and our own service that is managing the shipping and order statuses of the artworks that are ordered online. In the front end I implemented some forms for that feature and some so called hooks. I felt like a fish in the water with these tasks, because the company that I had worked at before was a shipping company and I did the same thing from the other side of the API relationship. In the same language (Ruby on Rails).

Here is another fact about me that is a premises for what comes next: I enjoy to be a learner of new things much more than being an expert about things that I already know. Of course being an expert is also flattering the ego, but after a while it can feel repetitive and make your soul feel old and tired. While the experience of being new to something and not an expert at all can give you a rejuvenating prickle. On the other side it can be scary to admit not knowing something and also the ego will feel small and hurt. But I learned that overcoming the hurt ego is my way to happiness in life and I am practicing every day to let go of the idea of me being the admirable expert and commit to the philosophical open mind of the “The only thing I know, is that I don’t know.”.

In the search of something new that I don’t know I stretched out my feelers* and wanted to continue my learnings in Elixir. At my previous job I had done quite some work in that new language and still felt non-experty enough in it to feel the prickle of learning it. It turned out to be a bit difficult since there were not many people around me able to pair and guide me through some problems and the actual work to be done there did not appear to have enough priority to be taking so much of my work time.

One thing I personally care a lot about is the general health of the code base and that the whole machine of a distributed system is well oiled and working in a way that makes it easy to contribute to. This is why I also really care about updating dependencies and feel personally in charge of doing that without anyone giving me this role. That care and my first rotation at Artsy has made me cross the paths with the velocity team several times. I understood that the care for the health of the code base is a shared emotion with the velocity team members and I felt heard when they picked up this urgent topic in a platform practice (an open somewhat informal meeting for all the devops/infrastructure things). My curiosity for the velocity team grew.

But did I reach out to the velocity team to ask for some collaboration on the dependency updates? No. You have the privilege of having a female perspective here in the blog post and I can give you some insights on the barriers for me as a woman in tech. While the PX team is well equipped with women engineers, the velocity team only consists of men. This plus the aura of tech-geniuses that usually surrounds the people who do devops from the perspective of an engineer made me feel nervous and shy to reach out to the velocity team members.

Another thing that happened was that during some of these beloved dependency updates I had to change the docker file of the project, because I needed a newer ruby image. The ruby image we used was a specific artsy ruby image and we did not have one for the ruby version that I needed, so here I had to go and change the whole docker file set up. I did not know much about docker and what all these funny RUN commands actually did and just tried and guessed and broke stuff as I went along and at some point got stuck. So naturally I reached out to our dev-help channel and was quickly having the help of Andre, our velocity team member in the German time zone. He paired with me and explained the docker commands on the go and made everything seem so simple and straight forward, while also being super nice and approachable talking about his cat and his favorite Brazilian food, that I had to question my own prejudice about the velocity team members and come to the conclusion that they might all be men and super good at what they did, but that at least one of them was really good at demystifying stuff and helping me out.

The new learnings about docker lit a fire in me: I wanted to learn more of this mysterious world of infrastructure. After letting the thoughts and feelings simmer for another week or so, I had a 1:1 with my manager Christian. He has some magical question asking powers that truly trigger introspection and I ended up hearing myself say to him: “I wish I could do something like an internship in the velocity team!”. Here it was. The actual wish had formed into something that is actionable and as I learned totally possible. My Manager did some magic managing and piff paff here I was starting my 2 sprint (4 weeks) rotation in the velocity team!

At first I was still a bit scared and nervous. Would I be able to contribute anything or would I just be an annoyance to the team? Would I make a fool of myself and be branded as a stereotypical woman who doesn’t know anything about tech (this is coming from many years of being perceived as a person without any tech skills). I latched myself onto the thought, that Andre would be the person that I could talk to and that I knew he was not going to bite my head off. I also realized in the organizing process that there was Matt Jones, another velocity team member, who was reaching out to me and being super open and kind and having great communication skills and it made me feel less scared.

The first week came up and I found myself in meetings with the velocity team mostly listening at the beginning as I was lacking some context in the conversations, but not feeling super weird as I saw that there was nothing of the expected mysterious witch craft happening but a very straight forward Kanban board where the only unknowns where some names of tools that I had not used before. I was also glad to see some of the faces in the meetings showing signs of human struggle like tiredness, a very relatable thing. During my first week I didn’t really know what to do as some of the tasks suddenly got blocked or changed or appeared to be already solved and the only thing that kept popping up as a suggestion was to do some work on integrity, our little QA robot that automatically clicks through the staging interfaces and checks if it behaves as expected. The general reasoning of people in the team was “I don’t really know javascript and this cypress stuff, so it would be cool if you can take over some of the integrity tasks”. I was still too shy to loudly admit that I myself was also not knowing anything about JS or cypress. I felt that as a web developer coming to the velocity team they would think, but is there anything that she knows if she doesn’t know docker nor JS??

Matt Jones asked me for feedback on my first week as he wanted to make sure that this was a useful learning experience for me and I told him openly that I felt a bit confused on what my tasks were and that I also didn’t know how to fix these tests in integrity and that the only other task that was assigned to me was seeming like a dead end to me. That other task was to implement a new user role into something that is called the old admin! Somehow this made my neck hair stand up. Implementing a new feature into a half way sunset service? I looked into the code to see if this would be simple and straight forward and saw myself confronted with somewhat outdated coffeescript and felt a cold shudder.

The second week began and as Matt had made me join some meetings on discussing how and if we should set up an Artsy.net local docker container for the development and what other services than force (our main service for Artsy.net) would be useful to the engineers that usually worked on force. It was an interesting learning experience especially because Elena, the engineering Manager on the velocity and data platform teams, was asking some really good critical questions that made me realize there was more to it than just a simple docker run for a single project. She mentioned problems with node and the usability of the development environment, talked about previous experiences with such aims and how it was not working as smoothly as imagined.

The second week in the velocity team also brought along some team building time. I joined a coffee call and it was super interesting to finally also feel like I was getting to know the other mysterious team members of velocity. We talked about our international backgrounds and it was great to hear so many stories about where people had come from and what languages they had grown up with. This finally was melting my heart for the team and I lost my last little bit of fear and shyness.

As I then started to look at integrity and list some of the tests in it that were flakey Joey, our director of engineering, was helping me out leaving meaningful hints and comments on my notes. This plus the recent successful coffee call with the team encouraged me to my next very risky step: In a meeting called “velocity mob session” I proposed to the team members to mob together on some problem of integrity actually not being able to run certain tests from two instances in the same time. It felt risky and scary driving a mob session in a repository that I wasn’t familiar with, reproducing a problem that I wasn’t fully understanding and trying to solve it in a language/framework that I was not an expert in. But the risk was totally worth it. It turned out to be a super interesting and fun and collaborative mob session and we actually figured stuff out together with the whole team. Nothing feels better than having a whole team figuring something out together. It really bonds to see stuff happening as a shared experience.

The third week started and in a standup I admitted that I had not been following up on implementing a new role into our old-admin because I felt uneasy to add something new to a project that was stuck in a weird mid-deprecation state. The team reacted quickly with a super good idea: Instead of adding something to the old-admin I should set up something that was going to be the new-admin in our environment so that people could contribute to it on the upcoming hackathon. I was thrilled! Never have I witnessed the birth of a new project at Artsy and now I was supposed to set the whole thing up and go through each step of putting it out there? Awesome! Also I had absolutely no idea what that meant. Did I need to do some dangerous mystic exploration into the far away field of our system being set up in kubernetes or something? I wasn’t sure. All I had in mind were these complex huge graphs of our system and how all the services were connected. How could I add another service into this huge spider web?

The first helpful thing was a little check list that Joey gave me of things that had to be done. Breaking the task down into smaller task definitely already demystified the whole thing a bit. It looked something like:

get typical local development working
get hokusai dev start working (i.e., local development on docker)
create staging deployment(s) and get them working
CircleCI
create production deployment(s) and get them working
set up the project in Horizon

Also I knew that Roop was the point person for this project and had started setting up the github repo so I started to set up the project locally myself and then went to reach out to him for my questions that had come up. This is how I had my first ever pairing session with Roop and it was so much fun. After having the project working for me locally I was informed that Devon had already put a tutorial video out there on how to set up a new project with hokusai in the artsy system. I watched the video and followed the steps along and when I got stuck reached out to Devon directly and there I was having another interesting and fun pairing session on setting up a new project with hokusai.

As you can see each step demystified this task further and further so that in the end after another bunch of fun pairing sessions with Jian and Joey I got the new admin deployed and ready just in time for my week four of the rotation: The hackathon week! During that week I was continuing my integrity work, pairing with different people from different product teams on the flakey cypress tests and in the same time doing my own personal favorite hackathon project: Updating dependencies in our code base. I will basically use every excuse I get to update our dependencies, because they just need that love and I believe that we can get the code base to a point where it is not so painful anymore to keep stuff up to date.

As the end of the fourth week came up, so did my end of the rotation with the velocity team and I was sad to leave. I asked the team members to give me some feedback and was hoping to be able to work on the new admin and on integrity and on the Artsy local docker stuff more in the future.

The following weeks some awesome feedback came in and I was glad to see that I was not the only one really enjoying all the nice pairing and good times on the velocity team. As much as I like the PX team and also missed them, I secretly hoped that some day in the future I could switch over to velocity in the long term. I knew now that I would be learning so much and that none of this was scary anymore. And then it came: An email from Elena saying that the velocity team was internally searching for a new member! Maybe the timing was just right, maybe the universe just loves me so much… I don’t know, but my wishes became true and I am now moving over to velocity. I guess sometimes it is really worth leaning into the scary unknowns and going all the way there to see if it is really so scary.

*“Die Fühler ausstrecken” = German to stretch out your feelers stands for “to prudently try and see something that you don’t know” as a snail would do to find her way forward very slowly and retreat from something unpleasant.

Using Context to Simplify a VERY Large React Form

Tue, 01 Feb 2022 00:00:00 +0000

For those unfamiliar, Artsy is a fine art marketplace. Knowing that, it follows logically to say that the form via which our partners list artworks for sale is an integral part of Artsy’s core systems. This form, known only as “The Artwork Form,” is whispered about in the halls of Arty’s New York headquarters. It is legendary. It is a colossus. It is old enough not only to predate React v16.8 hooks and context APIs, but Artsy’s use of React entirely. The first version of the Artwork Form was built in 2014 using ruby and haml, and began its refactoring into JS/JQuery/React a full 2 years later, after having expanded considerably from the original implementation. That process (at least what we’ve gleaned from our git excavation) was incremental, experimental, and passed through many hands before it landed in the lap of the current Partner Experience (PX) team.

PX has since been tasked with the maintenance of this unwieldy kaiju, spending endless amounts of time on seemingly insignificant changes to behavior or UI, all while watching its performance degrade. Many of the people reading this are already familiar with the story we’re telling. Many have worked in their very own version of the tale, and borne witness to the fact that legacy code of this scale becomes a living, breathing entity. The developers that tend these beasts learn their patterns and idiosyncrasies, their little moans and groans, and for the sake of expediency work within those constraints to accomplish their tasks. But when is enough, enough? When does the developer time expended working within the constraints of an obsolete design begin to outweigh the time it would take to simply fix the code?

First, let’s be honest: there is no single right answer to this question. When working in software development we have to deal with certain realities: user experience vs. developer experience, lead time to the next release, buy-in from stakeholders, etc. These factors may weigh more or less depending on the shop and the product. At Artsy we’re very lucky, in that our engineering department is given the time to attend to our tech debt and to be deliberate about when and how we go about this. In the case of the Artwork Form, there were several issues that had become too glaring to ignore:

The data coming in, and subsequently being passed to individual components, was being completely obscured by the amount of prop drilling and spreading that existed within the composed form.
The prolific use of any when typing data was disabling typescript and consequently removing its usefulness while still imposing all of its burdens.
The components within the form were tightly coupled, the number and specificity of props needed for each disallowing reuse in other parts of the app.

A few of us on the PX team decided to take matters into our own hands and address some of these key problems with the Artwork Form. We came up with a plan to incrementally convert the form to use values from the Formik context, use these values wherever possible to reduce prop-drilling, add much more complete types to the components to get rid of all the any’s, and update some of our testing strategies as needed.

Here are the steps we took to do this conversion:

Create a hook that allows us to use the Formik context throughout the form

The hook wraps useFormikContext to allow all of our components inside of Formik to access the values from Formik context. It looks like this:
```
export function useArtworkForm(): FormikContextType<ArtworkValues> {
  const formikContext = useFormikContext<ArtworkValues>()
  return formikContext
}
```
We use ArtworkValues as the generic type so that when we are accessing values anywhere inside the component tree, values can be type-checked.
Convert all of the components in the form to functional components and to TypeScript. (Because the form is several years old, there were still many class components and many components that were not yet using TypeScript.) This step could be done in parallel with Step 1. One note here is that when converting files from JavaScript to TypeScript, we did not explicitly type the props in an interface. Once we can take advantage of our useArtworkForm hook, we will reduce the amount of props needed in each component, so we will hold off on typing the props until step 3.
The bulk of the work was making use of the useArtworkForm hook in the Artwork Form components. Starting with the lowest leaves of the component tree and moving up, we removed the props from the component definition and destructured any values we needed in the component from the useArtworkForm hook. Once we used everything we could from the context, we added back in any additional props that we would still need to have passed down from the parent. In many cases, this was no props at all—a particularly satisfying case. If the component still needed props passed down, we explicitly typed the props at the top of each component in an interface because we now knew exactly which props we would need inside of the component. Another key step here was going into the parent component and getting rid of any prop spreading (this: ...props) and instead explicitly passing down exactly the props needed in the component (if there were any).
1. A note on tests: Whenever we took advantage of the Formik context in a component, we were breaking that component’s tests, because the wrappers we were using in the tests did not have access to the Formik context and were being passed props that the component was no longer looking at. We created a helper test wrapper that we could use in all of our Artwork Form tests to wrap the test’s specific wrapper inside of a <Formik/> component and provide the specific values to use as the initial values for Formik. Many of our test cases involve passing different values into the component, which we originally did via props but now will do via the Formik context through the wrapper. Here’s what the wrapper looks like:
```
export const TestFormikWrapper: React.FC<TestFormikWrapperProps> = (props) => {
  const { children, values } = props

  return (
    <Formik initialValues={values} onSubmit={jest.fn()}>
      {children}
    </Formik>
  )
}
```
Here is an example of TestFormikWrapper used in a test:
```
describe("TestComponent", () => {
  it("displays values", () => {
    const wrapper = mount(
      <TestFormikWrapper values={{ name: "Andy Warhol" }}>
        <TestComponent />
      </TestFormikWrapper>
    )

    const name = wrapper.find("#name").html()
    expect(name).toInclude("Andy Warhol")
  })
})
```
Once we completed the conversion all the way up the tree to the root component, the ArtworkForm, we typed that component as strictly as possible and made sure to get rid of any’s. There were quite a few when we started the process.

So, where did we end up? Now all of the components in the Artwork Form are making use of the useArtworkForm hook if they were previously accessing any of the values from the Formik context from props. We have much less prop drilling between components and instead explicitly pass down the props needed from parent to child. It’s now much more clear for developers what data is passing between the components and what data is actually being used in the child. All of the components are also explicitly typed so we know exactly which props, if any, need to be passed down from the parent. If any of these props are removed, TypeScript helps us by failing loudly.

One of the main pain points of the Artwork Form is that it’s very difficult for new developers (whether new to Artsy or new to the Partner Experience team) to contribute and make changes to the form without breaking something or spending extra time figuring out how data is passed within the form. Hopefully, this change will make it easier for developers to understand the Artwork From.

How did the Artwork Form get so complicated? Well, as we shared, the Artwork Form is the key to achieving one of the PX team’s core goals: surfacing the most accurate and rich information about artworks to collectors. We have to allow partners to add more and increasingly specific pieces of metadata to artworks. The form has been growing and for better or worse, will need to keep growing. Even though we expect to grow the form to meet metadata needs, we do not put too much focus on the UX/UI of the Artwork Form in order to prioritize our collector-facing apps. (The Artwork Form is only used by a relatively small subset of users, mostly gallery partners.) Hopefully, this refactor will allow us to expand the form more seamlessly and will make it easier to navigate as it grows.

This refactor is still in its early days. The next steps for making the form easier to use (for both developers and our end users) will require larger changes. When we think about further progress on revamping the Artwork Form, our team is considering breaking the form up into smaller forms. Imagine, we are rendering several different top-level Formik components that include discrete sections of the form, instead of just one giant Formik tree as we have now. We would then combine these “mini forms” together, making better use of React’s core principle of composition.

Breaking up the form would be a big change for the developer experience (but hopefully made easier by this refactor). It could also involve big changes to the UI. The Artwork Form is not just hard to navigate for developers. It’s hard for users too. Over the next couple of months, our Product Manager will be working with one of Artsy’s user researchers to conduct user testing on the form. Feedback from galleries will help determine where we next take this project. We’re excited to have buy-in from our product team to work on a project that will elevate the user experience while allowing us to use that opportunity to improve the developer experience as well.