trapped in the technologist factory

The first time I was paid to use Clojure was only two years after the language was released. Needless to say, it was at an early-stage startup. The idea behind Runa was that we’d do dynamic pricing for smaller online retailers, offering targeted discounts based on individual shopper behavior. In practice, we offered a fixed discount to anyone who abandoned their shopping cart. By treating a small percentage of customers as a control group, we could calculate our effect on the retailer’s sales, and take a cut. Our biggest problem was convincing retailers to let us control their prices, closely followed by convincing them the math behind our calculated “lift” was sound. We told ourselves a big contract was always just about to close, permanently increasing our traffic by an order of magnitude. We used HBase as our primary data store, because it was designed to scale, just like us.1 In the future, we would power a loosely federated Amazon, composed of independent online retailers, knit together by our software. If we built it, they would come. But they didn’t. The only thing that kept us afloat was a bespoke product we had built for eBay, because our CEO knew an executive there. Later, after I left, that same executive moved to Staples and convinced them to acquire the startup outright. Nothing we had built was useful to Staples, it was just evidence of our ability to “innovate”. The resulting “skunk works” team has since been disbanded. My next job was at Factual, a more established startup that had built their product using Java and embedded Javascript, and were beginning to adopt Clojure. Their product was an index of all the places in the world, generated by applying hundreds of different heuristics to web crawl data. Each entry had a canonical id and various properties like name, address, and hours. More interestingly, it also had the equivalent ids from different platforms like Yelp and Foursquare, and we offered an API that could resolve an an incomplete (or even partially incorrect) set of properties to the corresponding entity in our index. Our product could stitch together thousands of disparate datasets, so long as they had some geographic component; I liked to describe it as a “foreign key for the world”. Unfortunately, there’s not much money in ontologies. By the time I had joined, the dataset had already been used as the basis for Facebook Places and as part of Apple’s then-new map offering. These were large contracts, but there are only so many big tech firms in the world, and even fewer with a geographic component to their data. No matter how much we tried to make up the difference with our smaller customers, we didn’t have a path to a billion dollar valuation. We began to offer services powered by our data. Internally, this was presented as less a pivot than a demonstration of the intrinsic value of our data; once people saw what we could do with our data, surely they’d have some ideas of their own. Our first customers were adtech firms, hoping to do better geographic targeting for mobile ads. We hired salespeople and marketing executives with a background in digital advertising, and when sales started to take off we hired more. Even as I built services to ingest billions of daily ad impressions and bespoke almost-databases to power on-premise services for our adtech partners, I told myself it was all incidental; our business was data, not ads. I was wrong, of course. The advertising side of the company was where all the growth was happening, and our product direction was whatever helped them close deals. But I wasn’t the only person who was slow to acknowledge this fact. In my exit interview with the CEO, just after I had given notice, I told him that he should be more direct with everyone that Factual’s focus had permanently shifted. He told me what I had only just stopped telling myself: ads were a temporary diversion, and we’d return to the underlying data before too long. In the years that followed, however, Factual became increasingly focused on location-based advertising, until it finally merged with Foursquare. When a startup takes venture funding, its singular goal becomes growth; it will either reach a billion dollar valuation or die trying. With that growth comes ever-greater technical challenges. Some of these are obvious: more users will not only require more machines, but a more complex infrastructure.2 Others are less obvious: as the userbase grows, each individual becomes increasingly abstracted in the business logic of our software. To deal with the first challenge, we must have technical acumen. To deal with the second, we must have a deep understanding of our business domain. This means that as long as a startup continues to grow, its engineers can (and must) continue to improve their knowledge and acumen. But most startups fail. Some of those failures are because the company couldn’t keep up with their own growth, but these are the exception. The common, boring reason for failure is that the startup barely grew at all. Confronted with stagnation, an engineering organization can have two reactions: they can try to help reinvent the company, or they can continue to prepare for the growth that must, inevitably, come. At Runa, we chose the latter. At first, the engineering leadership adopted promising, poorly-understood technologies (Clojure! HBase!) which would give them the necessary leverage down the line. But once they launched, and the stagnation set in, that preparation became a need for constant readiness. Any early technical mistakes, however regrettable, were baked in. Trying to revisit them, to build something better, would just leave us flat-footed when the growth finally came. I joined Runa only a few months before the engineering organization entered its defensive crouch. I was too inexperienced and too distracted3 to question it. Every day we played Jenga, trying to jostle the existing codebase as little as possible with each new commit. Later, as I realized I wasn’t proud of any of the work I had done so far, I began to energetically push for rebuilding key pieces of the system. After a few months, I was told to either drop it or take three weeks severance. I took the severance. Of course, even if I had gotten my way, nothing would have changed. All of us were carefully ignoring the actual problem, which is that our business model didn’t really work. Our product and pricing model both required unjustifiable levels of trust from our prospective customers, but none of us saw that as our problem to solve. We were downstream of the business model; our job was simply to wait and prepare for its eventual success. I joined Factual because I found the problem space fascinating, but most of it had been solved by the time I joined. About six months later, the leadership began to acknowledge that our business model, while successful, wasn’t going to get us to a billion dollar valuation. So began my descent into adtech. I wanted to prove that I had learned my lesson; if the current business model wasn’t working, I’d help them build a new one. I built services to consume the raw data, and on-premise servers to provide the processed result to our customers. I attended sales meetings, wrote design docs, and painstakingly analyzed packet dumps to prove to partners that latency spikes were occurring elsewhere in their system. In the end, I learned that I didn’t like the advertising industry very much. When a brand wanted to run a digital ad campaign, they went to an agency, which is be responsible for spending their budget to the greatest effect. The agency then went to a demand-side platform (DSP), which was be responsible for making real-time bids on individual ad impressions on an exchange. To inform those decisions, the DSP went to a company like Factual, which could distill a phone’s current and past locations into binary features relevant to the campaign. Crucially, when you’re not selling something online, it’s almost impossible to measure the efficacy of a digital ad campaign.4 This means that the agency could say almost anything, so long as it closed the deal. The DSP, in turn, sold whatever they thought would help agencies close their deals. And at the end of this chain was Factual, selling whatever we thought would entice the salespeople enticing the salespeople enticing the brands. I thought I had become product-focused, ready to solve any problem that would help the company to succeed. In truth, I was only really paying attention to the technical problems. Once they were largely solved, it became harder to ignore my ambivalence for what the product actually did. Eventually, I left. I learned two lessons from this time in my life. The first was personal: I am, at heart, a technologist. I like to generalize, to abstract. While I believe it’s crucial to understand the context around your software, I’m happiest when that context is other people’s software. The second lesson was broader, and less obvious:5 our industry is designed to foster people like me. This was surprising because it seems so clearly against our own interest. In almost every case, companies fail because they build the wrong thing. Unless your customers are themselves engineers, I’m the wrong person to help with that. You want someone comfortable at the periphery of your system, who wants to learn about the competitive landscape, who wants to talk to customers. You want a product engineer. But consider our standard interview questions: data structures, recursion, and computational complexity. It always feels a little strange when I see someone arguing that these don’t reflect “real” day-to-day software tasks; I write recursive functions all the time. But that’s a consequence of the abstraction; what might be a simple nested lookup on any specific datatype becomes recursion when you try to generalize over a set of possible datatypes.