You Don't Need a Dashboard

For at least as long as I've been working in data analytics, the clamoring from the datarati for Dashboards! Dashboards! Dashboards! has consistently risen year over year in pitch and volume.  Spotfire and Tableau produce two really popular products, and my company has at least three different "dashboard-style" plugins.  This week I also discovered Kibana, which frankly looks really awesome, because it's free and dead-easy to set up if you already have ElasticSearch (and ES is also dead-easy to set up; however, as I discovered the hard way, beware of multiple people screwing around with ElasticSearch prototyping on the same subnet, as the out-of-the-box configuration will make everybody's ElasticSearch node automagically join into one cluster transparently, with obviously undesirable results.  Data integration!)

The most common thing I hear about dashboards is that people want them to "spot trends".  And, as far as I can tell: No, you don't.  Not really.  One golden truth I have learned from working in data analytics is this: If you cannot pose a concrete question that you would like to answer or a concrete problem that you would like to solve, then you're wasting your money.  Dashboards do neither of these things, at least not in the way most people use them, which is often as command center props.  The process of formulating a question in a way that can be answered using data is not simple, and hence visualizing data in ways that is essentially static will not answer meaningful questions.  And by "static" I don't mean "has no temporal aspect".  Charts that show values over time are still static if the things being plotted cannot be changed easily and intuitively.  And, if you spend a lot of time changing the visualizations and exploring different hypotheses, then you're not using a "dashboard"; think about it: a car dashboard is a thing you look at to get an immediate read on something, like your speed or fuel level.  How often do you switch your speed readout from MPH to KPH?  So, if that's you: congrats!  You're a data scientist!  You should stop thinking about dashboards and start thinking about a real, scalable data analysis platform.

From the Archives

Note: This is a repost from my old blog, explaining my departure from academia; this came up in response to some recent discussions about a candidate who was deciding between attending CS grad school and getting a job, and how to dissuade them from the former, which led here:
http://anothersb.blogspot.co.uk/2014/02/goodbye-academia.html
Several people asked me to repost this in a place where they could read it, so here it is.

In case you've been coming here day after day, wondering why I have stopped posting, and where in the world is Carmen Sandiego, I thought I would update my (rapidly shrinking) fan base with my whereabouts: for almost a year, with the prospect of my funding drying up, and no publication in easy sight, I spent some time taking stock of my options, and deciding what I wanted to do with the rest of my life. And, the fact of the matter is: I was bored. I was unhappy. I really think single molecule biophysics is awesome, and fun, and there's great stuff happening. But I had discarded enough plastic pipette tips to last a lifetime, and was finding it increasingly difficult to care about the rate constant for phosphate release of E. Coli RNA polymerase in the presence of XYZ. Additionally, I had grown to really love the bay area, and didn't want to leave my friends, my girlfriend, and my hang glider behind.

My options, as I saw it, were:

1. Keep on keepin' on, and try to find a professorship at the University of Wallamaloo, or wherever I could, and hope that things would be more fun and interesting as a mid-grade intellectual at a mid-grade university.
2. Back out, and try to find another postdoctoral appointment doing something completely different, and hope that, four years down the line, at 38, I wasn't totally burnt out on that as well. I considered computational evolution, or even getting a masters in EE or CS, and seeing where that took me.
3. Get a real job.

I talked to a lot of people, and received some encouragement from some corners, most notably from Ben Ovryn at AECOM who strongly encouraged me to not give up on academic science. Just as conspicuously, I received no such encouragement from my research advisor, who, when I discussed my options with him, basically shrugged.

I applied for biotech jobs, software jobs, and a professorship at City College of New York, because I thought it would be fun to live there and teach there. I considered taking a year off to write a book, or become a professional hang glider pilot, or both. I thought of opening an artisanal sandwich cart in San Francisco with a friend of mine, because, let's face it, you can't get a decent deli style sandwich in the bay area. In the end, I had a job offer from a high flying biotech startup that would have required me to move to the east coast, and a job offer to work at a friend's software company, and I chose the latter.

So, this is where I am. I'm currently employed as a Forward Deployed Engineer at Palantir Technologies in Palo Alto. The software is incredible, the people are amazingly smart and fun, and my group is mostly comprised of Ph.D.s who left science to try something else, and wound up here. I do a bit of everything: I project manage, I code, I do some outreach, I integrate data, and I sometimes look for new and interesting ways to use our product. I've been here for four months, and it's been pretty much non-stop excitement.

I've learned a lot. First, that there are a lot of really smart people out in the private sector, if you look in the right places. Scary smart people, the kind that academics will tell you don't work in the private sector. Second, I've realized that many of the dysfunctional relationships I had in the academic world were not actually due to my personality flaws, but were largely due to the peculiar culture that tolerates (and in some cases rewards) dysfunctional interpersonal relationships in academia. It's refreshing to work with people who are smart, engaged, enthusiastic, and who genuinely want to work together to create something worthwhile and powerful. I think, to some extent, the archetypal academic interaction is the pissing contest, where people jockey for status, because status is the only currency in the academic world. All other forms of interaction are subordinate to the pissing contest. It's refreshing to step away from that world. And, third, it took me two or three months out of academia to realize how really bored I was with what I was doing. It's not that I think it's intrinsically boring; it's just that it wasn't really driving me to do more and accomplish more, but I had had myself convinced that this was the way to go, that this was interesting because everybody else said it was. With some hindsight, I can see that if I had found it really that fascinating, I would have been eager to get up and get in there to do more. And there just wasn't that drive, and it was making me miserable.

So, I'm here, I'm finally liking what I'm doing, and I'm liking the people I'm doing it with. I'm getting up in the morning excited to come over here and face the challenges of the day. I'm still advising the grad student who's following up on my work a little bit, and I'm even doing some consulting for a biotech startup in Silicon Valley, just to stay in the game for fun. And, if I ever start to get bored with what I'm doing, I'll remember what it feels like, and I'll do something else. I don't know if I'll keep updating this blog again. Now that I've come back and gotten the long-overdue explanation out of the way, I may just post little sciencey tidbits here and there to amuse myself. We'll see.