An Offer You Can't Not Refuse

On a mailing list, received today:

From: XXX 

Subject: Seeking Technical Co-Founder  

I'm working part-time with XXX, an early-stage start-up providing an activity-planning and metrics-gathering platform for hospitality and tourism organizations. XXX is looking for a technical co-founder to join the team initially in a part-time, equity-based capacity.  The co-founder should have experience with mobile and web development, including in a LAMP environment, and be able to contribute in both hands-on and strategic fashion. 

If you or someone you know might be a fit or you would like more information, you can contact the CEO XXX directly at XXX

Not only will you not pay anything, and not contribute anything, you're still using PHP.  (Actually, more likely, you just don't know WTF a "LAMP environment" is.)

Big Data: It's Not the Size That Matters

Over a year ago, Ari Gesher asked me to sit on a panel at Georgetown Law called "Swimming in the Ocean of Big Data".  After I agreed to it, he sprung on me that being on the panel required me to write 10,000 words for their journal.  So, it finally arrived physically on my desk today (they sent me actual paper copies, how quaint!).  You can find it here.  The editor's abstract:

In this article, the author argues that the premise of “big data” lies not in the amount of data that we can generate, collect or store, but in the ability to use data to make informed decisions. He explores both the needs of data analysis projects and how to apply these types of projects to real-world circumstances. The article concludes that whatever data systems are ultimately put in place, privacy and civil liberties should be a primary concern and not simply an afterthought; the author proposes methods that can be implemented during the design of big data programs in order to curtail privacy violations.

The academic in me finds it hard not to be proud that I have another publication to my name, but the reformed-academic-turned-engineer in me knows that publications are a poor substitute for Actually Doing Things That Make a Difference.  Nice to have it both ways for once.

You Don't Need a Dashboard

For at least as long as I've been working in data analytics, the clamoring from the datarati for Dashboards! Dashboards! Dashboards! has consistently risen year over year in pitch and volume.  Spotfire and Tableau produce two really popular products, and my company has at least three different "dashboard-style" plugins.  This week I also discovered Kibana, which frankly looks really awesome, because it's free and dead-easy to set up if you already have ElasticSearch (and ES is also dead-easy to set up; however, as I discovered the hard way, beware of multiple people screwing around with ElasticSearch prototyping on the same subnet, as the out-of-the-box configuration will make everybody's ElasticSearch node automagically join into one cluster transparently, with obviously undesirable results.  Data integration!)

The most common thing I hear about dashboards is that people want them to "spot trends".  And, as far as I can tell: No, you don't.  Not really.  One golden truth I have learned from working in data analytics is this: If you cannot pose a concrete question that you would like to answer or a concrete problem that you would like to solve, then you're wasting your money.  Dashboards do neither of these things, at least not in the way most people use them, which is often as command center props.  The process of formulating a question in a way that can be answered using data is not simple, and hence visualizing data in ways that is essentially static will not answer meaningful questions.  And by "static" I don't mean "has no temporal aspect".  Charts that show values over time are still static if the things being plotted cannot be changed easily and intuitively.  And, if you spend a lot of time changing the visualizations and exploring different hypotheses, then you're not using a "dashboard"; think about it: a car dashboard is a thing you look at to get an immediate read on something, like your speed or fuel level.  How often do you switch your speed readout from MPH to KPH?  So, if that's you: congrats!  You're a data scientist!  You should stop thinking about dashboards and start thinking about a real, scalable data analysis platform.

From the Archives

Note: This is a repost from my old blog, explaining my departure from academia; this came up in response to some recent discussions about a candidate who was deciding between attending CS grad school and getting a job, and how to dissuade them from the former, which led here:
http://anothersb.blogspot.co.uk/2014/02/goodbye-academia.html
Several people asked me to repost this in a place where they could read it, so here it is.

In case you've been coming here day after day, wondering why I have stopped posting, and where in the world is Carmen Sandiego, I thought I would update my (rapidly shrinking) fan base with my whereabouts: for almost a year, with the prospect of my funding drying up, and no publication in easy sight, I spent some time taking stock of my options, and deciding what I wanted to do with the rest of my life. And, the fact of the matter is: I was bored. I was unhappy. I really think single molecule biophysics is awesome, and fun, and there's great stuff happening. But I had discarded enough plastic pipette tips to last a lifetime, and was finding it increasingly difficult to care about the rate constant for phosphate release of E. Coli RNA polymerase in the presence of XYZ. Additionally, I had grown to really love the bay area, and didn't want to leave my friends, my girlfriend, and my hang glider behind.

My options, as I saw it, were:

1. Keep on keepin' on, and try to find a professorship at the University of Wallamaloo, or wherever I could, and hope that things would be more fun and interesting as a mid-grade intellectual at a mid-grade university.
2. Back out, and try to find another postdoctoral appointment doing something completely different, and hope that, four years down the line, at 38, I wasn't totally burnt out on that as well. I considered computational evolution, or even getting a masters in EE or CS, and seeing where that took me.
3. Get a real job.

I talked to a lot of people, and received some encouragement from some corners, most notably from Ben Ovryn at AECOM who strongly encouraged me to not give up on academic science. Just as conspicuously, I received no such encouragement from my research advisor, who, when I discussed my options with him, basically shrugged.

I applied for biotech jobs, software jobs, and a professorship at City College of New York, because I thought it would be fun to live there and teach there. I considered taking a year off to write a book, or become a professional hang glider pilot, or both. I thought of opening an artisanal sandwich cart in San Francisco with a friend of mine, because, let's face it, you can't get a decent deli style sandwich in the bay area. In the end, I had a job offer from a high flying biotech startup that would have required me to move to the east coast, and a job offer to work at a friend's software company, and I chose the latter.

So, this is where I am. I'm currently employed as a Forward Deployed Engineer at Palantir Technologies in Palo Alto. The software is incredible, the people are amazingly smart and fun, and my group is mostly comprised of Ph.D.s who left science to try something else, and wound up here. I do a bit of everything: I project manage, I code, I do some outreach, I integrate data, and I sometimes look for new and interesting ways to use our product. I've been here for four months, and it's been pretty much non-stop excitement.

I've learned a lot. First, that there are a lot of really smart people out in the private sector, if you look in the right places. Scary smart people, the kind that academics will tell you don't work in the private sector. Second, I've realized that many of the dysfunctional relationships I had in the academic world were not actually due to my personality flaws, but were largely due to the peculiar culture that tolerates (and in some cases rewards) dysfunctional interpersonal relationships in academia. It's refreshing to work with people who are smart, engaged, enthusiastic, and who genuinely want to work together to create something worthwhile and powerful. I think, to some extent, the archetypal academic interaction is the pissing contest, where people jockey for status, because status is the only currency in the academic world. All other forms of interaction are subordinate to the pissing contest. It's refreshing to step away from that world. And, third, it took me two or three months out of academia to realize how really bored I was with what I was doing. It's not that I think it's intrinsically boring; it's just that it wasn't really driving me to do more and accomplish more, but I had had myself convinced that this was the way to go, that this was interesting because everybody else said it was. With some hindsight, I can see that if I had found it really that fascinating, I would have been eager to get up and get in there to do more. And there just wasn't that drive, and it was making me miserable.

So, I'm here, I'm finally liking what I'm doing, and I'm liking the people I'm doing it with. I'm getting up in the morning excited to come over here and face the challenges of the day. I'm still advising the grad student who's following up on my work a little bit, and I'm even doing some consulting for a biotech startup in Silicon Valley, just to stay in the game for fun. And, if I ever start to get bored with what I'm doing, I'll remember what it feels like, and I'll do something else. I don't know if I'll keep updating this blog again. Now that I've come back and gotten the long-overdue explanation out of the way, I may just post little sciencey tidbits here and there to amuse myself. We'll see.

The Civilized Soldier

"The civilized soldier when shot recognizes that he is wounded and knows that the sooner he is attended to the sooner he will recover. He lies down on his stretcher and is taken off the field to his ambulance, where he is dressed or bandaged. Your fanatical barbarian, similarly wounded, continues to rush on, spear or sword in hand; and before you have the time to represent to him that his conduct is in flagrant violation of the understanding relative to the proper course for the wounded man to follow—he may have cut off your head." --Sir John Ardagh, discussing the necessity of "dum dums" (or expanding bullets) for international warfare, at the Hague Convention of 1899, which subsequently banned their use for international conflicts. Via Wikipedia.

Reflections from Bouldering

I've been spending a lot of time bouldering recently, a few times a week.  Besides strongly incentivizing me to lose 10 lbs, I have started to learn some interesting lessons, the hard way.

Indoor bouldering is like rock climbing, but the highest it gets is about 17 feet, the floors are padded about two feet thick, and there are no ropes.  That means I can show up whenever I want, alone, and climb for as little or as much time as I want, and not need someone to belay me.  It also means that the first few times you climb, it can be pretty unnerving because when you fall, you just fall, boom, onto the mat.  In fact I noticed that as I grew more tired during climbing, if I thought I wasn't going to be able to make it to the top, I would frequently climb or jump down while I still had control, even if I had some power left,  because I wanted to avoid being all the way at the top and having no strength left, forcing me to fall uncontrolled from the top, as opposed to falling in a controlled way from halfway up.  But, after you fall from the top a few times, this turns out to be a mistake: falling from the top doesn't hurt.  That's why they let you do it and don't get sued too often (although, the place is actually blanketed in cameras, in deference to our tort-happy society: just in case someone does something stupid and sues, they have you on record.)

But, the more interesting thing that I discovered was that the barrier to failure was often simply exhaustion rather than skill.  And this has a particularly interesting consequence: often, the best next move is making the next move.  As a beginner, your instinct is to stop at each hold, look around, and see where the next move is.  Which hold can you reach without falling over?  But, watching other skilled climbers in the gym, they do it differently: first, they study the route before they start climbing.  Then, once they're on their way up, they move gracefully and smoothly from one hold to another, and importantly, they keep moving.  While you're stopped, looking around, your arms are growing tired, your tendons are aching, the skin on your fingers is starting to grate under the hand holds.  And what I found, through brutal trial and error, was that I was much more consistently successful if I just kept moving.  In an indoor bouldering gym, the holds are laid out somewhat logically, but also somewhat deviously, so that it's not always obvious what the solution is.  But your brain moves pretty quickly, and without even realizing you're doing it, you're not even considering 9 out of the next 12 possible moves.  The extra 5 seconds that it takes per move to decide amongst those three remaining moves is probably the difference between near-complete exhaustion and complete exhaustion.  And complete exhaustion means failure.  I'm a big fan of stopping and thinking about what you're doing, but the lesson is, when you're resource constrained and time is not on your side, don't think too hard.  You might back yourself into a corner, but it's no worse than falling on your ass.

We Apologize for the Interruption in our Interrupted Programming

Not much call for blogging these days; most of the interesting Data Blogging Topics(tm) have been around the Snowden NSA leaks, and I've been trying to slog through a number of other things at work, so it's hard to find the energy to get invested in it.  But, I wanted to stop by the blog and give you a brief update, to the assembled masses who may read this later (and, I have found out the hard way, blogs left unattended can come back and bite you in the ass.)

When the Snowden/NSA leaks first started coming out, the scope was pretty limited.  Phone call metadata logging was the big topic, and my comments were primarily technical in nature.  My decision to not express a particular opinion on the politics might have been construed as a tacit approval, or at least a lack of outrage, and I think the latter was probably not far off.

In the meantime, however, a lot more things have come out, such as the fact that the NSA has p0wnz0red the entire internet, and we've been eavesdropping on foreign heads of state and American citizens for essentially no reason.  So, I wanted to update, for clarity, my feelings: this is odious, unamerican, and a fundamental breach of the public trust.  The excellent New Yorker article about Alan Rusbridger, the editor of The Guardian, indicates that we're still just seeing the tip of the iceberg; the only limiting factor is how fast the journalists can process and understand the documents they've been given*.  If you want to read more, the incomprable Bruce Schneier is your go to source, and I truly couldn't hope to add anything.

Sadly, in my search for hyperbole to compare this with, my mind goes back only as far as the buildup to the Iraq War, and it's hard for me to draw a comparison really: it's apples to oranges.  The Iraq war was a breach of the public trust in a fundamental way, but it involved a lie which resulted in the deaths of over 100,000 humans.  It's hard for me to draw a meaningful comparison there that doesn't minimize those deaths.  But the consequences of fundamentally weakening the internet is hard to grasp, in both its scope and consequences.  The ripples from this tidal wave will continue to leave marks in the sand well into the next generation, and only time will tell.

*The article refers also to Rusbridger's memoir in which he interleaves his Herculean work publishing the Snowden leaks with his year long struggle to master a particularly difficult work by Chopin.  I immediately thought, "He must not have any children," but of course, we find out, he does.  It is times like this that I am reminded of the late great David Foster Wallce's characterization of Wilhelm Leibniz in one of my favorite books ever written, "Everything And More: A Compact History of Infinity".  He describes Leibniz, one of the inventors of the calculus, as "a lawyer/diplomat/courtier/philosopher for whom math was sort of an offshoot hobby", which he tags, in typical David Foster Wallace fashion, with a footnote, saying only, "Surely, we all hate people like this."

Less Talky, More Hacky

Light posting lately.  I'm spending my time learning MongoDB, CouchDB, jQuery, Bootstrap, and node.js, in pursuit of various projects.  Gotta get hip with the web technologies, dontchaknow.

I was an invited speaker at Sibos last week in Dubai, and spoke on money laundering and counter-fraud in commercial banking, and specifically how the same types of data comes up over and over again, so it doesn't make sense to field a different platform for each problem.  My talk was apparently well received.

Theory Thursday: The Central Limit Theorem

Back when I was employed researching the mysteries of the universe by pipetting lots of stuff, I used to say that physics was the study of things that are pointy in the middle and small on the ends, so that we can ignore the ends.  Essentially, the idea in much of science is to figure out how to boil a phenomenon down to a few variables that have reasonably well defined values, i.e., a mean or average value.  All measured variables have a distribution, but not all distributions are pointy in the middle, and only for distributions that are pointy in the middle does it make sense to calculate the average value.  As a classic counterexample, the power-law distribution doesn't have a pointy middle:

You can still, of course, calculate the mean value of this distribution by summing over all the values and dividing by the number of values.  But the point is that it won't mean much intuitively: "most" of the values won't be "around" the mean value.  They're all over the place.  So, if we want to be able to talk about measuring a "variable", we'd like it to have a peak in the middle.  In particular, it would be really handy if our distribution was a Normal Distribution (aka, the famous Bell Curve, or Gaussian, after the legendary mathematician and physicist Carl Freidrich Gauss.)

A normal distribution has a couple of very nice properties that make math a lot easier:

• The mean, median, and mode are all the same.
• It's mathematically tractable to work with and has a simple form.

Luckily for us, the Central Limit Theorem has our back.  What it says, basically, is that if you take a whole bunch of random variables, what you get out will probably* be pretty close to a normal distribution.  And this is good news for people who like things high in the middle and flat on both ends**.  Most real processes in the world are the result of a bunch of sub-processes, each of which has its own distribution.  For instance, the average number of fish in a lake may depend on the average rainfall, the average temperature, the average number of fisherman, and the average amount of food, each of which in turn is affected by a number of other variables.  When we mush these all together, things tend towards a normal distribution, which lets us deal with most natural processes in a tractable way mathematically, giving us a universe in which many things of interest have well defined average values, because they're peak-y.

*Without getting too deep in the weeds, this is true assuming your distributions have both a finite mean and a finite variance.  Some power-law distributions do not have a finite variance, because they have what's called a "fat tail": basically, they don't converge to zero fast enough, so there's lots of stuff way out towards infinity.  If all your variables are like this, you're in trouble.  Luckily for us, the real world is mostly composed of things that have finite variance.

**As opposed to Ohio, which is high in the middle and round on both ends.

Fingerprint Scanners and Network Privacy Effects

Yesterday, I had some snark for the assertion that Apple using biometric identification in a consumer product amounted to then taking your fingerprints "against your will".  I also considered the ethical aspects of whether your neighbor's privacy choices affect yours.  But from a technical perspective, I find myself still very interested in Jacob Appelbaum's assertion that this will have an impact on overall privacy (or, specifically, his privacy) via "network effects," and found myself thinking through what this might mean.  What follows is a probably overly pedantic analysis of the idea of privacy network effects in general.

First, let's define what a "network effect" is in this context: technically, network effects of technology are ways in which the adoption of a technology by someone else makes that technology more or less valuable for me.  As an example of a positive network effect, e-mail is more valuable if more people use it, because I can reach more people using e-mail.  An example of a negative network effect is traffic or network congestion: the more people who use cars, the more traffic I have to contend with.  I think, technically, we would construe a network effect on privacy for the iPhone fingerprint scanner to be one in which adoption of the device by others reduces @ioerror's privacy if he uses the same device.  However, I think we can safely conclude that @ioerror won't be using an iPhone 5S, or if he does, he'll use a sharpie to disable the fingerprint scanner.  So, more broadly construed, we might consider network effects in which other peoples' adoption of the iPhone 5S reduces @ioerror's privacy, or even more generally, reduces the privacy of other people who don't use the phone in general.

It's important to distinguish this from simple consumer choice: there may be an overall reduction in privacy because of peoples' choice to use the iPhone 5S fingerprint scanner, but they may make that choice entirely based on considerations of convenience.  This is an important distinction because, in the absence of network effects, it means that there's effectively no moral angle to the fingerprint scanner: the very fact that a large market exists for such devices means that community standards accept such choices as valid*.

There are a few mechanisms by which we can imagine privacy network effects being propagated.  I think it's clear from context that the case that @ioerror is worried about is the normalization of biometric identification: including fingerprint scanners in phones which lots of people use will make people less more complacent about fingerprint scanners in general.  Is there evidence for this?  There are certainly a lot of cases of the public accepting lower privacy standards for specific purposes.  For instance, when TSA imposed full-body scanning at airports, a lot of people shrugged and walked through the scanners.  But, there was no obviously identifiable network effect: we didn't start to see full-body scanners replace metal detectors at federal buildings or schools (although it may be too soon to tell.)  It may be the case that there are downstream effects: have we seen a profusion of metal detectors in public places (ball games, emergency rooms, schools) in general?  Probably; I can't find statistics on this, but casual observation strongly suggests it.  Is there a case to be made that this is due to normalization of security technology into our everyday lives?  Again, very possibly.  But is that due to a network effect, or due to simply government policy and heightened media attention?  That is much harder to establish.

Perhaps a more compelling example is the profusion of sites that now let you log in using your Facebook ID instead of tracking logins on their own.  As such logins become more common, it's easier to shrug at the (very real) privacy considerations of linking your Facebook account to each additional site.  An important difference between these two cases is cost: metal detectors and full body scanners are expensive.  Software is cheap.  Which leads us to a second potential mechanism for network effects: By including such devices in their mass produced phones, Apple will effectively bring the cost of such devices down to the point where other phone manufacturers may start using them, and it may come to the point where it is difficult to buy a smart phone without one.  This, I think, is a much more easily demonstrated mechanism of network effect.  However, both are highly indirect: the adoption of the technology by party A does not directly impact party B's privacy: it's only through a very indirect set of policy, economic, and attitude changes that such effects could be propagated, and it's far from clear that these effects are even close in magnitude to the simple market demand for such devices.

Then, of course, it bears questioning: how would fingerprint scanners actually impact our privacy?  First, there's what I call The Strong Hypothesis:

I've updated the iPhone fingerprint scanner diagram with an important detail Apple left out. pic.twitter.com/k63flx8y6j
— Dylan Tweney (@dylan20) September 10, 2013

The Strong Hypothesis is that the NSA will gather fingerprints en masse from iPhones and other devices, then use them to create a national database.  Six months ago I would have rated this tinfoil-hat-silly.  But, of course, the revelations of the last few weeks make a lot of us look pretty silly for thinking that way, so it's no longer possible to simply disregard that possibility out of hand.

The Weak Hypothesis is that, for instance, the FBI will be able to subpoena your fingerprints from Apple in order to compare them against fingerprints they've collected, when previously they would have had no way of getting your fingerprints short of hauling you in.  Whether or not this could happen depends a lot on how the technology is managed, and it seems more likely than not that Apple will store the fingerprint data on the device in a way that precludes remote access.  But Apple has done stupider things before, and trusting their commitment to privacy is probably not a good strategy.

So, to close the loop, a network-effects-privacy-impact might look something like this: Apple's introduction of the fingerprint reader to the iPhone 5S lowers cost and social barriers to similar devices, and we start seeing fingerprint scanners not just on phones, but on laptops, cars, at the airport, and even at the gym.  Oh, wait...

*The same argument doesn't necessarily hold for things like cigarettes though: sale and consumption of cigarettes imposes externalities on people who don't consume them, in the form of second-hand smoke, and increased public healthcare costs.  Even if the sale of cigarettes is evidence that the community approves of cigarette consumption in and of itself, the effects on others have highly complicating moral effects.  This is why it's important to establish whether there are network effects in deciding whether there's a moral aspect.

Fingerprint Panic!

So, it sounds like the shiniest new iPhone will have a fingerprint scanner for security.  Bruce Schneier, naturally, has some interesting and relevant technical considerations which he voices, in particular, about what happens if Apple decides to store fingerprint data in the cloud.  But it seems like there should be a secure way to do this: nobody is realistically going to actually store an image of the fingerprint, not even on the phone itself.  Instead, they'll store a hash of some set of metrics derived from the fingerprint.  If you add salt to the hash, and the salt is stored on the phone, then you still need an unlocked physical device for the hash to be useful at all; the cloud-stored version is useless, just like a salted password file.

On the other end of the spectrum, there's this:

Allow me to suggest, for the paranoid, a few practical steps to be taken here to foil The Man:

1. Don't buy an iPhone.  Problem solved.
2. If you absolutely must play Angry Birds, turn off the fingerprint reader and use a passcode.  Better yet, use a non-numeric passcode.
3. If you don't believe that turning off the fingerprint scanner will foil the NSA's backdoor into your phone, try using a Sharpie to color over the fingerprint scanner window.

With respect to the network effects: what do you care if someone chooses convenience over privacy?  People do it all the time, with their Safeway club card, their credit card, their choice to go through the scanner instead of get a pat-down at the airport, etc.  Privacy is a very personal decision.  Some people crave it, and it's their constitutional right which I support staunchly.  But lamenting the "network" or "societal" effects of other people choosing security, convenience, fame, or money over privacy makes you little different than a church pastor denouncing the gay lifestyle because of the effect it will have on children.  Society as a whole is constantly making decisions about their personal trade offs of privacy versus convenience, and you can always go Galt and peace out to a cabin in the woods if the unwashed masses refuse to hear your speech.  You might want to give up tweeting if you go that route though: you give up far more privacy in practical terms through Twitter, Google, and Facebook than you would from a fingerprint scanner on your phone.

What I Did With My Summer Vacation

This was my third year attending the Burning Man festival in the Black Rock Desert in Nevada, although "attending" isn't really what you do at Burning Man; you are there to experience it.  It is a 60,000 person city that springs into existence for a week, with interactive art, events, and parties, all of which is either burned or disassembled at the end of the week.  Like a sand mandala, it will never be there in the same way again.  This year I took very few photos (if you'd like to see some incredible photos, see these by Jim Urquhart, a Reuters photographer who I met at last year's burn).  But I spent a lot of time in quiet contemplation and introspection, a lot of time meeting new people, a lot of time dancing alone and with others, and a lot of time hugging people and making new connections.  A short list of things I learned this year:

1. Not all of the pieces of my personality are well suited for all purposes.  I can be impatient, overly solution-focused at the expense of the feelings of others, and intimidating.  I often beat myself up over these "flaws", but I realized that these are not, in fact, flaws.  Every part of my personality, my humanity, is important, is a part of who I am, and is valuable.  Sometimes, you bring some parts of your personality to bear in certain situations, and sometimes you let other parts take a back seat, because it's the best place to get you from A to B.  But that doesn't mean that those parts of your personality are bad, they're simply the wrong tools.  Next time you think that by not saying something, you're being untrue to your authentic self, ask yourself whether you're really being inauthentic, or whether you're just choosing to utilize a different skill (patience, forbearance, acceptance) rather than the one that's easiest and comes most naturally to you.  And, then, when you're out with your friends later, speak unvarnished truth, be funny, and abrasive, and a bit loony.  They love you for it.  That's why they're your friends.
2. Solid state hardware can, in fact, succumb to dust and heat and fail in the desert, the same way things with moving parts can.
3. Don't try to write your greatest hits anthology while you're on your second album.  Don't think about how you're going to tell your story to people while you're still in the middle of it.  Experience it mindfully and fully, and absorb it.  Let the emotion, the joy, the sadness, the frustration, the come to you, experience them without judgment, and remember them.  Wait until it's time to tell the story to put them into context.  You never know what's around the next bend that could change the whole thing.
4. PVC is light, but it's not an excellent structural building material, and can buckle and fail easily and spectacularly under strain, especially in high heat.  Always over-engineer anything built with PVC.
5. Open yourself to spontaneity.  As Woody Allen said, half of life is just showing up.
6. You've got a lot of love to give.  Give freely of it, and it will come back to you in the form of joy.  At the burn, it's easy to connect with others, because people are open to the experience.  It's harder back here in Palo Alto, but less hard than you might think.

L'Shana Tova.

I Will Show You Fear In a Handful of Dust

No posting until after Labor Day.  I'll be off the grid in the desert for a little over a week: no phone, no e-mail, no SMS, and no internet.  Although I will be bringing a small Raspberry Pi-based server with me, so technically, there will be WiFi, but only on the local intranet that I'm setting up, with no uplink to the outside world.  See you when I get back.

Theory Thursday, with bonus Metaphysics: Gödel's Incompleteness Theorem

Gödel's Incompleteness Theorem is one of those things that makes pop science writers go gaga.  And, I'm not going to be contrarian: it's profound, it's meta, it's paradoxical, and it has a lot to say about what we can accomplish as human beings.  But, at its core, it's really a programming problem, albeit one with very interesting implications*.

In the 1920s, in the shadow of The Great War, David Hilbert set out to do something monumental, something that would bring together mathematicians from all over the world, in a grand challenge: to formulate all of mathematics rigorously, and provably, starting from a single set of axioms.  It came to be known as Hilbert's Program, and the name has an unintentional level of acuity: it really boils down to a system of encodable rules which one can use to generate mathematically true statements.  Then, by utilizing these rules in a fully deterministic fashion, you can prove mathematical theorems.  By iterating with those rules, you could theoretically enumerate all possible mathematically true statements: it's a programming language for generating All Mathematical Laws!

Except...for that "theoretically" bit.  Turns out, it's not true.  Kurt Gödel spent a lot of time trying to prove the completeness and self-consistency of such systems, but eventually wound up proving just the opposite: he demonstrated that you could construct the following types of statements in this mathematical language:

1. Statements about the provability of other statements (e.g., Statement X is unprovable)
2. Statements that reference themselves (e.g., This sentence has N characters.)
3. By combining the two, you can construct statements that reference their own provability, e.g.:

This theorem is unprovable.

This is similar in construction to the classic form of Russel's Paradox ("Does the barber who shaves everyone who doesn't shave themselves shave himself?"), but with a twist: In this case, if the statement is provable, it's a paradox.  However, if it is actually unprovable, there's no paradox.  Therefore, it is both unprovable but also true.  Hence, Gödel showed that the system is incapable of proving a demonstrably true statement, and cannot be complete.  As you might guess, there's a lot of overlap between this finding and computability theory, and it can even be expressed in terms of the latter, which brings it strongly into the realm of computer science.

But, one of the things that has always stuck out to me about the Incompleteness Theorem is what it has to say about what it is we can possibly know about the world, and what this means about our perception of reality.  I was raised in a moderately religious household, and eventually, in the course of my own explorations, became very observant, went to Jewish summer camp, studied the Torah, and tried, to the best of my ability, to puzzle out what it was God wanted out of us.  But, over the years, the relentlessly empirical life of the academic scientist took a toll on my faith, at first subtly, but like a river grinding out the Grand Canyon, its effect was profound over the 10 years I spent as a physicist.  And, one day, I woke up and realized: there was no room in The Mechanical Universe for an omniscient, ominpotent being**.  Plus, the psychological and sociological reasons for us to invent such a being were too clear to ignore.  It was just too facile a solution to the difficult problem of What Do I Do With My Life.

But even so, I've never been able to call myself an atheist, because, in spite of all my book larnin', I know what I don't know.  On the one hand, at the most granular scale, almost everything I've ever learned about the physical world is premised on an unprovable hypothesis: that the things that we observed yesterday are a good guide to what we will observe tomorrow.  It's unprovable, because the only evidence that we have for the inductive principle is the inductive principle itself: it has worked well in the past, so it should continue to work.  I have no problem using the inductive principle as a guide, as a way to decide which insurance to buy, or how to spot a mu meson.  But in matters of spiritual life or death, can we really count on something so flimsy?

And then, on the grand, cosmic scale, we have the Incompleteness Theorem: I know that there are things out there that are true, but can't be systematically proven.  In fact, Gödel proved that there's an infinite number of them.  If that's true, I can prove and prove and prove until my hearts' content, but I can never be sure that I haven't failed to observe something which could change my entire understanding around.  There's just too much truth out there, and not enough time.

So, what are we to make of the universe?  Nothing is provable, and there's an infinity of unprovable true things that we'll never know.  Given all that, can we really rule out, conclusively, the existence of something outside of this mortal coil?  And so ends my catechism.

*However, while it's really cool, it's doesn't have something to say about everything: Carl Woese once asked me to write a paper on what the implications of the incompleteness theorem were for biological evolution, and Douglas Hofstadter happened to be giving a colloquium that week.  I went down early before the colloquium and accosted him over coffee to ask him his thoughts on the matter.  He considered it briefly, and then said, "I don't think there are any."

**The Mechanical Universe used to be shown on PBS in Chicago when I was growing up, and I found every single episode, from Newton's Laws to Relativity, to be absolutely mesmerizing.  It had a strong impact on my decision to study physics.

Is Earlybird or X-Pro II better for your wads of cash?

I was at a conference once on technology and law enforcement, and there was a gentleman presenting some sort of Twitter-monitoring technology on behalf of IBM.  Because I was at the booth next to him, I got to hear his pitch dozens of times, describing a specific scenario in which a small time drug dealer was coordinating with clients over Twitter.  As the day went on, the size of the fish he had caught got bigger and bigger, and the tone of his pitch more and more frenzied, until finally he jumped the shark and told someone that "nowadays all crime is coordinated over Twitter."  Hah, Twitter is so 2012!  In 2013, all crime is coordinated over Instagram:

Aspiring rapper's Instagram photos lead to largest gun bust in New York City history

Also in today's news: Supreme Court Justices don't use e-mail, which probably makes it difficult for them to opine on things like net neutrality, whether metadata qualifies for Fourth Amendment protections, and a host of other technologically relevant issue that are currently defining our society.  Technological innovation is occurring at an ever faster rate, and has clearly passed the rate at which Supreme Court justices die, which means that this will be a permanent state of affairs.  They may begin to understand e-mail around the time when they are asked to rule on the legality of remote-scanning your flying car's hyperdigicube transponder.

The Three P's of Big Data: Plumbing, Payment, and Politics

In a panel discussion at Georgetown Law School, I discussed some of the barriers to actually using data, at scale, in a large organization, to drive change.  I lump these barriers into three categories, which I call The Three P's: Plumbing, Payment, and Politics.  We're a long way from being able to just throw all the information into a computer, ask it a question, and get an answer out, and the things that stand between here and there are my raison d'etre.  This weekend, The Sunday Times dipped its toe into Big Data commentary, with a contrarian take titled Is Big Data an Economic Big Dud?, in which the author asks, bluntly, What's taking so Goddamn long?

[T]he economy is, at best, in the doldrums and has stayed there during the latest surge in Web traffic. The rate of productivity growth, whose steady rise from the 1970s well into the 2000s has been credited to earlier phases in the computer and Internet revolutions, has actually fallen. The overall economic trends are complex, but an argument could be made that the slowdown began around 2005 — just when Big Data began to make its appearance.

This is a worthy topic of discussion, but the article is a mish-mash, lumping things like streaming video, text messaging, Cisco, and analytics companies under the title "Big Data", and asking why this "Big Data" hasn't made an impact on the economy (while tacitly admitting that the economy has been in such flux since 2005 that asking what the impact of any particular development is amounts to asking why the wind direction didn't change when you passed gas.) In some sense, the very fact that the New York Times has covered it means that it may have already peaked, because once the Sunday Times becomes aware of a technological development, it's reached the level that geriatric crossword-puzzle enthusiasts might actually be dimly aware of it. Witness:

Other economists believe that Big Data’s economic punch is just a few years away, as engineers trained in data manipulation make their way through college and as data-driven start-ups begin hiring.

(Emphasis mine.)  I don't know where the Times gets their information about high tech hiring, but judging from this sentence, it's probably from the classifieds page of the New York Times.

Paul Krugman, as usual, has a more nuanced and interesting set of questions:

OK, we've been here before; there was a lot of skepticism about the Internet too — and I was one of the skeptics. In fact, there was skepticism about information technology in general; Robert Solow quipped that “You can see the computer age everywhere but in the productivity statistics”. But here’s another instance where economic history proved very useful. Paul David famously pointed out (pdf) that much the same could have been said of electricity, for a long time; the big productivity payoffs to electrification didn't come until after around 1914.
Why the delays? In the case of electricity, it was all about realizing how to take advantage of the technology, which meant reorganizing work. A steam-age factory was a multistory building with narrow aisles; that was to minimize power loss when you were driving machines via belts attached to overhead shafts driven by a steam engine in the basement. The price of this arrangement was cramped working spaces and great difficulty in moving stuff around. Simply replacing the shafts and belts with electric motors didn’t do much; to get the big payoff you had to realize that having each machine powered by its own electric motor let you shift to a one-story, spread-out layout with wide aisles and easy materials handling.

I think it's reasonable to suggest that Big Data™ writ large may never (and probably won't) have the same impact on the world as electricity or The Intarwebz.  But the question of what may be slowing its impact is something on which I am well qualified to comment.  I will make more extensive posts about this in the coming weeks and months, but in terms of the three P's, the plumbing pieces are still in their infancy, and are hugely labor intensive: getting data from unstructured sources, like documents, images, and video is nearly impossible without significant cost and data-specific tuning.  And, even making full use of structure data (databases, XML, etc) is painful because of the amount of context and integration required.

More importantly, though, the politics of data-driven decision making are subject to the politics of any paradigm-shifting change, particularly ones that have the impact to show that the Chief Executive Officer is wearing no clothes.  Large organizations have been Doing It Wrong™ for so long now that convincing them to take the implications of their data seriously is an uphill battle against politics and entrenched interests.  In theory, the market should correct and allow data-driven decisions to lead if they provide a competitive edge.  But in practice, pop-culture management books and pop-science provide managers with confirmation bias that their gut instincts are all they need to run a giant organization, data be damned.  And, even when a CEO wants to make a large scale change, he has to push it through using hired thugs just to get any traction.

Essentially, we're still at the infancy of the Big Data movement, both in terms of its ease of use, and its cultural adoption.  The potential to monetize how we use data for decision making is huge, but there are still significant obstacles to leveraging its potential.

Data Sharing Done Right

By my hackweek teammate Andy I: http://www.marketsforgood.org/beyond-alphabet-soup-5-guidelines-for-data-sharing

Read this if you ever plan to share data publicly.

I Do Not Think This Word Means What You Think It Means

Poor Peter Shih.  His rant about the things he hates about San Francisco (ugly girls, homeless people, and not being able to park his car) generated so much derision that he had to take it down, and clarify that it was "satire".  This is a common defense of bigots and jerks everywhere.  Here's the thing: satire is a tool used by the weak against the powerful.  It's a way that someone with no power can even the odds, through wit and insight.  Here, Shih is the powerful: he's YCombinator backed male with a car and a job making fun of women, the homeless, and bicyclists.  If it were funny, he'd at least be able to claim he was poking fun at himself.  But it's not funny.  There isn't even a hint of self-deprecation.  A vitriolic stream of misogyny and self-entitlement isn't called "satire", it's called being a jerk.  And so ends my catechism. 

Getting (Real) Things Done

I'm a technologist. I like technology. Specifically, I like bringing real change to real problems, using technology. I spent 10 years as an academic, engaged in, as my father liked to say, the examination of the frontiers of human knowledge using a magnifying glass. But, by far, the thing I loved more than anything else as an academic was to build things, both machines and software. Instead of relying on paper notebooks, I rolled my own web-based lab notebook, in a combination of object-oriented Perl and JavaScript (this was a decade ago, and to this day, my colleagues insist on endorsing me for Perl on LinkedIn, which I think is about as valuable as being endorsed for trepanning.) In the end, one of the reasons I left academia was precisely because I liked building things a lot more than I liked discovering things of marginal value, and one of the reasons I love my job is because the things I build have immense value.

I put all this forward as a form of credentialing, though, to give you my bona-fides as a techno-optimist. Because what I really want to say is: the vast majority of the problems in our world will not be solved by apps, hackathons, or data jams. They can only be solved by talking to people.

Let's be honest: most engineers and nerds hate talking to people. I met my wife through a dating website, because I am gripped with terror at the idea of introducing myself to a woman in a social setting. I book my restaurant reservations using Open Table because restaurant hosts at upscale restaurants are some of the most odious people to talk to on the planet.  So, faced with a problem, we will opt to write software to solve it. Witness, for instance, this article about StopBeef.com, a “matchmaking app for murder mediators”, as PandoDaily puts it:

A high profile shooting on Mother’s Day that injured 19 people got hacker Travis Laurendine thinking about how homicide was impacting the city. The National Civic Day of Hacking was fast approaching, so he decided to try “hacking the murder rate” for the New Orleans version of the event.

Laurendine was friends with Nicholas through the rap community, and he turned to him for advice. “I was like: you know this inside and out, what do you think would help?” Laurendine said.

Nicholas told Laurendine about a recent mediation experience he’d had. Two guys were beefing and carried guns. The conflict had escalated over time, and it was starting to look like one would try to take the other out. But Nicholas knew both of them, and one of them asked him to help settle the issue.

An intervention was staged, and the two beefers showed up with their entourages to Nicholas’ restaurant. “They both really wanted to stop the beef,” Nicholas remembers. “I told them it didn’t make sense for them to die over nothing.” The entourages agreed, and with their friends (and a local celebrity) asking them to stop, the beefers were able to end the disagreement without looking like they had backed down.

When Laurendine asked Nicholas for hacker app ideas, Nicholas relayed this story and said, “You need to do something like that! You can hit this app, log in, tell them who you’re beefing with, and someone from the neighborhood can come spoil.”

…

After submitting the stopbeef app to the government, Laurendine was picked as one of the 15 top “Champions of Change” nationally in civic hacking. He’s headed to the White House to receive recognition tomorrow.

I don't mean this as a dig against StopBeef.com, which I'm sure has provided some really valuable outcomes, and may have even saved lives (although the linked article doesn't cite any specific success stories.)  But, it's pretty clear that the key element here is people talking to people. And I have to wonder: if Laurendine et al had spent the time learning about the community, talking to people, doing volunteer work with local tutoring organizations or Boys and Girls clubs, or finding a way to actually engage people directly, would it have had a higher impact than the equivalent amount of work spent trying to get their CSS to look just-so?

Social problems are particularly prone to this kind of wishful thinking. When I first started looking for volunteer opportunities when I was in graduate school, I reasoned that there must be dozens of organizations looking for someone to build them a web site, or a database, or maintain their PCs, and that I could bring my highly valuable skill set to bear on these problems, to great plaudits. What I in fact, discovered, is that volunteer organizations need people to do things for other people, not for machines. And, the less interesting and high profile the work, the more they need it. Habitat for Humanity had so many church groups, schools, and companies calling on them to volunteer that the local chapter didn't even bother to call me back. The San Francisco Food Bank has a several week waiting list if you want to come in and help sort canned goods. Even the Martin De Porres soup kitchen in my neighborhood sometimes has too many people doing dishes when I come in to volunteer. In grad school, I wound up driving a van for a program called Mothers Too Soon. On Thursday nights, in the winter, I would pick up 15 – 17 year old African American girls from the outlying areas of Champaign-Urbana, and drive them and their young children to a local church. There, I would sit and read journal articles for a couple of hours while they had dinner and counseling, and then would drive them back home. It wasn't particularly fun, but that was the point: nobody else wanted to do it, which was why there was a need. The only skill I had that it utilized was a drivers license, and a willingness to help out.

But, techno-optimism bedevils the corporate world just as much. 99% of my job could be done from the comfort of my home office. I can video teleconference, Webex, call, SSH, and pretty much do all of my job functions without ever so much as standing up. But I hop on a southbound train from San Francisco to Palo Alto every morning at 7:19AM like clockwork, and I fly back and forth to DC and New York (as I am doing right now) several times a month. And I do it because, in spite of all the things I can do with just a computer and a wifi connection, nothing important ever gets accomplished unless I'm there in person. You can commit code to the git repo for years on end, but it's not until a person is actually using it that you've accomplished something.

For example. I have occasionally planned trips to client sites not because I needed to get something done there, but because being there, in person, forces the client to make sure I have something to do when I get there. So, I tell them that I'm coming in to configure the networking, and they need to have the servers racked by Wednesday. The truth is that, if they just plug in the Remote Access Console, I can configure the rest of it from Palo Alto, without the need to fly anywhere, or even put my pants on, for that matter. But if I did that, I could spend the next month or more calling, e-mailing, and sending carrier pigeons, for all the good it would do me; the servers would get racked when the IT team had nothing else to do, which is not a condition that has ever prevailed since the invention of the abacus (“Abacus help-desk hours are between 2AM and 4AM on the first new moon of the harvest season. If you require assistance outside of these hours, please consult our rice-paper scroll for self-help and Frequently Asked Questions.”) Is this a huge waste of jet fuel, money, and 3-ounce bottles of shaving cream? Yes. But it's less wasteful than flushing three months of a six month software pilot down the drain while I wait for the servers to get racked. And, while I'm there, I can stop by the analysts' desks and chat with them about their work, their workflow, and what kinds of actual, real-world problems keep them up at night. I can take the clients' project lead out to lunch and let them dish face-to-face about who's trying to block the roll-out, and what kinds of changes we might want to make to turn their opinion around. I can ask for an introduction to the head of a different department or someone who works at a different firm, and see if they're having the same problem, and whether we can try to solve it for them too. And I can hit up my college friends who live in New York and DC, toast heartily to our health, and hear about their latest project, whether or not it has any bearing on my work or not, just to expand my mind a bit.

Software and the internet has changed our world, and in most ways, I think it's been for the greater good. People like to throw digs at Facebook for its triviality, but I landed my current job because I connected with someone on Facebook who turned out to be friends with someone who I had met years before and not seen since. So, we reconnected, and started talking, and when my postdoc was over, he suggested I interview.  Not only that, but in the ensuing years, he's become one of my best friends, and was the officiant at my wedding. I can confidently say that my life would be less rich and less happy without Facebook. But the good that Facebook, and JDate, and GMail did for me were that they brought me into contact with people who mattered. The internet has lowered the barriers to human interaction, but it is always the human interaction itself that makes the world a better place. Next time you want to write an app to solve a problem, ask yourself: could I do more by volunteering, or community organizing, or even just reading and learning about this problem, than I could by debugging jQuery? Or am I secretly serving my own interests by doing something I enjoy or learning a new marketable skill, or maybe just avoiding the fact that, for better or for worse, the problems in our world are really, really hard?

Something You Will Never Hear in a Spy Movie

"Just run it through one or two databases.  No need to go overboard."