Web API Lifecycles and Hypecycles

I've long been fascinated by the explosion of APIs over the past years, captured by the excellent ProgrammableWeb site.

Curious about how categories were evolving over time, I mined ProgrammableWeb's index for interesting patterns. I focused primarily on categories with at least 50 APIs, dividing them up into semesters from the second half 2005 to the first half of 2010. One important detail to be aware of: the PW index includes the last modified date of the API, not its creation date. So think of these graphs as a measure of activity in a particular category. For example an API may have been created in 2006 but if it was updated in 2010 it will count towards that last bar on the graph.

So what's hot? Social APIs, unsurprisingly, show a feverish activity: every site is busy creating or expanding their offerings in this space.

Enterprise APIs too are seeing a lot of movement.

Encouragingly, so is Shopping. A harbinger of an economic turnaround, or just wishful thinking? :-)

What about up-and-coming API categories to watch? Of the ones with over 30 APIs Travel and Utility have seen the most movement over the last year and a half.

Here are the remaining 13 categories with 50 or more APIs. Other strong performers include Government, Telephony, and Tools. Categories in relative decline? Reference and Video. 

DTerm: Useful Omnipresent Command Line for OS X

Found this sweet free utility called DTerm recently. It enables you to pull a context-aware pop-up that you can use to run command line utilities from whatever program you're currently using. What do I mean by "context-aware"? DTerm will automatically change directories to the one your program is currently in. Moreover, for those of us using multiple spaces, any programs you run from DTerm will open their windows in your current space.

Here's an example. Say I want to package up a bunch of images. Simple: hit Shift-Cmd-Return to invoke DTerm, its window overlay on the Finder's, and I can then run a "tar" command. That's it. I could even stay in DTerm and copy pics.tgz to a different drive, or scp it to another server.

This is a very useful tool. Here are a few other things you can do with it:
  • Quick calendar: "cal" will display this month's calendar, hit Shift-Cmd-C and you'll have it in your clipboard (cal 2010 will give you this year's calendar)
  • Starting TextMate: Typing "mate ." from a Finder window will open TextMate in project mode in the current directory
  • Comparing files: Select 2 files in the Finder, run DTerm, type "cmp" or "diff" then Shift-Cmd-V to paste the names of the files you selected into DTerm
  • MD5 checksum: Select the files you want to sum and run "md5" + Shif-Cmd-V
  • Info on all files in the current directory, including hidden ones: "ls -al"
  • Create a series of folders in the current directory: much faster to type "mkdir foo bar foo/bar" than to use the Finder
  • Quick lookup info on a domain: "dig google.com"
  • Want your mac to read you something? Select some text, copy it, invoke DTerm and type "say" followed by pasting the text surrounded by quotes
  • Byte, word, line counts: "wc" and the file(s) you're interested in

Not all these examples require DTerm's features but having a terminal window at your fingertips, without needing to switch context, is very useful.

And it's another reason to make better use of all those command line utilities!

Hat tip to @azaaza for the pointer.

Staying in Sorrento: Relais Regina Giovanna

This summer our family spent four days in Campania, Italy, visiting Mount Vesuvius & Pompeii, the island of Capri, as well as the beautiful Amalfi Coast. We didn't want to stay in Naples, so we chose Sorrento for our base of operations. 

Across the bay from Naples, Sorrento was still centrally located for our purposes: easy public transport to/from Naples, close to Pompeii, and Amalfi. Just as importantly Sorrento is a small town with great atmosphere. The hotels, though, made you pay dearly for that atmosphere!

We settled on a different hotel, the Relais Regina Giovanna. Focused on the burgeoning agritourism movement, the Regina is a large renovated farm house located on a few acres of olive and citrus groves outside of Sorrento. It's a very relaxing setting. You can enjoy the terrace with its view of the bay, the gardens, and even a little private (pebble) beach for some swimming and sunning.

The rooms were well appointed and very spacious, with high ceilings. We took two rooms and still ended up paying less than one room would have cost us in a nice Sorrento hotel. Thankfully all rooms had air conditioning  and the cool terra cotta tiles were a nice bonus.

Would we stay there again? Definitely!

Travel tips:
  • One downside of the Relais: no internet in the rooms. There is a laptop you can share with guests. We just plugged in the ethernet jack into our own laptop when we needed net access
  • Second downside: lots of TV channels... but in Italian only. That said, you didn't come to Italy to watch TV, did you?
  • Don't bother eating at the Regina's restaurant: it's expensive and we found the food disappointing
  • If you walk up the road, you'll find a cheaper options including a little deli with decent but simple panini (sandwiches) and good family-owned restaurant (Ristoria Kalimera). Moreover, if you ask nicely, they'll give you the key to their wifi so bring your laptop and check email while you wait for your food to arrive
  • Getting to the Regina is a bit of a pain if you don't have a car (the hotel has a large parking lot BTW). A taxi from the train station costs over 20 euros, which can get expensive very quickly
  • Our solution: take advantage of one of the downtown tour buses that bring you to sample Limoncello (the local liqueur) and see Capri. They depart every hour, costing 7 euros for adults with children riding free. The bus passes right by the Regina. The first time we took the tour and ask the driver to drop us off at our hotel on the way home. After that we just tipped him a couple euros and he'd drop us off on the way out of Sorrento. And we'd get to hear him enthusiastically sing opera every trip! :-)

 

Diseases make you Dumber... and Smarter?

Couple of articles in The Economist caught my eye recently. The first on the effects of toxoplasmosis and human behavior, the second on the link between disease and intelligence.

Toxoplasma gondii is a parasite whose lifecycle alternates between rodents and cats. When it infects rats and mice it lodges itself in their brains and causes them to behave in an erratic, risk tolerant, manner. It may even make them attracted to the smell of cats. Once the infected rodent is eaten by a cat, the parasite eventually ends up in its feces, to be ingested by a rat. Repeat ad infinitum...

It turns out toxoplasma has this effect by producing dopamine which then acts on their hosts' nervous systems. What then is its impact on humans? Some studies show a correlation between toxoplasmosis and schizophrenia. Others, a higher level of road accidents in infected drivers (there's that increase in risk tolerance again). But "some researchers go further and propose that entire societies are being altered by Toxoplasma".

"In 2006 Kevin Lafferty of the University of California, Santa Barbara, published a paper noting a correlation between levels of neuroticism established by national surveys in various countries and the level of Toxoplasma infection recorded in pregnant women (a group who are tested routinely). The places he looked at ranged from phlegmatic Britain, with a neuroticism score of -0.8 and a Toxoplasma  infection rate of 6.6%, to hot-blooded France, which scored 1.8 and had an infection rate of 45%. […]

To repeat, correlation is not causation, and a lot more work would need to be done to prove the point. But it is just possible that a parasite’s desire to get eaten by a cat is shaping the cultures of the world."

The second article reviews a study comparing national IQ and a country's disease burden, i.e. the "disability-adjusted life years lost caused by 28 infectious diseases". They found a 67% correlation between the two and though they tried to find other causes, they kept coming back to the impact of disease on IQ.

The article is worth reading in its entirety. As with the case of toxoplasmosis, correlation is not causation, but if true, it's a key finding.

"If [the researchers] are right, it suggests that the control of such diseases is crucial to a country’s development in a way that had not been appreciated before. Places that harbour a lot of parasites and pathogens not only suffer the debilitating effects of disease on their workforces, but also have their human capital eroded, child by child, from birth."

So if we have evidence of diseases' deleterious effect on humans, couldn't other diseases make you smarter, stronger, or healthier? Wouldn't this give them a better chance at long term survival? 

In May of this year, scientists presented evidence of just such a effect: a bacteria linked to increases in learning behavior.

The researchers found that that mice fed live Mycobacterium vaccae "navigated the maze twice as fast and with less demonstrated anxiety behaviors as control mice" and speculated that "that creating learning environments in schools that include time in the outdoors where M. vaccae is present may decrease anxiety and improve the ability to learn new tasks."

All this makes me wonder how prevalent such effects are in our lives. Could it be that these little symbionts have shaped our evolution unbeknownst to us? And how would we know?

Here's one way: let's see if our collective IQ decreases as we all use increasing amounts of anti-bacterial soap! :-)

Defcon Day Two Highlights

If there was a theme to the presentations I saw on Saturday, it's that as a technology is increasingly closed, its security decreases exponentially. The solution is sunlight: bring the products and their vulnerabilities out in the open. Yes, it does mean running the risk of vulnerabilities becoming known. But it's the only solution we've found that actually produces fixes. An obscure, insecure product helps only the black hats.

Insecurity Engineering of Physical Security Systems: Locks, Lies, and Videotape by Marc Weber Tobias, Tobias Bluzmanis, Matt Fiddler
A good example of this was a talk by three locksmithing experts. Though their preamble was too long, the main part of presentation was fascinating. They showed how to break five different types of locks: from a re-keyable mechanical lock to a fingerprint reading lock. All were defeated with simple attacks, some so simple that they beggared belief. The fingerprint reader, for example, has a standard bypass lock in case the battery runs out of the reader... With the insertion of the paperclip in the bypass lock, it opened like a charm. Wired has a great writeup, including videos.

Extreme-range RFID Tracking and Practical Cellphone Spying by Chris Paget
Chris gave two great presentations. The first showing how to read RFIDs at ranges of a couple hundred feet. The second focused on how to build your own GSM base station. Both talks were full of technical information and Chris did a good job at clearly walking us through the steps he'd taken. The GSM talk was fascinating. In essence, it is surprisingly easy not just to create your own base station (cost ~$3,000) but it's also trivial to spoof an existing carrier such as AT&T. When audience cellphones connected, Chris' fake tower would instruct them to drop encryption (a fact that handsets don't advertise to their users BTW) enabling the capture of phone conversations. While this currently only worked for outbound calls, it was still an impressive demonstration. One solution? Switch to 3G, it's a lot more secure than 2G.

We Don't Need No Stinkin' Badges: Hacking Electronic Door Access Controllers by Shawn Merdinger
This pres was a good example of the evils of security by obscurity. Electronic door access control is ubiquitous throughout the business world, yet these systems are usually run by building management. These folks may know a lot about physical security, but not information security. The result? Vendors supplying shockingly insecure systems that are never patched. Shawn focused on a product by S2 Security but claimed many competitors also had flaws such as insecure default configurations, full access to nightly database backups, an unprotected URL to reset the device to factory defaults, leveraging vulnerable software components, etc. etc. etc. Basically, if your company's door access controller is on an (internal hopefully!) network, you had best isolate it as much as possible. To my knowledge Shawn hasn't uploaded his pres anywhere here are the four S2 CVEs he submitted.

You're Stealing It Wrong! 30 Years of Inter-Pirate Battles by Jason Scott
A lighter look at the history of pirate groups and much much more. Scott, a computer historian and Defcon regular, gave a highly entertaining presentation and provided a wonderful trip down memory lane for many an audience member (myself included!). We gave him a standing ovation at the end of his speech (something I've rarely seen at Defcon). Jason, make sure you come back next year. Oh, and if you, dear reader, have old computer stuff you want to get rid of... Don't! Send them to Jason instead.

Malware Freak Show 2: The Client-Side Boogaloo by Nicholas J. Percoco and Jibran Ilyas
These two gents from Trustwave demo'ed four examples of malware found at client sites over the past year. Five years ago, they said, attackers focused on "smash and grab": find a vulnerability, exploit it, get as much info as you can, get out. Nowadays attackers are writing custom targeted malware that stays under the radar, allowing them to slowly infiltrate their victims' networks. Not sure what their sample size was but they claimed that on average malware infiltrates a site for 156 days before being detected. That's a long time.

Jackpotting Automated Teller Machines Redux by Barnaby Jack
Arguably the most talked about presentation at Black Hat and Defcon, Jack blew the doors wide open on ATM security. There are a lot of articles about his talk on the net, so I won't repeat it here. Jack basically found a number of vulnerabilities in these Windows CE devices (yes, Windows CE), including a remote exploit allowing him to reprogram the ATM. One of the most dramatic moments of his pres came when, in a matter of seconds, he popped open an ATM (cabinet master keys are apparently trivial to obtain), inserted an SD card with his own code, and power cycled the machine. Once the ATM booted you can see what appeared on the screen below and watch the video to see what happened next!

Defcon Day One Highlights

While a few of Friday's talks contained little new, original, or useful information (disappointingly the former Facebook CSO's talk was particularly inane), the majority of the presentations were interesting. A few were eye-opening. Here are some short summaries of my favorites.

Crawling Bittorrent DHTs for Fun and Profit by Scott Wolchok
Scott presented his research on creating a very comprehensive database of Bittorrent Distributed Hash Tables. Suffice it to say that his approach and findings will unfortunately prove very useful to record companies if they aren't already using these techniques. File sharers beware!

The Law of Laptop Search and Seizure by the EFF legal team
This talk focused on what law enforcement can and can't do (but may still try to get away with!) when seizing your laptop. There were a lot of details presented... orally. EFF, why no presentation? A few key points from my notes (oh, and in case you hadn't realized: IANAL!)
  • In general law enforcement can't just take your laptop and search it, your rights are protected by the fourth amendment
  • If law enforcement does want to search your laptop they need a warrant or you need to fall in a exception category such as: you have a public share on your computer, you're sharing via P2P, you've given consent, there's immediate danger that you might destroy the info, etc.
  • You can revoke consent at any time (i.e. if you first let law enforcement look at your laptop, you can change your mind)
  • If there are multiple users of a computer, any one of them could give consent, though courts have recognized that this consent only goes so far as the authorizing user has access (though the forensic tools they use make no such distinctions... Beware!)
  • All searches that occur at a border are considered reasonable. No suspicion is needed for any searches to occur, nor is a warrant needed (in other words: your rights go out the window!)
  • You cannot be forced to give over your encryption keys, courts have found that this is a fifth amendment right, and the gov't hasn't appealed this decision
  • Remote Computing Services, e.g. online backup or file sharing (like the very useful Dropbox). It is very easy for the gov't to get this data. They just need a subpoena, sometimes not even. Probably cause isn't required, since searching these cloud-based files often is how the gov't shows probable cause. They're not required to notify you within a reasonable time frame
  • Electronic Communication Services, e.g. online mail services like gmail. Your data is only protected for the first 180 days. After that the gov't doesn't need a warrant to get access to this info. However the gov't doesn't think this law applies to emails you've read, drafted, and sent. This is being appealed and the DoJ is fighting it. The EFF, ISPs, and others are trying to get a better law passed, maybe next year (the sooner the better!)
  • The EFF's advice: POP your mail, don't leave it in the cloud, and avoid online backups if possible

Lord of the Bing: Taking Back Search Engine Hacking from Google and Bing by Rob Ragan and Francis Brown
The most interesting talk of the day. These guys have taken google search engine hacking to a whole new level. Very creative. Sadly I haven't found their presentation online but the tools they wrote are. One of my favorite sections focused on combining google hacking with custom searches into a massive RSS feed for real time updates of vulnerable sites crawled by google. I'm sure we haven't heard the last of this...

Weaponizing Lady GaGa, Psychosonic Attacks by Brad Smith
Brad is an excellent speaker and by far the most entertaining of the day. He discussed the uses and misuses of psychosonics: the generation of (generally undetectable) sound patterns designed to alter a target's state of mind. One of the funniest parts of his speech came when he listed the top 10 sonic torture songs... :-)