Data, Big and Small
Nov. 2nd, 2012 08:31 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
jducoeurThis week's notable link from LinkedIn is this delightful roundup of Five Trendy Open Source (Big-Data) Technologies. It goes through some of the newer hot products -- not stuff that's gotten mature like Hadoop, but newer concepts like Storm, Dremel, and Hana. Worth a read if you're doing any sort of big data at work, especially if you are in any way influencing architecture -- the enterprise world is driving advances in data processing at *remarkable* speed.
That said, it makes amusing reading for me right now. Everybody is talking about Big Data as the way to make money from enterprises. So I guess Querki might best be labeled the first truly serious Small Data project I've seen in a surprisingly long time. I'm explicitly not going after enterprise at all, at least not yet. (In a few years, if Querki is successful with consumers, we'll probably spin off a business-focused subsidiary. But first things first.) Indeed, for the time being I'm going to strictly limit the number of Things you can have in a Space, to somewhere in the tens-of-thousands range -- not even pocket change by Big Data standards.
Querki's underlying theory is that, while the Big Data problems are sexy to computer scientists and businesspeople, they have relatively little to do with the ordinary person on the information superhighway. Normal people are always trying to deal with *little* problems, involving only thousands, hundreds or even tens of things to keep track of. They don't care about lightning-speed processing of billions of records -- they care about being able to *easily* manage the small, everyday problems of the real world. And right now, they are looking sadly neglected.
I'm really quite enjoying this: there's nothing more exciting than finding a problem that nobody's dealing with well. Let's see if we can start a Small Data revolution, while the giants are all focused on the mountains in the distance...
That said, it makes amusing reading for me right now. Everybody is talking about Big Data as the way to make money from enterprises. So I guess Querki might best be labeled the first truly serious Small Data project I've seen in a surprisingly long time. I'm explicitly not going after enterprise at all, at least not yet. (In a few years, if Querki is successful with consumers, we'll probably spin off a business-focused subsidiary. But first things first.) Indeed, for the time being I'm going to strictly limit the number of Things you can have in a Space, to somewhere in the tens-of-thousands range -- not even pocket change by Big Data standards.
Querki's underlying theory is that, while the Big Data problems are sexy to computer scientists and businesspeople, they have relatively little to do with the ordinary person on the information superhighway. Normal people are always trying to deal with *little* problems, involving only thousands, hundreds or even tens of things to keep track of. They don't care about lightning-speed processing of billions of records -- they care about being able to *easily* manage the small, everyday problems of the real world. And right now, they are looking sadly neglected.
I'm really quite enjoying this: there's nothing more exciting than finding a problem that nobody's dealing with well. Let's see if we can start a Small Data revolution, while the giants are all focused on the mountains in the distance...