jducoeur

I haven't posted many diary entries lately -- that's mostly because I've been pretty cranky, and I never like posting in such a mood. That said, November was an illustration of a principle I've often found: that the universe tends to keep my life interesting, so long as I pay attention to the opportunities it gives me. So the next few entries will be catching up on what's been going on, as life not only became a bit of a roller-coaster, but one gaining speed pretty rapidly. Part one is about work and stuff.

When last we left our hero, he was trying to emulate a good British stiff upper lip, and halfway managing it, but let's get real: losing the CommYou project hurt like hell. While I'm really excited to see all of these ideas I've been talking about for years being put into practice, I had kind of been hoping to be the one to make the splash there. It's a blow to the ego to realize that, while I am very good at startups, I'm not very good at the entrepreneur-CEO role.

The adjustment took about three weeks, from the point at which I realized that Google Wave had washed CommYou out to sea (yes, yes -- get used to the bad metaphors), before really coming to terms with it. There have been two main things that have eased the transition:

• Google aren't stupid: they know that, if Wave is really going to succeed, they can't just build a tool -- they need to incite a whole new infrastructure. Part of giving that any credibility is fostering not just open protocols, but reference implementations of those protocols that aren't under Google's thumb. So I've wound up diving into that end of things. In particular, there is a small group of folks who are pretty serious about seeing this not only happen, but happen right. Most are pretty clear about the weaknesses in Google's initial implementation, but understand the potential in the idea. I think most of us tacitly agree that the objective here is to foster competition, and that means making a real stab at beating Google at their own game.
  
  It's pretty chaotic right now -- among other things, everyone prefers different server implementation languages -- but I believe we'll gradually pull it together. So CommYou or something like it may yet be back eventually, in somewhat different form.



• I no longer have an excuse to spend a day a week on CommYou, though: my business plan is shattered. This means I'm now working fulltime again. The silver lining in *that*, of course, is an immediate 25% boost to my salary, which doesn't suck. Indeed, the effective boost is more than that -- I've been paying a huge amount of money to run the CommYou server, which I'm about to claw back. So the total rise in my take-home is probably more like a third.

So my ego is bruised, but it definitely makes my life easier.

There *was* a sense of being adrift, and not knowing what to do next, which isn't a sensation I enjoy: that was the source of much of the crankiness. But remember what I said above, about the universe keeping things lively? Well, we'll talk about that in the next couple of entries...

I've been putting off writing this entry for a week now, but I really should just wrestle with it.

I've spent much of the past two weeks playing with Google Wave, experimenting with it, and talking with people about it. Right now, I'm reading into the nascent project to build an open-source version of it. (Which Google claims to support 100% -- they're trying to develop a whole new Internet infrastructure here, and are fully aware that nobody's going to take it seriously if they have the only implementation.)

A lot of people have been asking, "What the heck *is* Wave?". The answer is complicated, and I'll get into it more in a later series of posts, but the short answer is, "CommYou".

It's really pretty startling, not least because I'm fairly sure that this isn't quite what Wave set out to focus on. When you look at what it is good at, Wave is principally a co-editing system. It's a generalized infrastructure for allowing people to work on Stuff together, live in realtime as well as more gradually. It is basically taking a lot of ideas that have been floating around for many years (not least, in the game industry), and applying them in a pretty rigorous and generalized way.

But of course, CommYou has always been largely about that, just specifically for conversation. I've talked a lot about "multi-modal" or "semi-realtime", and this is exactly what I've been trying to describe: conversations that speed up nicely when multiple people are present, slow down when zero or one are there, but generally keep going and stay relatively distinct and on-topic. The latter point is essential: like CommYou, Wave is about *accomplishing* stuff together, which is what distinguishes it from a random chat room. The ethic of Wave, like CommYou, is that off-topic threads should (ideally) get taken to a separate wave, so conversations are broken down by community *and* topic.

They wound up arriving at bloody near exactly the same answers I did, enough so that I've occasionally found myself wondering if they hacked into my server and stole my ToDoList. Heavyweight threads: check. Semi-realtime interactive conversation: check. Summary window that live-updates when a conversation changes, and shows when and how much changed: check. UI that efficiently pops open an new-message window below any random message: check. Tracking which messages you've already read, so you can catch up on new stuff easily: check. Metadata and plugins: check. Etc, etc -- while it's not identical, it is *very* close to what I've had in my mind when trying to describe CommYou's design.

Make no mistake, Wave is damned cool -- there are a lot of nay-sayers, but they're mostly missing the point: they're taking an early alpha (which is what we've got now), full of bugs and lacking in features and integration, and saying "there's no there there". But having spent the past two years thinking about this stuff, I see *exactly* where Google is going with this, and it's a game-changer. We're at the beginning of a new model of communication and collaboration, probably more important than the rise of IM, nearly as important as the Web itself. (And vastly more important than Twitter and its micro-blogging ilk, which will gradually be subsumed into the Wave-like systems over the next 2-4 years.) Many people have complained that there's nothing *new* in Wave, which particularly misses the point: as I've long said about CommYou, it's not about coming up with a single new-and-revolutionary idea, it's about taking all the existing communication models and combining them *correctly*.

But the sense of "squish" is palpable, and I'm still wrapping my brain around it, and processing the emotions. Mind, Wave doesn't yet do *everything* that was in the CommYou design, not even everything that is already in it. But there is nothing in CommYou that would be *hard* to do in Wave, so I expect them to catch up to me fairly fast, and they already do a lot that I hadn't even contemplated: Wave's purpose is similar to CommYou's, but it is even more grandiosely ambitious.

Most importantly, they've almost casually swatted down my business plan. The idea was always that CommYou would be a cheap-or-free consumer service, as a loss leader for selling integrated high-quality conversation to websites. But Google has already announced that they will be giving away site embedding, and have even begun to demo it. So I'm left without the possible future income stream that justified all that work and present forgone salary.

I may be ambitious, but I'm not dumb, and I don't like tilting at windmills. It's one thing to grab at the brass ring of a startup, knowing that the odds are weak. It's quite another to try to compete against Google giving something away for free, especially now that they're starting to realize what they've got. I'm not going to win this one, at least not in the way I've been thinking.

So the existing CommYou plans are looking to be toast. I'm crushed, but part of surviving in the startup world is being good at crying in your beer for a little while, then picking up, moving on and getting the hell over it.

All that said, I don't think it's all going to be entirely for naught. Those two years of work have taught me a huge amount, not least about how a system like this can and must work. The open-source project is still pretty early, and I have a huge amount I can contribute to it. I've still got the passion for this project (which, remember, started as me trying to build the conversation system I've always wanted), and a lot of relevant knowledge. So I'm diving in head-first, trying to catch up to all the developers who have been in the loop for five months already while I was in denial about it.

We may even see the return of CommYou. We'll see how it plays out, but as I said -- there is a lot that Wave doesn't yet do. They've made great progress on the conversation problem, but they're missing a lot yet and they have made some decisions I don't necessarily agree with. (In particular, that realtime conversation is character-by-character Talk style, not message-by-message IM style: that's cool, but I think it's a mixed blessing.) And they've scarcely touched the community side of things, which has always been as important to me as the conversation part.

We'll see. I probably can't justify doing this as a true part-time job any more, so it'll have to turn spare-time, and I don't have enough of that. But there's a strong temptation to help the open-source project get up to speed, and then revive CommYou as an experimental variation of Wave that ignores the co-editing part entirely and focuses on conversation and community. If so, I'm going to do it right this time, as a pure open-source project end-to-end, using seriously cutting-edge tech. Programming geeks out there, think about whether you might be interested in playing.

BTW, for those who asked for invitations: sorry, they're mostly gone. I got a lot more people asking than I had invites, so I've been doling them out gradually and carefully, but I'm down to a single one left. (They mostly went to people who have been particularly helpful on CommYou, unsurprisingly -- those are folks who I know have a clue about this stuff, and have been demonstrably passionate about using it.)

Thanks to fairdice, I've now got a Wave account, and am starting to read into it. My initial reaction is that it's not really trying to do *quite* the same things as CommYou, although it's possible that it is close enough that it will suck away all the energy in the space. That is, the focus is notably different, but the overlap in ideas is non-trivial -- they clearly came to some of the same conclusions I did about how an interactive system ought to work. We'll see; I'm trying not to get excessively stressed about it, although the competition inevitably is on my mind.

Really, though, I need to do some experimenting here, to get a better idea of how it works, and Wave is fundamentally about multi-user, particularly real-time. The high concept (which is much clearer now that I can get my hands on it) is that they've re-envisioned email as a real-time, interactive process. The UI is reminiscent of Gmail, but the "messages" are live, interactive objects that everybody involved can update and modify in real-time. It doesn't appear to be principally a conversation system, or a community system -- rather, it's a co-editing system that happens to have realtime chat embedded in it. (Seriously realtime: the chat is part of the realtime Wave, so you see everyone editing their chat messages as they do so.)

So: who wants to play? If you have Wave, and are interested in playing with it (especially tomorrow, when I'll have more time), please reply or drop me an email. If you are really interested, and want an invite, tell me, but please don't do so *too* casually: I only have a modest number of invite codes, so I want them to go to folks who are going to really kick the tires hard. It appears to be particularly relevant and useful if you have stuff you want to do collaboratively in real-time online -- probably doubly so if you are already used to GMail and Google Docs...

Thanks to mindways for the link to this fine little essay, about the fact that modern society, too often, separates activities too neatly into the buckets of "career" and "hobby", missing the fact that there are vocations that don't necessarily match either well.

This point comes quite personally to me right at the moment. Indeed, it's been the main focus of the negotiations over my new job -- I think they've been a bit puzzled over my demand for a four-day week. The CEO even said, when I was interviewing with him, "Never confuse your career with your hobbies". And that's true, but kind of misses the point.

For me, the CommYou project has become something of a calling. I've been forced to recognize that it's not necessarily a career: while I might someday make money off of it, I can't count on that. But it's by no means a "hobby", either -- there's a connotation of casualness about "hobby" that doesn't fit here. While the damned project is moving far, far slower than I'd like, it is something I *have* to do, just as much as a writer must write or a painter must paint.

Hence the four-day work week. If I tried to put CommYou purely on the back burner, I'd never be able to do a real day job: it would consume and frustrate me. So instead, I'm employing my long-practiced skill at separating the parts of my life: I can keep CommYou in a box if and only if there is a decent-sized box to put it into. Four days a week belong to the job that pays real money; one goes to the calling. I suspect it's actually going to work better than the consulting gig did -- by having clearer lines of what time belongs to whom (as well as physical separation of *where* I do each, and on what computers), I should be able to *focus* on both better than I've been doing, which will help me really get going on CommYou 2.0...

Say to yourself ten times each day, "Raw Integers and Stringified Integers do *not* make acceptable IDs for your model objects. You should be using strongly-typed ID objects whenever possible."

Yeah, I know -- I really should know better, but it was so *easy* to just use the ints from the initial database representation (and derive type from context) that they wound up laced through the system. But while they haven't caused any Horrible Disasters yet, it's probably only a matter of time, and they will probably fail scalability eventually. So I'm currently doing penance by going through the whole system and replacing them with properly-typed ID wrappers everywhere that I can find them, which should make the next steps a *lot* more solid...

Another of those disciplines of programming. It's hard for a good programmer to not write unnecessary code -- the temptation to gold-plate, and add extra functionality for completeness' sake, is very strong.

What's even harder is *removing* unnecessary code. Sometimes, you've built up a system that is complete, but you've found that you're no longer using all that functionality. It's painful, but the right thing to do is often to delete the no-longer-exercised code. Not only does it present danger (if it's not being tested enough, it may introduce bugs), but it can make refactoring harder in the long run. So it is often easier and better to re-create it later when and if it ever becomes relevant again, rather than keep it on the books without a clear use case.

(Yes, I'm doing exactly this right now. Some of the mechanisms from the old Facebook UI just aren't being used in the new UI, and should probably be restructured if they ever do come back. So the low-level support for them should be snipped...)

It is oddly comforting to receive the daily emails of who has tried to break into commyou.com. While it's not great that the attempts are happening, it does show me that everything's up and running, and working hard to keep the system running on an even keel...

It is occasionally fascinating to step back, watch myself program, and critique it.

Today's observation is that I am still scared of database programming. I'm *doing* it, mind you, but I don't have that innate comfort with it that I do with nice in-memory OO programming. There's a lingering sense that I'm impersonating a real DB programmer.

This is particularly coming up today because I am realizing that I need to do a slightly nasty database refactoring for CommYou. (Currently, Communities are linked to Persons -- that is, a Person is a Member of a Community. I am realizing that this is just making my life harder: Communities should be linked to the Identity that they are imported from, which would make synchronization vastly easier and more efficient.)

In principle, this is straightforward, and it's the sort of thing I do in ordinary code without even blinking. After all, the IDE protects me from most of the simple errors, and the test harness from the rest. But refactoring the DB is just plain *scary* to me. I understand the process just fine, and can outline the steps I need to take in detail with no difficulty, but I find myself procrastinating about actually *doing* it.

I can give some objective reasons for that. For one, the IDE does *not* protect me from errors in this very well. And while it's very straightforward for me to test new data created after this change, testing the migrated data comprehensively is much harder. It would be easy for subtle bugs to sneak in.

But mostly, I suspect I'm just still a bit intimidated by the prospect, because I haven't done it much. Nothing for it but to press through, make the changes, and convince myself that the world didn't end...

It always surprises me nowadays when I think, "Surely this tool must exist on the Internet", and it doesn't seem to.

One of the topics that came up due to today's LJ outage is, "Is CommYou ever going to be a social network unto itself?" To which the answer is "no", I'm deliberately avoiding doing that. CommYou is about Conversation Dammit, and I don't want to mix it up with the 97 other features that people assume a "social network" should have. And I am bound and determined to open it up to *all* social networks gradually, so I don't want the temptations that come with it becoming one itself.

That said, I *do* occasionally contemplate the idea of building a sort of "micro-network", sitting over on the side as a separate system, that would serve as something for people who don't already have a compatible network. This would be a real and for true "social network" and nothing else: a system that lets you log in and manage friend lists. It would have *no* other native functionality, but it would probably serve as an OpenSocial container. (Because hey -- why not?) It might also allow outside manipulation of the network, so that external apps could, eg, provide UI to manage your friend list within their own context. Moreover, it would be heavily metadata-based, so that an external app could attach information to both objects and relationships. Basically, it would be the true social-network-as-platform.

To my astonishment, this doesn't seem to exist yet. It seems obviously useful to me, and has great potential for end users: it would be modular, so that you had exactly the functionality *you* care about. It would be built entirely on the open protocol stack (OpenID, OAuth, OpenSocial, PortableContacts) -- indeed, its entire raison d'etre would be that stack. Ideally, it would be constructed to be a network-of-networks, so that many instances of it could be run, and users could just cross-connect across them. (Or indeed, should be able to link to anybody via OpenID.)

Hmm. Does this exist? Does anything even close to it exist? It really should, but it looks like every social-network implementation gets the core idea mixed up with big bundles of extraneous ad-hoc features.

Damn -- I *really* don't need another project right now, but I abhor open market niches. And the core would be super-easy to write: all the complexity would be in opening it up completely. This is sorely tempting...

A couple of people have mentioned that, while LJ has come back up again, CommYou login is still failing. Far as I can tell, this is because while it's up, LJ is limping pretty badly. Loading your homepage (which is a key step in the authentication process, believe it or not) is so *slow* that we're timing out when we try to log you in.

I assume that this is transient, and that LJ will get its act together soon. In the meantime, my apologies for the interruption. And yes, this has sparked a discussion on how best to deal with such outages...