jducoeur

Back from Pennsic, and back at work. I'm spending this week on a variety of smaller enhancements and bugfixes that have been rankling me for a while.

One of them is kind of entertaining. I've been saying for a long time that Querki Is Not A Social Network, and I still mean that: between FB, G+, LJ, Twitter and so on, there are more than enough social networks out there, and I would rather that Querki work *with* those existing networks instead of trying to duplicate their functionality. Frankly, I've long been annoyed at the trend for every freaking service to create its own little special-snowflake "social network": social networks are an underlying infrastructure that should be widely shared.

(Yes, the big names are annoyingly commercial, evil and privacy-violating. One problem at a time -- the principle is correct, but we need some more disruption in the social-network space. But I digress...)

The thing is, though, I've started to realize that Querki *does* have its own "social network", just one that is very different from the big players. This is the network of shared acquaintance through shared Spaces. That is, if you and I are both in Space Foobar, that is a non-trivial connection. More importantly, Querki should make it easy for me to invite you to join one of *my* Spaces, similarly to how I would do with a Facebook friend or something like that. It's essentially an *implicit* social network, more like the one that exists in Facebook Groups than the actual FB friendlist.

I'm implementing the first draft of that now (frankly, I'm tired of having to look up peoples' email addresses over and over), but I'm trying to find the right term for this relationship. "Friend" is too strong -- this network is defined as "People I share Spaces with", and many of those aren't really "friends". But "Acquaintance" is kind of bloodless. (And has too many syllables.)

Any suggestions?

And *that* is why I love Querki so. Having invented a cocktail an hour ago, it just occurred to me that I should start writing down my not-infrequent experiments, so I can remember what works. I began to write it up as a simple LJ entry, and then said, "Waitaminnit -- Querki *is* the right tool for this job".

And while I have to be honest that it's nowhere near perfect yet, Querki did the task as intended: I spent less time building my first-draft Mixology Space than I did coming up with the recipe -- it took only a couple of minutes to create the Space, even in the current crude state of the system. (And even after the medium-strong drink that inspired it in the first place.) That's encouraging: at least once you know Querki, it is beginning to achieve its primary goal, that creating a new Space for a particular need is lightning-quick.

So I present the Falernum Flash -- an off-the-cuff invention based on my discovery about 90 minutes ago of the existence of Falernum Bitters. I'll probably keep updating this Space as I come up with new inventions. (And eventually will turn it into an RSS feed, once I finally write the What's New plugin...)

*Whew*. I've been putting this off for years, but it needed to be dealt with before incorporation.

The Querki project has been going for a *long* time -- I wrote the original prototype, ProWiki, back around 2003, and had named its successor "Querki" and begun to design it by 2005 or so. I actually registered querki.net in 2007, but there was a snag: a squatter had already snatched up querki.com, apparently solely on the basis that it had six letters and was pronounceable. And since Querki was just an interesting personal project until mid-2012, it wasn't worth paying the extortion.

But since the project became real in 2012, that's been bothering me. The price for the domain had gone all the way up to $1200, which was deeply pain-inducing, so I've been reluctant to pay for it. But I know that after incorporation the price would probably go through the roof -- far as I can tell, they often raise the price to the tens of thousands once they start detecting people accidentally typing the domain. And the danger of a competitor picking up the domain is pretty serious once the product gets real.

So I finally bit the bullet yesterday, and paid the ransom. (Which had risen to $1300 over the past two years.) For some reason, they had fixated on calling it QuerKi.com -- my only guess is that that is two words in some language -- but that's irrelevant, since case doesn't matter to domain names.

On the plus side, HugeDomains (the squatter) proved smooth and easy to deal with. IMO their line of business is unethical at best, and really ought to be illegal, but at least they seem to be entirely above-board in how they conduct it. (I did some research online, of course, and was pleasantly surprised to find no serious complaints about them.) They cited a specific price online instead of haggling, charged my credit card the correct amount, turned around the process email within minutes, and made it fairly easy to transfer the domain over to PairNIC, where all my domains except jducoeur.org live. (This brings me up to 18 domains; I really need to let a few of them expire when they next come around.) Knock on wood, they seem to be people you can do business with.

So querki.com is now legit, but for the foreseeable future is just going to point to www.querki.net. The .net version will remain the primary domain for various reasons, not least that I don't want to pick a fight with the well-established Quirky.com. (I believe I'm clear of their trademark by both the spirit and letter of the law, but it would be stupid and pointless to piss them off. Ironically, quirky.net is *still* in the hands of squatters.) I have some idea what the .com version might eventually become, but that's a few years down the business plan...

... I am frantically *coding* for it.

Story of my life these days: when Lochleven recently re-org'ed, I volunteered for the newly-created job of Quartermaster -- keeping track of the Stuff. A fine application for Querki, but I quickly decided that easy photo import was critical for it -- if you're going to do inventory management well, you need to be able to take pictures of the stuff you're inventorying.

Yesterday was, blessedly, highly productive. With the aid of no less than three open-source libraries, I am now able to take a photo from the browser in my phone, upload it to Querki, reduce the size to Web resolution (my current contention is that most Querki use cases don't require more than 1 megapixel, so I'm clamping it to that size for now), and upload the results to Amazon S3 for long-term storage and web serving. It's actually rather neat: until recently, I had assumed I would need to write apps in order to do things like take pictures and geolocations, but HTML5 turns out to let me do a great deal in-browser.

Now to actually get all of this tied into Querki per se, so that you can simply treat a photo like any other property value. Still a lot of code to write, but once we're done, it should be one of Querki's cooler features: you'll be able to attach photos to Things quite easily, which turns out to be relevant to a lot of use cases.

But if you see me wandering around camp with my bloody cellphone out during Pennsic, that's why. It's going to be a bit of a working vacation for me, figuring out everything that we have and taking pictures of it all...

This is related to a topic I tossed out in the Querki Dev Journal yesterday. The more I think about it, the more important it seems to be -- it's a subtle detail, but likely to affect the contours of Querki's overall social structure. So I'll ask the larger crowd here.

The issue is privacy settings. To summarize the above article, Querki allows users a lot of fine-grained control over security, but to make it usable we're going to need to sum that up into a few easy-to-use options: you pick one of these, and then tweak the details if you care. I expect relatively few people to do much tweaking.

The question at hand is what the *default* option is when you're creating a Space, Public or Private. For reasons I describe in this comment, I don't think we can entirely duck this decision -- at the least, we're going to wind up subtly influencing user choice, so I'd rather do that deliberately instead of by accident.

So the main question is, when you create a Space, which of these options is the default (or at least, on top of the list):

Public: The Space is (mostly) publicly-readable and commentable, a la a typical LiveJournal.

Private: The Space is (mostly) not readable by non-Members, a la a typical private Facebook group.

(There will also be "Hidden" -- you can't even tell this Space *exists* unless you are invited in -- but I do not believe that should be the default.)

Closely related and still important: should we be explicit about this default, or should we require that the user creating the Space make a decision? (That is, should one of the radio buttons be checked initially or not?)

Opinions solicited, at least in this quick poll. I'm very much on the fence here -- it's a less easy issue than it looks at first glance. Comments and thoughts also welcomed...

[Poll #1974791]

A week or two ago, I mentioned the crowdfunding project for ProtonMail, a secure email platform based in Switzerland. Today's little fillip to that story is that PayPal apparently (in that way PayPal does) peremptorily cut off their funding for no good reason. Of course, PP claims that this was just a mistake, but it fits too closely with too many other stories I've heard.

All of which cuts a bit close to home. Sometime next year, Querki is going to have to start taking memberships, and that is inevitably going to require getting in bed with *some* sort of solid payments processor. And the truth is, in a lot of ways I'd actually *like* that to be PayPal: I like the general design of the service, and have been a member for many years. (And while, yes, their security may not be 100% airtight, it is *vastly* better than the credit card system in general.) But I just don't know whether I can trust them enough to use them for a key business service -- they have too much history of screwing over their customers on a whim.

I really do wonder whether PP is really as clueless as they seem to be, about how much damage they have done themselves through too many arbitrary decisions. They've wound up opening the door to competitors (including Amazon, as well as a host of other players) who would be *more* than happy to steal their business. And it's exactly the startups who tend to choose a processor and then stick with it for a fair while who they are driving away...

[Posting this here, to get a wider range of inputs than the dev journal.]

Querki is, by pretty much any standard, a complex and sophisticated piece of software. I'm following the KISS principle as far as I can, but the fact of the matter is that it is *not* all that simple. There are a large number of levers you can pull if you want to make it really sing.

That said, most people aren't going to need or want most of that functionality. So I am planning on trying something of a crazy experiment: having distinct "levels" of Querki. This has nothing to do with paid vs. free, or anything like that. Instead, it's purely a matter of letting the users choose the experience they would like to have. As currently designed, I am planning on three levels, with roughly this functionality:

Easy -- the UI exposes only the most core functionality. This level is designed to let you build relatively simple Spaces, using the core Types and the most common Properties. A few basic Functions show up, but not many. When Apps come along, you'll be able to use those. The target audience here is the true general public, who (I believe) will mostly want to use Apps, tweak them a bit, and *maybe* create very simple Spaces themselves. They aren't likely to create QL expressions of more than 2-3 clauses. There might be some extra hand-holding.

Standard -- this UI exposes all of the commonly-used Properties and Functions. (Probably several times as many as Easy.) Some of the common meta-Properties will be visible. The ability to create Apps will be visible. The ability to create Model Properties will be exposed, and in general most of the UI functions will be visible. The target audience here is folks who want to try some serious tinkering, but don't want to be presented with the full gamut of bells and whistles.

Advanced -- the "this goes to 11" setting, mainly intended for sophisticated programmers who are building complex Spaces, and App developers. It exposes all of the non-internal Properties and Functions. (Again, 2-3 times as many as in Standard.) Exposes all of the meta-Properties, letting you do things like create your own User Value Types. Lets you build fully-custom CSS. Etc.

These levels aren't necessarily hard and fast -- I suspect that we will need the ability to temporarily swap into a higher level to do specific things -- but the intent is to help people find their comfort zone. For most typical users, I'd like to provide them with a UI that is clean and focused, and not full of crap that they will never, ever want to use. For people like me and

mindways, I want to have all the tools at my fingertips when I need them.

(Note that taming Advanced is an important part of this. There are a large number of Properties that I expect relatively few people to use, and I would like them to *not* be in everyone's faces all the time.)

So the first question at hand is, am I on crack? This seems like an obvious approach to me, but I honestly can't think of a system that does anything quite like this. It's a very coarse-grained way of doing things, but that's kind of the point: I don't want the typical user to be faced with a wall of switches to turn on and off, I just want to be able to define a few core profiles that fit expected usage patterns.

The second question is, does the broad notion of the levels seem right? They match my intuitions of the right buckets, and in fact much of the system is *already* tagged with these three levels. (This has been in the works for over six months now.) But I'm kind of going by gut instinct here.

The third question is, are the names appropriate? The overall term was originally "User Levels", but I decided that sounded pejorative. Internally, the code currently is named "Skill Levels", which doesn't sound quite as bad, but I'm not sure it is right. (I'm halfway tempted to just call them "Flavors", or something like that, to remove any sense of one being better than the others.) The names of the three levels above are the best I've come up with so far, attempting to make each one sound reasonably positive (and attractive to their target audiences), but they are by no means final.

Opinions welcomed. Taming Querki's complexity is one of the really *big* challenges for this project, and this seems like a straightforward way to take at least a first stab at it. I'm under no illusions that we're going to get it all right to begin with, but I'd like to put a stake in the ground before we start getting a non-trivial number of people involved, and I'm trying to puzzle out what a best guess would be...

The past three weeks have been basically a hash as far as Querki is concerned. Between the five-day trip to visit in-laws, the six-day trip for Origins, and last week's vertigo problem (which seems to have finally resolved itself on Sunday, knock on wood), with only a few days between each, I'd simply gotten out of the habit of getting things done. The result is that, instead of coding, I spent all that time designing -- that is to say, massively over-thinking everything. And when I started to implement the over-thought designs, I simply tied myself in knots.

On Monday, I finally cracked that: instead of trying to get every detail Just So and building from the bottom up (which was incredibly unsatisfying, since it meant that nothing was actually functioning), I finally grokked that I was screwing up, and went back to my usual development style: top-down, getting skeletons functioning as quickly as possible and then fleshing them out. The result is that I've made more progress on Notifications (the current project at hand) in the past three days than I have in the past month.

And getting things working has shone a bright spotlight on the places where my designs, while fine ideals, are too elaborate for the first draft. Get it sufficiently functional first, enter UX bugs as necessary where the workflows need improvement, but keep moving. (Today's realization was that, yes, the display of Notifications *should* be a popup pane the way that FB and G+ do it, to avoid the workflow interruption of a separate Notifications Page. But getting the bloody things functioning comes first. When I have time to go do some serious jQuery hacking, I can make the workflow prettier, but I don't need to spend those two days on gold-plating right now.)

The moral of the story (which isn't news, but I need occasional reminding): don't navel-gaze. Design is good, but only to the point where I broadly know how it's supposed to work. Once that is done, write out the Tasks and start slamming them in. The resulting code winds up every bit as good, and it's one heck of a lot faster...

I am often scathing in my opinions about how badly the major social networks deal with issues of Identity -- in particular, their stupidly coarse-grained view of the world, which often ignores the fact that many people would like to be able to easily use multiple distinct online identities. (Work vs. Home vs. Play vs. Kink, etc.)

That said, the Querki project is reminding me that doing this stuff right is pretty hard. Consistently distinguishing between a User (an actual, real-life person, accessing Querki on their desktop or phone), and a potentially unlimited number of distinct Identities for each of them, requires constant, careful thought. Doing it *well*, so that Identities don't "leak", is especially tricky.

Mind, I'm not promising HIPAA-grade protections at this point, much less life-and-death security: it's infeasible for me to do that level of security auditing any time in the next year or two. But that *is* the long-term goal. I should be able to have a Facebook account, multiple Google profiles, an LDAP profile from work, a Twitter account, a native Querki login, an LJ login, and so on, all connected to the same Querki account, and I should be able to easily say that certain of those accounts aren't visibly the same person.

(Why? Because I'm trying to not build Yet Another Bloody Social Network. I would like folks to be able to use their existing flists from the social networks inside Querki, and have it Just Work. But that requires dealing with the fact that those flists aren't necessarily fungible: I might well be trying to keep them distinct. Querki should facilitate that, insofar as we can.)

The problem pervades almost everything. Take my current project, for a typical example: implementing Notifications. The trick here is that a Notification (a System Message, a Personal Message, a Space Change Notification, whatever) is sent from an Identity to another Identity, but is actually *delivered* to a User. That is, I should be able to easily see all of the new Notifications from all of my Spaces, regardless of which Identity I am using in any given Space. And when I reply to one, it should automatically come from the appropriate Identity, without me needing to think about it manually.

(Yes, there are nasty edge cases if I want to have multiple distinct Identities within the same Space. For the moment, I'm simply not allowing that -- it opens up all *sorts* of sock-puppeting abuses, and needs to be thought through very carefully.)

I don't think I actually know any service that has ever actually tried to do this right, although it wouldn't surprise me if a few exist. Making it usable is going to be a heck of a project. Mind, I'm not tackling the thorny UI issues yet -- so far, we only have the native Querki logins, and we don't yet have the ability to link those together. But if I'm ever to have the slightest hope of accomplishing this, I have to get the data structures and communication right, and if we're not going to have accidental breaches the system has to play completely fair internally: the knowledge of how Identities relate to Users is tightly controlled internally, and most subsystems don't have direct access to it.

It's a fascinating project, and it does give me a *little* sympathy to the social network companies. It doesn't require malice to not want to deal with this -- the simple truth is that, if you don't build it in from the beginning, with a crisp distinction between your concepts, it's probably nearly impossible to do it well...

A quick pointer for folks interested in UX/UI who aren't already following the Querki Development Journal: I just posted a question looking for opinions, on what is the best default UI for choosing between unpredictable collections of "choices". If you have an opinion, please come chime in...

Howdy, all. My Google-fu is failing me, so I'm wondering if anybody has or can find me some leads.

I'm currently implementing "Conversations" in Querki -- basically, every Thing can have Comment threads associated with it. The trick is displaying them properly. I don't want to display the Comments as part of the Thing's own page, for a couple of reasons. (To speed up load times, to avoid the work of loading the Conversations if I don't care about them, and to separate my concerns.) But I don't really want them in a totally separate page -- they pertain to this Thing, so I'd like to be able to see both the Thing and the Conversations at the same time.

So I believe what I want is a "windowpane" jQuery control. Basically, at the bottom of the page there would be a small tab, showing the number of comments and maybe a highlight if there are comments I haven't read yet. When I drag or click on that, it opens up the bottom of the page, and loads the comments into it, so that (on a larger screen, anyway) I can see the Thing in the top half, and the Conversations in the bottom half, each scrolling separately.

The question is, does a good jQuery control like this already exist? The ones I've found so far each have at least one major problem:

-- only deal with the side of the window, not the bottom;
-- are hard-coded to just deal with menus;
-- don't have flexible enough tabs; or,
-- have unclear or unacceptable (eg, GPL) licenses.

I'm sure I could write this sucker if I had to, but I'm hoping to avoid the work. Anybody have any recommendations of a good open-source control along these lines? Thanks...

[Continuing Friday's deep programming burble, and figuring out how to use Gists instead of typing the freaking code directly into my entry. Hopefully this will all work...]

So when last we left our hero, he had just gotten his first Monad working, and there was much rejoicing.

Then I tried a more interesting use case, and it all fell apart. The problem is a well-known one in Functional Programming circles, which I knew perfectly well from working with other for comprehensions in Scala, but hadn't quite internalized yet: Monads Don't Compose. That is, you can build very powerful comprehensions using a single Monad, but as soon as you try to mix apples and oranges, the compiler chokes on them.

So here's a further burble, going into this problem, and how I'm working around it.

The example in question is _showLink, the Querki Function that receives either a local or external Link, and turns it into a conventional HTML <a> tag. The original code (based on my first pass at the Invocation class) looked like this:Example 1 -- original _showLink code

Yes, that's an inconsistent mess, hence the desire to clean it up. So I did the same exercise as last time, asking myself what I *wanted* it to look like, and got this:Example 2 -- first attempt to rewrite _showLink

Note the three lines marked as Problems. Those are places where what I really *want* is to embed a different Monad in the same statement.

Remember that the whole point of InvocationValue is that it's a nice, consistent Monad that I can simply lace through a for comprehension -- each line transforms it, and if an error happens it just skips the remainder of the processing and returns that error.

But Problem 1 is that "inv.contextElements" -- I want that to take the received context, turn it into a *list* of contexts, and process each one of those. That's a snap for the List Monad -- but I don't have the List Monad here. Similarly, Problems 2 and 3 are functions that returns Option[something] -- that is, they either return a value or not. Again, you can easily have a for comprehension of Options -- but you can't mix them in a comprehension of some *other* Monad.

In the end, I decided that the solution was to cheat. (Really, I suspect that I wound up implementing a poor man's version of scalaz's Monad Transformers, but I don't understand those at all well yet, so I simply dealt with the 98% case.) Specifically, I decided to follow Querki as my model.

The thing is, Querki does *not* treat its Collections as nice, neat Monads -- and now that I hit this problem, I was reminded of why. Monads are beautifully elegant so long as your basket is made up entirely of apples -- but Querki isn't nearly that neat. I specifically want my users to be able to mix Optional (the Option Monad) with ExactlyOne (the Identity Monad) with List (the List Monad, duh) easily, so it intentionally squishes out the differences. This is *horrifying* from a pure category theory point of view, and means that some use cases won't be so elegant -- but it means that 99% of Querki code is ridiculously obvious and simple.

So I rewrote InvocationValue to take a *sequence* of values, instead of exactly one. The original version expected that an InvocationValue[T] would take exactly one T if there wasn't an error. Now, it takes an Iterable[T]. If there is exactly one, that's an Iterable of length 1; if there are many, there are many; if there are None, it's just empty. The code comes out like this:Example 3 -- resulting InvocationValue

Finally, to get _showLink working the way I wanted, I added the .opt() and .iter() methods to Invocation. These simply take any Option[T] or Iterable[T], and transforms it into an InvocationValue[T], so that they can be used inside the same comprehension. So my calling code comes out pretty close to what I was originally hoping for:Example 4 -- resulting _showLink

Much better -- with just a little tweaking, I can now use Options and any sort of Iterable inside my for comprehensions. The result, I believe, will be that hundreds of messy, complex functions inside Querki are shortly going to be reduced to relatively straightforward (and consistent) for statements...

[Yes, I know, it sounds sordid. I couldn't resist, because even programming virginities are intimidating the first time. The truth is, this is a deeply geeky programming post, for the serious engineers. This is a fairly long burble -- I encourage the curious to ask questions. The experienced functional programmers will mostly just nod sagely, recognizing a newbie getting past a hurdle, but those who are still trying to get a handle on Monads may be interested in how I got through it.

NB: this is getting into fairly advanced Scala -- levels A3/L2 in the standard rankings. Note that you don't need this sort of stuff to *use* Scala most of the time, just if you want to write really powerful libraries and frameworks. I'm painfully aware that posts like this tend to lead to the common "But Scala is *hard*" whinging online, and it isn't actually true. Scala is *powerful*, and using that power to its fullest *is* a bit hard. But if you just want to write routine application code, it's as easy or easier than Java. This post is basically an example of how an Architect *makes* it that easy for the application engineers.]

Querki is chock-full of Functions -- that is, the code defines a lot of Functions that users can use in QL to transform their data. There are dozens of them now; there will be hundreds in the medium term, and it wouldn't astonish me if that eventually climbs into the thousands as we add more specialized App Libraries to mix in.

To date, writing those Functions has been laborious, and rather boilerplate-filled, which has irked me for a long time. I realized quite some time ago, in a vague way, that this was the sort of problem where Monads are supposed to help, but for a very long time I didn't grok them well enough to make that meaningful. But today I crossed the Rubicon, and wrote my first Monad. To my great satisfaction, it worked correctly right off the bat. So I think I'm finally starting to get it.
( Deep dive under the cut, showing my work for extra credit )

[Basically a programming diary entry.]

So about a week and a half ago, I mentioned that I was going to refactor Querki to use the Ecology Pattern. I confess, I was quite excited, and looking forward to it. I'm a fairly fundamentalist believer in refactoring as the way that you discover your code's correct design. (Having been through too many projects that spent months designing, and still never came terribly close to getting it right, I find that nowadays I mostly prefer to think about it until I begin to hit diminishing returns, and then just start coding -- in the long run, it's almost always more efficient.) I knew that I'd put it off too long, but I enjoy refactoring a lot.

Needless to say, it's been ten days of hell.

Yes, I knew that it had been too long to put off a serious refactor, but 100+ files and 17,000 SLOC is *much* too long. I really should have done this eight months ago.

That said, the process is drawing towards a close, and I'm happy about how it's come out. When I started out, an enormous amount of code was dumped into far too few huge files, mainly grouped by what they *were* instead of what they were *doing*. All of the interesting Types were in the file models.system.Types. More horribly, all of the Properties (many dozens of them) were in the file models.system.Properties. The various specialized kinds of QValues were defined in QValue.scala, because it had been a convenient place to dump them. models.system.SystemSpace depended on *everything* -- and everything depended on SystemSpace.

Initialization was horrifyingly ad-hoc -- the system was mostly composed of static objects, with deeply incestuous relationships. Properties required Types and Collections, which were themselves composed of Properties. Occasionally this would cause boot to crash due to recursive initializers, and I could do little but hack around the problems.

Now, that's been split into (quick check) 28 distinct subsystems, each implemented with its own Ecot and exposing no more interface than it needs to. The fraction of the files that cause the world to recompile has been reduced from about 90% to about 10%, and I'm continuing to improve that. Every file has been heavily rewritten, and most have been moved into more sensible locations; a rather scary fraction of the total lines of the system have been touched in one way or another. Each Ecot is now explicit about exactly which other interfaces it depends upon, and specifically which ones it needs in order to initialize. Lots of previously-public code, even some of the system Types, are now hidden away entirely inside of the Ecots that care about them.

Best of all, I just about have it to the point where there is *no* static state in the entire system. The Ecology is laced into damned near every object, and you can get to everything from the Ecology. This means that, when and if we have time, we should be able to run an arbitrary number of tests in parallel, each with its own distinctively stubbed version of the Ecology, without those tests stepping on or even knowing about each others' existence. Haven't quite gotten rid of the One Big Static Pointer to the Ecology yet, but I hope to be there by the end of today.

Indeed, the end result of all this is that Querki is impressively close to pure-functional, at least post-initialization -- I think there are on the order of a few dozen vars in the whole system, all of them tightly encapsulated. That probably doesn't *matter* much in the grand scheme of things ("almost pure functional" is one of those things like "almost pregnant"), but it makes it much easier for me to be proud of my work.

In retrospect, while my Engineer character could still stand to take another few levels of Refactoring, I can't really argue -- ten days to go from a monumental ball of spaghetti to a pretty well-decomposed Ecology, with clear internal interfaces and well-defined conventions for how to build each module, is pretty damned good. The code's ten times more maintainable, simply because it's now much clearer what belongs where. But still, I'll be glad to get done with this, and back to working on bugs and features, and feeling like I'm making progress...

[ETA: ahhh -- and now I'm getting to that refreshing downhill slide at the end of a big refactor, where I am now mostly *deleting* code. The feeling that each little tweak is making the system cleaner is delightful...]

... pretty much every big company downtown is closed due to the snow, but it's just another day at the office for you.

Things look quiet on the Querki front right now, mostly because I'm engaged in a gigantic refactor of the entire system. The brief summary is that I've come to realized that I have so many cyclic dependencies that every change is forcing me to recompile the entire world. Each recompile is *reasonably* fast, but now that it's over 100 files that is still a minute or so, dozens of times a day, and that's adding up. It was dumb of me to let it get to this point, and so I'm doing what I always do, and rebuilding the system using The Ecology Pattern, my personal favorite version of a dependency-injection framework, which I have used on essentially every major project for the past 15 years. Not sure offhand whether I've ever described that architecture here -- at some point, I'll write it up. (I highly recommend it -- I originally got the design from Tom Leonard at Looking Glass, who crafted it to cope with the peculiarities of large-scale programming in C++, but it turns out to be a good approach for almost any "conventional" large-scale program, regardless of platform or language.)

Hope everyone is managing to stay safe and warm...

... there is something that is just soul-deep *satisfying* about finally getting a proper test harness working.

Up until now, Querki has basically been skating along with no proper automated testing. (Aside from the Markdown tests, which I inherited from the library I started with and have been maintaining as I evolve the language.) I've known from the beginning that that was a bad idea, but the system was evolving so fast for the first 9 months or so that I would have spent way too much time rewriting the tests over and over again to match it. And, frankly, I didn't know the Scala/Akka test tech very well, and had too many other things to learn first. So I've simply been hand-testing stuff as I went.

Those excuses ran out a while ago, though, and Alpha has demonstrated just how fast Code Rot has set in --

mindways found a *host* of dumb bugs, many of which would have been caught by proper test coverage.

So I've spent the past day or two finally writing the beginnings of a real test suite, beginning with the QL Language itself and the user-visible methods. I started out with a couple of methods that I'd been looking at and had noticed code smells -- I wrote some proper unit tests, confirmed the expected bugs, fixed the bugs and validated the fixes properly. Then, for good measure, I added a new feature (making it easier to use pictures in Querki) in a proper test-driven way: write the test, demonstrate that it fails, add the feature, and confirm that it now works.

Man, that feels good. I hadn't quite realized how deeply I'd internalized that particular Agile tenet until now: working without automated tests has been making me twitchy.

It's going to take a fair while to pay off all the test debt, even for just the back end. (The UI is much harder to test, and it going to have to wait longer.) But as of now, I'm putting correct procedures into place: when possible, bugs and fixes will be validated by unit tests, and I'll generally be shooting to add tests whenever I change code. With any luck, the time spent on that will be more than made up for in system stability...

Endless waiting, endless delay. So many distractions, all of them important, but each one another little delay.

And then the excuses end.

Change the number. Check it in. git pull. Compile. Go.

Alpha.

It is hard to express how utterly *terrifying* it is to expose the system to the light of day, after more than a year of planning. I am so horribly conscious of the fact that this is barely 10% of the vision -- how limited the functionality still is, and how many bugs there are. But we're definitely up to the point where it is *useful*, if far from a work of art yet, so it's time to start getting other folks' input.

I'll be inviting a few folks to join in my own Spaces today, and beginning to slowly upgrade people to full-User status. (The difference is basically whether you can create your own Spaces.) It'll be gradual over the next month -- ideally a couple of people a week -- but if you'd like to kick the tires, please drop me a note (or comment here) and I'll add you to the list.

(And a voice in the back of my mind is cackling maniacally -- "IT'S ALIVE! MY CREATION IS ALIIIIVE!".)

I've spent the past couple of days doing some extremely complicated refactoring -- the details don't matter, just suffice it to say that it was motivated by the desire to make it easier for Querki Space Invitations to include links to specific pages in the Space. (Mostly because I've done this a lot in the Wedding Invitation Space, and the syntax kinda sucks now. It *should* Just Work.)

This morning, the Architect woke up, pulled the Product Manager aside, and said, "Wait -- why exactly are we bothering with this? I mean, yes, it's a nice feature and all, but don't you always tell me that features should fight for their lives? If we weren't already doing this in the Wedding RSVPs, would *anybody* have thought to ask for it? Can't we just put it aside for a while?"

So the two of them got into a deep argument for an hour or two (with the Project Manager weighing in on the Architect's side), and eventually the Product Manager admitted that, yes, this feature is a "nice to have", and not even actually that high on the nice to have list. So we should tie it off as neatly as we can, and move on -- for the time being, invitations can simply point the invitees to the root of the Space, and that'll do for now.

But of course, now the Engineer is going, "Wait -- what? I just spent two freaking days on this! The code is *good*! I'm ready to implement this feature now! Waaahhhhh!"

You'd think that having all of these people in the same head would make for fewer arguments, but that doesn't seem to be the case...

I'm finally getting to the point where I can spend some real time on the *fun* part of this project -- moving all my personal databases over to Querki.

To that end, I spent an hour yesterday setting up my Teaching Company Space, which replaces the wiki page I used to use, and has been updated with a pile of more-recent courses. As usual, the CD and Cassette courses are potentially loanable if you promise to be careful with them and get them back to me before horribly long.

It's a good illustration of a very ordinary Querki Space -- take something that has become an unwieldy text file, and turn it into a structured Space instead, so I can keep better track of it.

(And as always, I learn from each Space. This one really wants the "Review" Type that I've promised

mindways -- more incentive to get that done relatively soon...)

I just went to send a nicely-styled email through Querki, and was distressed to find that it *completely* failed. I had figured that email support wouldn't be complete CSS, but absolutely nothing stylistic happened.

A bit of research, and what do I find? It turns out that many mail readers (including, notably, Gmail) simply do not support modern CSS in the slightest. You can't do anything with real stylesheets -- you have to go back to 1990's styling, with explicit inline styles everywhere.

Of course, Querki was built to modern spec -- it *thinks* in terms of stylesheets, and exposes and uses styles that way. There is, quite deliberately, no support for old-fashioned inline styles, since they are deprecated by most good style guides these days.

*Sigh*. Pain in the tuchus. I've added style support in email to the to-do list, but folks should note that it'll be a fair while before it happens -- it's going to be a lot of work to make it work even minimally, and email is very much a "because we have to" feature. For now, Querki-sent email will necessarily be fairly plain...

Anybody have a better word for this concept?

My Mixology Space -- Falernum Flash

Querki.com ransomed from Squattage

While everyone else is frantically sewing for Pennsic...

Should Querki Spaces default to public or private?

And this is why PayPal makes me nervous

What should "skill levels" be called?

God, it feels good to be productive again

Identities vs. Users

How best to display a "choice"?

Anyone know a good "windowpane" control?

Oh, right -- Monads Don't Compose

My first experience with Monadism

Not quite Zen Master of Refactoring yet, but getting there...

You know you are working at a startup when...

You know you have good programming habits when...

Milestones

Querki makes my brain a very strange place sometimes

Another week, another Space

CSS in email: now that is annoying

Profile

July 2025

Syndicate

Most Popular Tags

Page Summary

Active Entries

Style Credit

Expand Cut Tags