You know you are working at a startup when...

[jducoeur]

... pretty much every big company downtown is closed due to the snow, but it's just another day at the office for you.

Things look quiet on the Querki front right now, mostly because I'm engaged in a gigantic refactor of the entire system. The brief summary is that I've come to realized that I have so many cyclic dependencies that every change is forcing me to recompile the entire world. Each recompile is *reasonably* fast, but now that it's over 100 files that is still a minute or so, dozens of times a day, and that's adding up. It was dumb of me to let it get to this point, and so I'm doing what I always do, and rebuilding the system using The Ecology Pattern, my personal favorite version of a dependency-injection framework, which I have used on essentially every major project for the past 15 years. Not sure offhand whether I've ever described that architecture here -- at some point, I'll write it up. (I highly recommend it -- I originally got the design from Tom Leonard at Looking Glass, who crafted it to cope with the peculiarities of large-scale programming in C++, but it turns out to be a good approach for almost any "conventional" large-scale program, regardless of platform or language.)

Hope everyone is managing to stay safe and warm...

Comments

[goldsquare.livejournal.com]

That would be interesting to read.

And don't be so sure about the start-up. I put in a very full day yesterday as well. :-)

Although in some ways my employer is a bit like many-start-ups. We do lots of projects, usually in teams of 3-4 for a period of a year or less. The speed and pressure are very aggressive, even though the financial stability is good. And we're around 200 people now.

For example my current project has 2 engineers and one quality engineering person (me), as well as a half-time project manager and some time and attention from one of our customer-management folks. (And, because this is a medically regulated project, we get an occasional look-at from our quality assurance/regulatory person.) Basically 4 FTE overall or slightly less, with 3 of us actually doing the work.

We are WORKING for a start-up, which means that we work at the pace of a start-up. Actually, we tend to work a lot faster than the client, which is why they are willing to pay us money. :-)

[jducoeur]

That would be interesting to read.

Okay, will do. There are no particularly Deep Insights about it, but the Ecology Pattern is just a nicely reliable program architecture -- basically a way to break down your components so that you get strong decoupling, rigorous separation of interfaces from implementation (used properly, it's highly mockable for testing), and (the unusual part) an official pattern for initialization order, which in most systems is an ad hoc mess. I'm always surprised at how few systems take initialization seriously: in a complex and interdependent system, getting initialization right is *not* easy.

Like I said, I pretty much always use a version of it -- I find that most of the DI frameworks are lacking in one respect or another (most often initialization), and I always get best results when I implement Ecology at the heart of my program, and tune it to that program's requirements. (For instance, sometimes I need to deal with asynchronous and/or multi-threaded initialization, sometimes not.) I'm kicking myself for not just starting out with it: my current project is a remarkably delicate refactoring dance.

And don't be so sure about the start-up. I put in a very full day yesterday as well. :-)

Actually, I meant today instead of yesterday. Downtown is basically shut down -- Liberty Mutual called Kate yesterday to say that the entire (mammoth) office will be closed today, and I gather that most of the big companies here in town have been encouraged by the city to do the same.

But yeah -- "startup" is highly subjective, and certainly I know from "company that feels like a startup". Looking Glass was in most ways like that: not really a startup in any meaningful sense by the time I got there, but had that pace...

[goldsquare.livejournal.com]

Given that I was deeply immersed in distributed and remote computing, I always tend to separate the interface from the implementation. (You can't work for the authors of Java RMI and not get that beaten into your brain.)

I agree with the initialization bit - but it always astounds me that many systems fail to handle graceful shutdown either. Dependency loops, or hidden dependencies seem to plague these situations for a lot of software.

Modern distributed software that has built-in robustness makes me think I am re-reading "The Adolescence of P1" again. :-)

[jducoeur]

Given that I was deeply immersed in distributed and remote computing, I always tend to separate the interface from the implementation. (You can't work for the authors of Java RMI and not get that beaten into your brain.)

That's reasonable, although my position there has become a bit more nuanced in recent years. In particular, Scala's approach is less hardline than Java's -- instead of "interfaces", you have "traits", which are explicitly allowed to contain some functionality, and usually do. Objects are often composed from multiple traits, in a "mixin" style. (Not quite multiple inheritance, but close. Yes, there are well-defined rules about things like name collisions, and how traits within a single object work together. It works nicely.)

What the past year has gradually reminded me of, though, is that while this approach is very, very nice, you still need to keep interface and implementation *mostly* separate, simply because if you don't, you wind up with Dependency Hell. (Which is where I am now.) In particular, while small-scale use of complex traits is fine, your major system components need to be more rigorous about the interface/implementation divide.

Curiously, none of this has anything to do with remoting for me -- indeed, the plan for Querki is that the Ecology is going to be completely identical on each node. Remoting is being dealt with via Akka, as is threading: those problems are Akka's sweet spot, and it's an excellent architecture for them. Very different focus, though: things like Java RMI tend to be more about distributed services, whereas Akka is more focused on massive scaling -- designing things in such a way that your system is essentially identical whether it is on one node or a thousand. (In other words, Akka is more about parallelism, and blurring the local vs. remote question. Your system is made of thousands-to-millions of independent Actors; where they reside is a matter of configuration, not code.)

For this project, the Ecology is strictly for the *functional* (specifically, stateless) parts of the code -- which, I am beginning to realize, will eventually be 75%+ of the system. I hadn't actually expected that, which is how I blindly wandered into my current mess. I have all of the stateful and distributed parts nicely encapsulated into Akka Actors, and had relegated "everything else" to various utility libraries. But I've realized that those "libraries" are, in fact, most of Querki, and they are wildly interdependent, so it's currently a wreck.

I agree with the initialization bit - but it always astounds me that many systems fail to handle graceful shutdown either.

Yaas. Worse, many systems explicitly pooh-pooh the notion that orderly shutdown even matters. Ecology is pretty explicit there: each component has both init() and term(), and while most don't avail themselves of term(), they are encouraged to do so if relevant.

(I *do* usually make the simplifying assumption that termination order is the reverse of initialization order -- we unwind in reverse-dependency order. That's never actually bitten me on the ass to date, but I do recognize that it's not necessarily a safe assumption.)

Dependency loops, or hidden dependencies seem to plague these situations for a lot of software.

Yep, and people ignore this stuff until one of their initializers crashes because of a hidden dependency loop. Truth to tell, while my current project was started mainly to break the *static* dependency hell (that was slowing down compilation), I'm also breathing a sigh of relief that it's going to force me to get more disciplined about *runtime* initialization dependencies, which have been much too ad hoc heretofore.

Modern distributed software that has built-in robustness makes me think I am re-reading "The Adolescence of P1" again.

Haven't read it, but yeah, it's quite neat working with modern architectures. I'm actually very active in the Akka community these days, and it is *really* neat participating in that, helping to craft a common understanding of how to build things in a reliable and massively-scalable way...

[goldsquare.livejournal.com]

Having spent an hour plus clearing snow and cars and walkways, I must get back to work.

BUT: I don't know Akka (I will, someday). But from your capsule description, RMI is not all that different in concept. You specify that an interface can run anywhere, and handle the additional exceptions that network-based failures can provide - all in addition to whatever your interface provides.

There is no reason that RMI-based interfaces have to run in any particular place. It's just that RMI is a bit of a P.I.T.A., so most people don't invoke the overhead unless they are certain they need it.

So, as practiced, your understanding is correct. But it's not intrinsic to RMI, it is intrinsic to its difficulty.

[curiositykt.livejournal.com]

It just looks like those companies are shut down due to snow, in reality we are all trapped in our home offices working 15 hour days over slow VPN connections... I feel like I am on house arrest.