jducoeur: (Default)

For the relatively serious programmers, I commend the article Asynchronous Programming and Scala. It's somewhat dense stuff, and as written is entirely in Scala, but the principles are pretty generic. It's all about how to think about asynchronous programming, and makes some important high points:

  • Asynchrony is not the same thing as Parallelism, although they are closely related.
  • Callbacks are a wretched way to deal with async, since they don't really compose. (I have learned this one through much pain.)
  • Futures and Promises are less wretched, but still problematic.
  • If you really want to do this stuff right, proper functional-programming techniques rock.

Of course, this is largely a rationale and advertisement for the Monix Library, which is a more or less state of the art library for "doing it right" -- but it's a pretty compelling rationale.

None of this is easy: he's summarizing stuff that's taken me four years to really internalize. (One of my medium-term but relatively challenging goals is to rewrite the pipeline for the QL language inside Querki from being Future-centric to Monix-centric: the result would be vastly more efficient and reliable.)

But it's important material, especially if you're designing systems. I encourage you to read and absorb it. Feel free to ask me "what the heck is that bit talking about?" questions, or even questions about the syntax and functions in the examples -- I always enjoy burbling about programming in general and Scala in particular...

jducoeur: (Default)
[Continuing the previous story, basically. Again, only interesting to the Scala-using crowd. None of this is rocket science, but since I had to piece it all together from scattered info I might as well post it. Note that this is stream-of-consciousness notes, as I develop this test harness.]

Next step: can we run our browser-based functional tests "headless"? This is awfully useful: Selenium is great, but it is *very* distracting to have a browser window suddenly pop up in front of you and start doing things by itself. (And that simply won't work in a truly headless environment, such as an automated test server.)

Fortunately, this also proves to be easy, if poorly-documented. This answer on SO works perfectly fine for ScalaTest + Play. The key is the useful little program xvfb -- the X Virtual Frame Buffer. Basically, it's a program that lets you set up a "virtual" window that doesn't actually exist, which you can then point your tests at.

So first, you need to install xvfb, which on Ubuntu is:
sudo apt-get install xvfb
Then you run it, like this:
Xvfb :1 -screen 5 1280x1024x8 &
(Note the capital "X" in Xvfb.) That is, run a virtual screen in the background (that "&" at the end), creating display number 1 and screen number 5, with the specified virtual resolution. That kicks the screen off as a background process (you can find it with "ps").

Finally, say this:
export DISPLAY=:1.5
This tells the processes in this shell that, if they're trying to show something in X, they should use display 1, screen 5 -- your new virtual screen.

Now you can run sbt and your ScalaTest + Play functional tests as documented. The browser will start and run as directed, but you won't see anything, since the "display" is all going to xvfb's bit-bucket.

A nice side-effect of this approach is that the xvfb stuff is completely external to your test. You can write this into your standard shell environment for normal use, but you can also turn it off when things are going wrong, or if you simply want to be able to observe the proceedings while you are coding up your tests.
jducoeur: (Default)
[Very Typesafe-stack-specific, but having spent the time figuring it out...]

Nothing to brighten my day than managing to solve what briefly looked like an intractable problem.

I'm trying to get full-stack functional tests running for Querki. (Not before time, I know.) To do this, I'm using ScalaTest + Play, which seems to be the approved solution. This is built on top of Selenium, the long-standing standard toolset for driving a browser for testing.

Problem was, when I wrote my first trivial test, ScalaTest refused to run it, claiming "Was unable to create a Selenium ChromeDriver on this platform". This was mysterious -- I have Chrome installed on this (Ubuntu) machine, and it works fine. A brief round with Typesafe suggested that the problem might be my X windows setup, which was a tad depressing, but I decided to get stubborn, rip the lid off the code and see what was actually going on.

It turns out, if you look at the relevant code in scalatestplus-play, that ScalaTest is intentionally flattening all exceptions from Selenium's ChromeDriver into a simple "that doesn't work here" cancellation. This is useful -- it's how you can write an all-browsers test suite in ScalaTest, and have it just quietly suppress the ones that don't make sense on this platform (eg, trying to run an Internet Explorer test on Ubuntu, or a Safari test on Windows). But it also loses what turn out to be some extremely helpful and informative exceptions that are raised by ChromeDriver.

So I added and ran this pseudo-test:
class DriverTests
  extends WordSpec
  with Matchers
  with ChromeFactory
  "I should be able to get a decent error" in {
    import org.openqa.selenium.chrome.ChromeDriver
    new ChromeDriver()
That gave me what I was looking for: the actual exception was
[info]   java.lang.IllegalStateException: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see http://code.google.com/p/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://chromedriver.storage.googleapis.com/index.html
[info]   at com.google.common.base.Preconditions.checkState(Preconditions.java:200)
[info]   at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:105)
[info]   at org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:75)
It turns out that, as it says, you need to manually download the relevant native drivers, install them, and point to them. Oddly, I haven't found this documented anywhere.

Anyway, fixing that produced *another* helpful exception, telling me that ChromeDriver required Chrome >= version 46, and mine was too old. I updated that, and *poof* -- it all works.

So: if you are having trouble getting ScalaTest's browser drivers working, I recommend the above little pseudo-test, so you actually get the underlying exceptions. And note that you need those drivers...
jducoeur: (Default)
For those who might be interested, I'm giving a talk next Wednesday at the Boston Scala Meetup, titled "End-to-End Scala: a Case Study in Integrating Akka, Play, Scala and Scala.js". It's going to be a fast dive into Querki's architecture and code, using that to illustrate how you can use the modern Scala ecosystem to build an entire huge application in a single language, with the front end, back end and APIs all written in pure Scala, within a minimum of boilerplate in the middle.

If you're at all interested in this stuff, I commend it to you. IMO, it illustrates one of Scala's "killer apps" -- the fact that I can add a new entry point to the system in *minutes*, fully reliable and type-checked, even with strongly-typed Exception propagation from the back all the way to the UI with essentially no hand-coded translation layers. It's amazing stuff, and lets me develop faster that I've been able to in any other environment, while still working more or less natively in the JVM on one end and JavaScript/DOM on the other.

And hey -- there's free pizza. What's not to love? Hope some of you can join us...
jducoeur: (Default)
[For the programmers, particularly for the architects.]

One of the trends that has happened so fast that I suspect most folks haven't even noticed it yet is the sea change that is occurring in Big Data processing right now. The short version is that, relatively recently, some folks from the Scala world pointed out that while Hadoop is a lot better than traditional RDBMS methods for dealing with data at scale, it still kind of sucks for many use cases. So a project got started to rethink the approach to Big Data around a streaming model. That became Apache Spark, and it is taking over the world with remarkable speed.

TypeSafe has posted a blog entry summarizing the benefits of Spark: it's fairly brief, and worth reading if you have any scaled-data requirements, to understand the strengths of the system. It includes a very concise tl;dr summary at the end. (Note, though, that it is written by Dean Wampler, who isn't exactly objective: his talk at NE Scala a few weeks ago kind of bragged about his self-described trolling of the Hadoop community getting the ball rolling in the first place.)

Querki isn't using this stuff *yet*, and probably won't for a year yet -- I have to focus on more critical-path issues for now. But I suspect I'll be adopting Spark before long, for things like automatic abuse catching. (I already know some of the obvious ways that wikispammers are going to try to game Querki, and a combination of event stream and graph analysis is probably going to be helpful to tame that.) And one of Querki's most game-changing features, App Communities, is going to be all about what happens when you combine Querki with Big Data. I suspect that almost any large-scale JVM-based system is likely to find this stuff useful in some fashion...
jducoeur: (Default)
[I really need a Scala icon for these posts.]

I'm going to start pushing a lot of my case-study posts, talking about Scala in the context of Querki, over to the Querki Development Journal. Today's post is one of those. It is *very* Scala.js-specific, but kind of insanely cool regardless. The tl;dr is that I spent today figuring out how, when doing a client/server RPC call, to throw an Exception in my server-side Scala code, and catch it in my client-side Scala code, with no boilerplate.

Being able to do end-to-end strongly-typed programming was cool enough to begin with; being able to throw strongly-typed Exceptions across that boundary is kind of fabulous. Every day that I work with Scala.js, I'm loving it more...
jducoeur: (Default)
One of the things I learned about myself many years ago is that, if and only if I believe in something to the tips of my toes, I am a really great salesman. That is, if I am truly, passionately enthusiastic about a subject, I am really good at communicating that enthusiasm.

So the primary lesson of the day is that, somewhere along the line, I became a Scala.js true believer.

Yesterday and today were the Northeast Scala Symposium. It was my first Scala conference, and a total blast: I learned a lot, and got to know more folks in the Scala community, including quite a number of folks who I've been reading and chatting with online.

Today was the "unconference": three tracks of smaller sessions, planned semi-spontaneously this morning. I wound up delivering one of the last sessions of the day -- I took the 75 minute presentation that I gave at the Boston Scala Meetup a few months ago, compressed it to an intense 30-minute speed through the slides, added a few nicely controversial assertions about where the ecosystem should evolve, and managed to spark half an hour of lively Q&A -- with not a little argument about my projections. I deliberately over-simplified the assertion (hence the argument), but stand by the heart of my point: that between Scala.js, Autowire and akka-http, I'm starting to want a web framework that is simply about client/server Scala programming, and mostly treats HTTP as a configuration detail.

Anyway, it was totally great: I'm still buzzed enough to be vibrating. (Admittedly, partly due to quite a bit of pre-presentation caffeine.) I definitely miss that; I suspect I need to seek out more opportunities to burble like that...
jducoeur: (Default)
I've burbled here about Scala.js, the increasingly-mature Scala-to-Javascript compiler that I am currently rewriting Querki in. (And I owe a writeup of my rant on the subject from a couple of weeks ago.)

One thing that Scala.js has lacked heretofore is a really solid *book*. There's a decent tutorial and a bunch of other useful information on the main website, and a growing body of community knowledge, but getting started looks a bit daunting, which is unfortunate: it's not actually a terribly hard technology to get started on, especially if you already know some Scala.

To help with that, Li Haoyi (one of the primary library-authors for Scala.js) today released a draft of Hands-on Scala.js, an online book aimed right at this problem. I'm just starting to read it myself, but at a quick skim it looks like just what the community needed. It starts off assuming only a fairly modest knowledge of Scala, Javascript and web development, and walks you through the rest: the rationale for using Scala.js, how to set up an environment, tutorials for various kinds of webby applications, and a final chapter going into depth on various major Scala.js topics.

As these sorts of books go it's pretty short (not having a publisher pushing up the page count helps), but it's still a book rather than an article: expect to take a while going through it. And please note that this a draft, albeit a late one: it's akin to a late-stage MEAP, so don't be surprised by an occasional typo.

Recommended reading to anyone who wants to keep their skills sharp and is interested in Web programming. IMO, Scala.js is one of the most important parts of the Scala ecosystem, and very much a tool whose time has come. Having a good guide for getting started in it should only make that better...
jducoeur: (device)
[For the programmers.]

Querki has come to a relative standstill, at least publicly -- it has been over a month since the last release, which is an eternity by my standards. The reason for that is that I'm doing the most dramatic rewrite of the project so far: I'm taking all of the scattered and inconsistent web pages and JavaScripts that comprise Querki's front end, and I'm rewriting them as a single, coherent AJAX web client -- written in Scala.

That's been a fascinating (if occasionally frustrating) project, *way* out on the cutting edge. As I often put it: I prefer to always work with leading edge technology, and Querki is mainly composed of bleeding-edge technology. The new Querki Client is written in *hemorrhaging*-edge tech: the compiler is at version 0.5.5, and most of the libraries are around version 0.3. We're only recently getting to the point where you can ask the question, "How mature is this stack?" with a straight face.

It's a little scary, but quite exciting, getting to use one language end-to-end for the whole system, with increasingly powerful libraries that let me, for example, simply call a method on the client and have it run on the server. I'm paying for it in a two-month delay now, but it should speed me up enormously going forward. (And it should help me with some key rewrites of UI components that have waiting for a year now.)

Anyway, having gone to the effort of learning all this stuff, it seems meet and just that I pass it on. So on November 11th 3rd, I'm going to be giving a talk to the Boston Scala Meetup: Using the Scala.js Ecosystem: a Case Study. Y'all are invited to come join us. This talk is likely to be interesting to anyone who is interested in the Scala language, but also to anyone who is interested in cutting-edge Web tech and wants to see a bit of the world beyond CoffeeScript.

The talk is currently slated for 6:30. Location is currently unknown, but likely to be Kendall Square-ish -- we've tended to meet in the Akamai building, but I don't think that negotiation has happened yet. Sign up for the Meetup, and it should update you as more information comes in. We usually go out for drinks and schmoozing afterwards, which is generally fascinating in and of itself.

Come on by, and please pass on word to anyone you know who might be interested...

ETA: Note that the date has been changed to November 3rd...
jducoeur: (device)
Not for the experienced programmers, for a change.

Several of my friends have, at one time or another, told me that they want to try out programming, and have asked where they should start. I've helped as best possible, but been slightly hindered by the fact that my language of choice is Scala, and the documentation on Scala, until relatively recently, sucked. Oh, it was fine for power programmers, and I even have a recommended book for Java programmers who want to learn Scala, but for learning how to program? Not so much.

However, I just came across Atomic Scala -- it's apparently been around for a year, but I hadn't noticed it before. A quick browse of the early chapters indicates that it is exactly that: a Scala book intended for the *complete* neophyte, teaching the elements of programming via Scala. Very step-by-step, very hand-holding (possibly to a fault), very focused on the relatively easy but important core of Scala -- the less-fancy bits where even I spend 90% of my time.

So if you're looking to learning programming from the very beginning, give it a look. I can't guarantee that it's the right thing (after 40 years of programming, it's a little hard for me to put myself in a beginner's shoes, and I'm too impatient to sincerely evaluate a ground-up book like this), but at first glance it looks like a pretty good option...

ETA: oh, and the reason this came to my attention is that they just released the Atomic Scala Activator Template, which provides you with runnable versions of all of the examples. Typesafe Activator is relatively recent, and is the newer and *far* easier way to play around with Scala.  I commend it to anyone who is dipping their toe in the water...
jducoeur: (device)
Strictly for the Scala programmers in the audience, but I've noticed a lot of folks using it to one degree or another recently: the Typelevel group is prepping to fork the Scala language. Read the article for more details, but none of it is especially surprising -- as Scala has become a "realer" language, being used in a lot of major enterprises, Typesafe (the official maintainers of the language) have become more conservative about changes, especially in the name of back-compatibility. The folks at Typelevel, who are by and large doing most of the most advanced and ambitious stuff in Scala, have become frustrated by this, and want the language to be able to evolve more quickly.

They seem to be doing it right: while they're planning on moving faster, they're also planning on making the changes in a compiler-mergeable way. So the overall effect is that the new effort is probably going to become a community-driven fork whose better ideas will gradually feed back into the official trunk of the language. Typesafe have officially declared their support for the effort, which should greatly reduce the potential for stupid politics. Overall, it's likely to be a win for everybody.

So if you're seriously interested in Scala, keep an eye on this project. I expect it to be provocative, sometimes controversial, but really interesting...

ETA: Just to head off other people having the same confusion I had, there is also a *second* fork of the Scala Compiler (although not so much the Scala language), from Paul Philips.  I get the impression that Paul has been working on this for a while, and he decided to go public in response to the Typelevel announcement.  There, there *is* a fair amount of stupid politics going on.  Nothing new and surprising, though: the friction between Martin (the primary author of the language) and Paul (who wrote much of the compiler) has been brewing for a long time now, and is pretty well-known: by now, they have deep divisions over how things should proceed.  Paul has long been on the record that (a) the compiler has become an unmaintainable mess and (b) since there is no fully-detailed language spec, that's unhealthy for the language.

Not quite sure how that's going to play out; my hope is that the Typelevel folks find a way to bring Paul into the fold, calm him down a bit, and give him a way to feed his work back into the "mainstream" without being directly under Martin's thumb.  He is unquestionably one of the most important long-term contributors to the language, and I have a nasty suspicion that his version of the compiler is going to be significantly more stable than the official one.  But he is explicitly *not* making a goal of compiler compatibility -- whereas Typelevel is trying to improve the language, he is mainly concerned with replacing the compiler with one that can be enhanced and maintained with less pain...
jducoeur: (querki)
This week's main project is to add scala.js to the Querki build. scala.js is -- well, let's put it this way. Querki is made up entirely of leading-edge technology. Some of the libraries qualify as bleeding-edge -- open-source bits that are very new and novel, but still decently mature.

scala.js is *hemorrhaging*-edge technology -- a project being run out of EPFL, the university where Scala was largely born in the first place. It's still officially pre-release, and has no commercial entity behind it. (Although a lot of us are starting to agitate for Typesafe to pull it in as an official piece of the Typesafe Stack.) But man, it is the coolest and most useful thing on Earth.

scala.js is a Scala-to-Javascript compiler. Everyone is beginning to acknowledge that Javascript is kind of a pain in the ass for serious work: while it is much more powerful than anyone understood at the beginning, it has all sorts of nasty limitations that you have to work around -- all sorts of table-stakes features like classes and modules and such have to be added as libraries, and all of it is dynamically typed, so you have to catch your errors at runtime. OTOH, those runtimes are now *blisteringly* fast, thanks to Google forcing everyone else to up their game, so you can basically treat it as machine code, and scads of languages are being produced that compile down to Javascript.

A couple of years ago, some folks decided to see whether it would be possible to compile Scala -- one of the most un-Javascript-like languages around -- to Javascript. It's taken a lot of releases and some very clever work, but the answer turns out to be yes. By now, scala.js works with the vast majority of the Scala language, as well as the most important libraries. And it's hit a point of maturity where a bunch of folks have been producing libraries that are specifically designed to be useful in the browser environment. (Big h/t here to Li Haoyi, who in his spare time has been doing some really amazing work.)

Anyway, I've now got that up and running, at least in test form. But today's "wow" moment was the tweak I just put in place, to get Source Maps working. This is a tech that Google has been pushing: basically, if you think of Javascript as machine code, this is the debugging symbols. The end result is that I can not only *write* my UI in Scala, I can (in Chrome, at least) *debug* it in Scala. I've now got things wired up so that I can, right in the browser, view the Scala source code for my client, set breakpoints in that code, inspect variables and parameters, the whole nine yards.

Of course, now I get to rewrite all of my existing Javascript code in Scala, which will be a mild PITA -- expect some bugs to sneak in while I iron that all out. But in the long run, this should make Querki's UI both more powerful and easier to develop. Javascript has been the biggest pain point in my programming life for many years now (I actually learned Flash in order to give myself an alternative to it), so the notion of being able to write Querki in end-to-end Scala fills me with absolute glee...
jducoeur: (Default)
[For the programmers]

Continuing yesterday's discussion of the shapeless library -- programmers who want to have their minds blown (or at least expanded) should take a look at the overview of shapeless 2.0. It goes *way* beyond the parts I mentioned yesterday, with concepts including:
  • Truly polymorphic functions: a way to build functions that operate cleanly on multiple arbitrary Types. (Which is normally impossible in Scala.)

  • The ability to use all of the usual List-like operations (head, tail, concatenation, etc) on tuples.

  • HMap: a variant of Map that allows you to have keys of multiple types.

  • Singleton Types: that is, "the Type that is the Integer 23", "the Type that is the string 'foo'", and so on.

  • Extensible Records: sort of a hybrid between a compile-time class and a runtime Map -- heterogeneous and extensible, yet strongly-typed.

  • Discriminated Unions (yay! One of the most-asked-for features in Scala).

  • Lenses: strongly-typed "views" of a subset of the fields of a Type (including arbitrary fields of nested Types inside classes).

  • Literal collections whose sizes are encoded in the type system, so that you can check that they match at compile time.
To me, the thing about all of this isn't just the library itself: it's that it is doing so many things that I wouldn't have even considered *possible* before. These functions do all sorts of things we do all the time -- but usually at the expense of breaking type-safety. Basically, shapeless is a whole different way of looking at the world, showing that in many cases you *can* have your cake and eat it too: if you're sufficiently clever, Scala allows you to do remarkable things with completely strong typing, and shapeless makes the process of doing so reasonable for real code.

(Mind, I gather all of this can be done in Haskell as well -- it's just remarkable to see it in a language I actually use. It's a fine demonstration that, when used really well, Scala can be far more strongly-typed than something like Java or C#, for equivalent problems.)

Of course, the downside is that I'm starting to realize lots of places where I should be using this stuff in Querki. Adopting shapeless properly is going to be a long-term project...
jducoeur: (Default)
I'm starting to dig more seriously into Scala's Shapeless library, and it occurs to me that some of my more intrepid readers might find this interesting.

One of the most powerful and scary websites you will ever find in programming is typelevel.org. This is a collection of the most fascinating and brain-breaking libraries you will find out there today. AFAIK, they all derive at some level from the old observation that Scala's type system is sufficiently complex, it is actually Turing-complex unto itself. And that means you can do remarkable, powerful, slightly insane things with it.

The best-known of the bunch is scalaz (pronounced "ScalaZed", I gather). While it is oversimplifying to say that scalaz is Haskell implemented in Scala, I believe that's a decent first-order approximation.

There is Shapeless, linked above, which implements "polytypic" generic programming. The part that everyone gets swept up by is the HList data structure -- basically a meta-type which is a "list" of arbitrary other types, combining the best aspects of lists and tuples in a type-safe way -- but I gather that that is basically an implementation detail. The real goal is being able to abstract over algorithms at a higher-kinded level: for example, abstracting out the concept of "tree traversal" without needing to depend upon any specific implementation of "tree". (I think: Shapeless was the topic of the latest episode of The Scalawags, which inspired today's surfing.)

Those are the best-known parts of typelevel, but I find that there is a lot more there now. There is ScalaCheck, which lets you describe the "properties" of your methods, and then generates randomized data to hammer at those methods and make sure your invariants are actually invariant. There's scodec, a functional library for working with binary protocols. There is Spire, a numeric library that is designed to let you write efficient numeric code that works for any implementation of "number".

Etc -- the typelevel page itself is a bit thin, but provides pointers for what to look for online. I expect to spend a number of months wrapping my head around the more-interesting bits and pieces here...
jducoeur: (Default)
[A short burble for the programmers.]

One of the odder concepts in Scala, which I'm beginning to truly appreciate, is the "self-type". Here's an example...

Querki is full of what I originally was calling "methods". ("Functions" is more technically correct at the moment, and I'm moving in that direction, but roll with "methods" for now.) There are dozens of these, scattered all over the code. They follow a few distinct forms, so I gradually invented a bunch of convenience base classes named "SingleContextMethod", "ThingAndPropMethod", and so on, which the various Methods inherit from.

So far, this is all well and good. But in the new Ecology-based world, it turns out that it would be useful for these definitions to know about their context -- basically, for the methods to know that they are defined inside an Ecot, so that they can access the Ecology through it. Moreover, it really should *not* be legal to define a method anywhere outside an Ecot, since the Ecots are what build the world.

After a little thought, it turns out to be trivial to do this, and it illustrates an aspect of how Scala's version of multiple inheritance works. I took all of these base classes, and bundled them up into a trait (which is kind of like the midpoint between a class and a Java interface), defined roughly like this:
trait MethodDefs { self:Ecot =>
  val Conventions = initRequires[querki.conventions.Conventions]
  class SingleThingMethod(tid:OID, name:String, summary:String, details:String, action:(Thing, QLContext) => QValue) extends InternalMethod(
  ... etc
What's going on here? Basically, that "self" declaration at the top says, "I am a trait that can *only* be mixed into an object that also implements Ecot". Of course, having declared that, it means that I can now use all of the methods of Ecot without any additional declaration -- this trait knows that it *is* an Ecot, and can behave as such. So it can directly use initRequires(), which is defined in Ecot.

That being done, a typical use case is as simple as:
class DataModelAccessEcot(e:Ecology) extends Ecot(e) with DataModelAccess with MethodDefs { ...
I just mix in MethodDefs, and *poof* -- I get the functionality. And MethodDefs can do anything that the contents of an Ecot can do. Moreover, mixing in MethodDefs implicitly declares that this Ecot is dependent on Conventions for its initialization. (Since creating a method requires the existence of the "Summary" and "Details" properties, which are defined in Conventions.)

There are many reasons why I am so fond of Scala. One of them is the fact that they finally got multiple inheritance just right -- strongly-typed, but massively powerful and easy to use. I'm only now starting to understand how important self-types are to that, and how they allow you to decompose code much more precisely and correctly than you can in Java or C#...
jducoeur: (Default)
Time for a small Scala Burble of the Day.

NSTIW, in a discussion on the akka-user mailing list (I finally committed to getting involved in the Akka community, and unsurprisingly have immediately started making trouble). Somebody suggested a tweak to my code which appeared to involve a trait (Scala's interface-plus-implementation mixins that make the language so powerful) using super to call a method on the *child* class. I pointed out that that was wrong -- but it turns out not to have been as wrong as I thought.

Enter the Stackable Trait Pattern, which is surely one of Scala's cleverer ideas -- a typesafe, statically-compiled way for mixins to work together similarly to how they do in dynamic languages. Basically, you build your concrete object out of a base class and any number of traits. Those traits can work *together* in a highly pluggable way. In this model, "super" doesn't just mean "the superclass" as it does in most static languages; instead, when used in this way, it means "the previous version of this method in the mixin list". Hence the "stackable" -- each trait acts as a *modifier* on the method, in an order decided when you actually build the final object.

It's a subtlety, but a really useful one -- it means that you can basically create a library of legos, which can be assembled in different configurations simply by constructing your concrete object with your traits in the desired order of "inheritance". Neat...
jducoeur: (querki)
Okay, time for a little burble. [This one is solely for the truly geeky programmers; the context is, of course, Querki. I don't have anyone to do code reviews with, so I'm afraid you get the burbles.]

So two days ago, I implemented _modTime, which produces the time that the received Thing was last changed. Yesterday, I realized that this was really only useful if you can sort on it, so I enhanced _sort to take a parameter, like this:
[[Page._instances -> _sort(_modTime)]]
But of course, that produces the page in time order, which is almost never what you want -- 9 times out of 10, you want to print things in *reverse* time order, from newest to oldest.

So I clearly wanted some sort of "_desc" modifier on _sort, so that you can tell it to sort in descending order. My first instinct was to just add a _reverse function, an easy and just plain useful tool to just reverse the list order, but that's not quite right: I want the results to be in descending order by time, but ascending order by name if the times match. I specifically want to reverse _modTime, *not* necessarily the entire results.

Thinking about it a little, I decided that the obvious syntax had to be:
[[Page._instances -> _sort(_desc(_modTime))]]
That is, "sort by _modTime, descending".

But wait a second -- what does that *mean*? I mean, _modTime is being applied to each element in the list; saying that an *element* is "descending" seemed almost meaningless, until I thought about it a bit more, and realized that what I'm doing is transforming the *type* of the result -- "_desc(_modTime)" clearly means "return each item's modTime, with the sort order of the returned type reversed".

That's kind of insane. And I was completely floored to discover that this is, pretty much, all the code it required (going into Scala now, and omitting documentation):
class DescendingType[VT](baseType: PType[VT]) extends DelegatingType[VT](baseType) {
  override def doComp(context:ContextBase)(left:VT, right:VT):Boolean = !realType.doComp(context)(left, right)

object DescMethod extends InternalMethod(DescMethodOID, toProps(setName("_desc")))
  override def qlApply(context:ContextBase, paramsOpt:Option[Seq[QLPhrase]] = None):QValue = {
    paramsOpt match {
      case Some(params) => {
        val innerRes = context.parser.get.processPhrase(params(0).ops, context).value;
        innerRes.cType.makePropValue(innerRes.cv, new DescendingType(innerRes.pType))
      case None => WarningValue("_desc is meaningless without a parameter")
That is, when you call "_desc", it processes its own parameter (params(0).ops), passing in the received context; rewrites the result using the same Collection (innerRes.cType), and wrapping the Type (innerRes.pType) in a pseudo-Type (DescendingType) that reverses the result of comparisons (doComp()); and passes that down the pipeline.

It simply works, and took less than half an hour to come up with. Wow. Yes, I understand that the above looks kinda cryptic, but seriously: I can't think of any other language I've ever played with that could do this, preserve type safety, and not have me tied in knots for a day or two.

(And the implications here are staggering. I'd added DelegatingType for a fairly minor boot-time requirement last month, and hadn't given it much thought. But if this works, it means that Querki can do almost arbitrary type transformations at runtime internally. Which means that sooner or later, we're going to be getting the same sorts of high-level Type operations in QL itself. Neat...)
jducoeur: (Default)
I decided yesterday that the OP Compiler's configuration file was becoming an unspeakable pain in the ass. It's a huge XML file that represents essentially all the semantics of the SCA: what the branches are, the Kingdom/Principality/Barony hierarchy, what awards each one contains, and so on. I have little taste for writing a DTD, so it's just a big block of XML, and when I make a mistake it can be a bear to track it down.

So I decided to switch to instead hardcoding the configuration into the code itself. For a one-off program, this seems entirely reasonable; moreoever, it's increasingly idiomatic Scala. (Scala's native build system essentially uses a little DSL program to define your build, so you have access to full Scala. Scala tries to be script-friendly, and that sometimes means that code is better than config.)

I've spent the morning fiddling with that, and gradually improving it, and the results are just lovely. For example, here is the listing for the Kingdom of Artemisia (the bits that are relevant to our OP so far, anyway):
          awards = Seq(
            "Golden Feather of Artemisia",
            "Grace of Artemisia",
            "Griffin's Heart",
            "Griffin's Talon",
            "Lady of the Papillon",
            "Maple Leaf of Artemisia",
            "Pillar of Artemisia"
          children = Seq(
            Barony("Arn Hold", awards = Seq(
              "Moose Combatant of Arn Hold"
            Barony("Loch Salaan", awards = Seq(
              "Crystal of the Salt Wastes",
              Award("Devoted and Unique Company of Keepers", "Devoted & Unique Company of Keepers"),
              "Falcon of Loch Salaan"
That's compiled code, that builds the data structures representing the Kingdom. While it isn't *quite* the absolute minimum number of keystrokes to express the concepts, it's pretty damned close. And it's about as readable as I could hope for.

What makes it really spiffy-keen, though, is that it's much more strongly-typed than it looks. (Scala is strongly typed to its core, with type inference that most languages can only envy.) So those lists of strings above, that give the names of the awards? Those aren't actually strings as far as the calling code is concerned. Instead, the compiler knows that it expects a Seq[AwardInfo] for the awards parameter. And I have the following implicit converter in scope:
implicit def string2AwardInfo(name:String) = AwardInfo(AwardName(name, Gender.Unknown))
That is to say, "When you are looking for an AwardInfo object, and what you find is a String, this is how to convert it into an AwardInfo". So each of those lines actually turns into an AwardInfo to get passed around. (And yes, the Eclipse plugin is smart enough to highlight these and provide tooltips about the conversion.) This way, I can just write a string for the "simple" cases where I just have a name, but give an explicit Award() listing (which is itself a special sort of function call, with lots of optional parameters) when I have more information to convey such as alternate spellings. (As in the DUCKs above.) That level of strong typing means that I get code as concise as most dynamic languages, but errors still usually get caught in the editor within a second of my typing them.

I don't think I can explain to a non-programmer the sheer delight of code like this, and the way it hits me on an aesthetic level. Think of it like sculpture, though. Most languages hand me basically a big hammer, and the result is that my programs come out rough and chunky -- you can sort of see what it's supposed to look like, but have to squint a bit. Scala, by contrast, gives me a whole assortment of chisels to work with, so that I can produce programs like this: finely detailed, with nothing extra to get in the way of its proper Platonic form. I spent the morning sculpting, and it is just *delightful* to get something pretty as a result...
jducoeur: (Default)
Okay, time for me to learn GitHub for Querki. And how better to do that than to use it for the OP Compiler project first?

Several people expressed interest in seeing how the OP Compiler works. I've now published it online -- you can find it at https://github.com/jducoeur/OPCompiler. The README covers the basics -- I encourage programmers to poke around at it, especially if you want some examples of what Scala looks like in the hands of somebody who's still learning the ins and outs.

You can also take a look at the log.txt file, which is the output of the Compiler. That has several major sections:
  • The beginning is the Court Reports section, showing what I've parsed so far from those. (And you can see that there's still a lot of cleanup needed.)

  • Then comes the Alphabetical parse section, which is largely uninteresting except that it shows where the errors are.

  • About halfway down is the much more interesting Alpha List, which shows the consolidated view of all the people, and what I currently think they have (including which data source each award comes from). IMPORTANT: I've only done the past ten years of court reports, and the "A" alpha listing so far. Don't panic if you don't find yourself there.

  • Finally, down at the bottom are all of the errors that we're currently encountering, most of which are places where the source data is so messed-up that we're going to have to fix the original.
So please go poke around if you're curious. I've put enough work into this program that I'm happy to talk about it and answer questions. (Both about the Compiler itself and the Scala code -- there's a lot of interesting magic there, and I'd love to chat about it.) Keep in mind that the project is as-yet-unfinished -- I'll be needing a bunch of help for the final cleanup phases, hopefully fairly soon...
jducoeur: (Default)
[For hardcore programmers only. Seriously: this is the stuff that pushes right at the bounds of my comprehension.]

This has been a good week for good explanations. It started on one of the Scala mailing lists, where one of the users was bemoaning the difficulty of trying to figure out what the heck continuations are and how to use them. For once, he got good pointers.

First up is the video Monadologie -- a concise but mind-blowingly useful little one-hour lecture. In about half an hour each, it describes what continuations are and why you might want to use them, and then what monads are and why you definitely want to use them. They're still advanced concepts, mind, and if you're coming into this cold you shouldn't expect to instantly internalize them. But it provides great handles on why you want to use continuations (to build complete control structures that are easy to use) and monads (to more easily compose dataflow), and gives a sense of how each tool works. Even if you aren't using Scala yet, the video is worth watching, since the concepts in it are becoming gradually more common nowadays.

Second, the conversation finally convinced me to buy Scala for the Impatient. Seriously: every hardcore programmer should read this book. It is exactly the book I've been looking for. It assumes that you already know at least one major OO language like Java or C#, and that you don't need explanations of things like functions, classes and stuff like that. Instead, it leaps right to the meat of the question, "What's different about Scala?". It's structured around the Scala Levels (a useful categorization of experience levels that Martin Odersky came up with a year or two ago), starting with the basics that everyone should know and gradually moving up to the incredibly sophisticated but powerful features. (With delimited continuations coming at the very end.)

The book is available in several forms, including Kindle (which is how I bought it), printed or other ebook. The early chapters have been made available for free from Typesafe (the consulting company that consists of many of the movers and shakers in Scala), and are well worth reading. I've been saying for several years now that Scala is the current best language on the block -- this is the book that demonstrates why, with sections relevant to the daily work of any programmer...


jducoeur: (Default)

June 2017

456 7 8 910
18 192021222324


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags