Thoughts about Nusenet

[jducoeur]

For a while now, I've been pondering the idea of, "What would / should Usenet look like if we were to rebuild it today?" As Reddit tries to go full Twitter, that topic is getting a little more timely.

So let's take the question seriously, and kick it off with some initial requirements analysis.

(I'm going to post this on both Mastodon and Dreamwidth; comments solicited on both.)

---

Personal context: Usenet was basically my introduction to the Internet, back in '87: I was one of the founding members of the Rialto, the SCA newsgroup (rec.org.sca), and pretty much lived on Usenet for about five years.

I've been contemplating this "what would a new Usenet be?" for a fairly long time. (I actually own nusenet.org, specifically to provide a home domain if this ever goes anywhere.)

---

For those going, "WTF is Usenet?", it was the original distributed forum system. Conversations on hundreds of topics, copied from server to server around the world. The tech was primitive by today's standards, but it was fairly cutting-edge then.

So let's think about requirements from a Usenet lens. What did it do well? (+) What were its problems? (-) What were we not even thinking about then?

---

+ Usenet was topic oriented, not person oriented. That's an important niche, and surprisingly poorly served nowadays.

+ "Topics" could include communities. Some of my favorite newsgroups were for particular niche communities (like the SCA).

+ The topic namespace was hierarchical; you could easily split rec.humor.funny out of rec.humor.

---

- The Usenet namespace (the list of groups) was controlled by centralized mechanisms that scaled fairly poorly. This worked for hundreds of topics; it wouldn't work for tens of thousands.

(The community quickly devised a workaround, in the form of unofficial "alt" newsgroups for topics that were too new or controversial. These weren't necessarily distributed as widely, but it generally worked.)

IMO, folks should be able to devise whatever groups they want: it shouldn't be centralized.

---

+ Other than the namespace, the system was highly distributed. Not only wasn't it centrally controlled, it was architecturally almost impossible to control.

(This didn't seem radical at the time, since the other major system was email. Now, it seems kind of radical.)

---

+ Conversations were explicitly threaded, and threads could branch as needed. No, this isn't obvious, and there are both pros and cons to it.

+ It was defined by the protocol, not by the specific client: more like email, less like Facebook. (Again, this isn't obvious, especially nowadays.)

---

+ You could block individual posters. For the time, that was a bit radical.

- I suspect the moderation tools weren't nearly good enough for modern requirements, although they were evolving pretty rapidly.

? I'm not entirely sure what moderation means for this sort of medium. Getting this right is important, and not simple. (This is a big topic.)

---

? While you could avoid reading the messages from a toxic poster, there was no way to prevent a toxic poster from seeing you.

(This was a concept that just plain didn't exist, and still doesn't exist in many systems. But a lot of folks in the Fediverse care about it, so it's worth mentioning and thinking about.)

---

- Spam was (and is) a problem. Usenet was where we really learned how much of a problem spam could be.

(Yes, this ties into the moderation problem, but is a different problem than bad behavior or toxicity, and probably needs to be looked at separately.)

---

Okay, that's an initial list, to start the conversation. What have I missed? Do I have some of the plusses and minuses wrong?

For now, let's focus on requirements rather than architecture -- "what do we want?" rather than "how should it be built?" (Or "does this already exist?") Those can come later.

Thoughts?

Comments

[alexxkay]

"IMO, folks should be able to devise whatever groups they want: it shouldn't be centralized."

I agree with this, but with a significant caveat. There will be groups like alt.WeLoveNazis, and the system will need to think about how to react to that. This is something that the fediverse is currently struggling with, I think.

To generalize, usenet had very poor tools for defense against bad actors (because bad actors were legitimately rare at the time). Nusenet would need to be robust against a wide variety of threat models.

[alexxkay]

A principle from ye olde quote file" "Usenet interprets censorship as damage, and routes around it." Which was a good first principle, but again, there needs to be serious thought about how to handle illegal forms of speech.

Following from that, I would like to propose as a principle that anonymity be explicitly supported, in such a way that the system *cannot* divulge much information to law enforcement and/or totalitarian governments.

[alexxkay]

Can we end the September That Never Ended? Is it possible to have a system that is better about onboarding shared culture than usenet was? I think the answer is pretty clearly yes, though the devil's in the details.

[alexxkay]

Some more modern features that I would want easy affordances for in any nusenet:
* Hiding spoilers
* Tags
* Cuts
* Something along the lines of markdown or other *simple* formatting.
** But dear god, no blink or other animated text.

[dsrtao]

Hypothetically, a group of sysadmins could have decided that personal trust among server administrators was the necessary ingredient missing from Usenet, and created a separate hierarchy of groups that they only propagated among themselves. The first tenet of fight club being that you don't send fight club messages on to any server you don't recognize as being in fight club. The qualification for knowing about it and joining it were basically the same: someone already in was willing to take responsibility for you. The hypothetical benefit was a high signal/noise ratio, and the major drawback being a very, very limited population.

Over in the non-hypothetical world of lobste.rs, someone had the same idea about membership in a forum very much like hackernews -- the social graph being explicit, and public, and used by the moderators in certain cases to not only prune aberrant leaves but also ask probing questions of their invitor. (Also, moderator decisions/actions all go into a public log.)

[dsrtao]

lobste.rs is currently at about 16,000 people in the graph, which is both laughably small as sites go but comfortably large for a tightly-moderated forum/newsgroup.

moderation

[dsrtao]

Self-moderation, the killfile, is a feature that every social system needs but is usually worse-implemented than usenet -- which is weird, because usenet does it client-side.

Two forms of moderation exist on usenet. The well-known one is the 'moderated group', where every post passes through to a human gateway before being allowed to post. It's a lot of work, and restricts traffic significantly. It works very well at a high cost and low throughput.

The other one is the anti-spam mechanism, which uses public key infrastructure to sign forged cancel messages which chase the spam around the distribution network. It generally did not work well.

[dsrtao]

Lessons from usenet: a social group can thrive on reputable pseudonyms. It can't have discussions at all if it responds to anonymous people.

If you can use TOR (or some similar system) to send messages which are authenticated to a consistent pseudonym, that's good. If you can send unauthenticated messages, that's bad.

Maintaining N pseudos by default ought to be easy and client-side: "Select the account to post this from:" [professional] [community] [friends] [family] [...]

[hudebnik]

Good point. Pseudonyms aren't a problem -- people can develop individual reputations and trust graphs -- but easy forgery is.

[hudebnik]

As I recall, individual host admins could decide which groups in the hierarchy to subscribe to. I suspect that most admins subscribed to all of the "standard" hierarchies, and either picked and chose among the "alt" hierarchy or skipped it completely. Don't know how important a feature that is; I guess Mastodon does something vaguely similar.

[jducoeur]

I agree with this, but with a significant caveat. There will be groups like alt.WeLoveNazis, and the system will need to think about how to react to that. This is something that the fediverse is currently struggling with, I think.

Not sure I agree with "struggling" per se. It's ongoing effort, and never perfect, but the Fediverse's architectural answer to that seems to mostly work. It accepts that toxic speech will happen -- there's no way to prevent that, in a distributed environment -- so it instead focuses on what amounts to a treaty-based approach: we'll talk to you so long as you enforce behavior rules that we agree with.

There are certainly gaps, and I'm sure there is room for improvement (especially in figuring out how to moderate at scale), but so far I'm cautiously optimistic about the approach.

[jducoeur]

Yeah, that's a good point. Most systems of communication have issues with this. (Heaven knows this is an area where Mastodon has been fairly weak.)

[jducoeur]

AFAIK, Mastodon does it more on an opt-out basis: servers declare which other servers they are going to refuse traffic from. The admin community has been gradually building some consensus lists of bad actors, and IIRC Gargron has finally knuckled under and accepted that the reference implementation should provide each mechanisms to import such lists in an automated way.

I suspect that Nusenet would want to be opt-in as a general rule -- the scale of the modern Internet is big enough that I wouldn't want to assume that every server accepts every group. But it might work similarly to how Mastodon deals with A/V traffic: stuff only hits your server dynamically, when requested. If servers were smallish (as most Mastodon servers are), then opting in to the groups that are requested by your users and not banned could make sense.

[keshwyn]

Legal: What do you do when illegal (and I'm thinking CSAM) content gets posted to a group? How does it get deleted? How do you handle the fact that since it's a distributed thing, that crap is now on *everyone's* machine?

[alexxkay]

Perhaps "struggling" was the wrong word. "Dealing with" might be closer to what we apparently both mean :)

[fitzw]

IIRC, some of the decisions for how the Newsgroups worked were driven by the nature of UUCP, and kept when NNTP started to be used. (Yes, I know that we're not talking about protocols for a new system, but the protocols for the previous system can’t really be ignored for how things worked under Usenet.)

I believe that the word “troll” was coined to respond to certain types of bad actors in Usenet.

Moderation tools (for moderated groups) can be important. Client-based killfiles are still a good idea, but group level killfiles for moderator use (or the equivalent) would be a good idea. (Welcome back, “plonk”.)

As already noted, a method to keep certain actors from seeing you would be useful.

Server-local groups (some groups are local). Server-local servers (all groups are local).

Moderation comes in multiple levels: pre-post moderation (posts must be approved by a moderator), and post-post moderation (tools available for removing posts). Moderation of messages, moderation of posters (approval, disapproval, removal). Didn’t moderation of posts in Usenet allow moderators to insert notes in approved messages?

Multi-level threading would be good. Easily visible multi-level threading would be good, but I think that that’s a function of the client software more than a function of the groups themselves (FB does multi-level threading, but the client software often stops displaying which comment is being responded to after the first level or so.)

[dsrtao]

> Didn’t moderation of posts in Usenet allow moderators to insert notes in approved messages?

Yes. Moderators can edit everything about the message; whether and how they do so is up to them. Moderated groups tended to have extensive charter documents stating what was up to mods' judgement and what they had to do.

Disemvowellment as a half-way between discarding a post and letting it through started on Making Light.

I'm not sure where shadowbanning started. (A shadowbanned poster is the only one who can see their own posts/comments.)

[brooksmoses]

Some of my general memories from Usenet:

* It was discoverable. Part of this was that although it was large, the group-space was functionally finite. Part of this was that there was some amount of hierarchy. Part of this was being able to follow interesting people to other groups (although I have no memory of how I did that). The result was that I was able to find several great communities about topics that I was interested in, and they were "the" community for that interest.

* The information-transmission system and the UI client were entirely separate. That led to a proliferation of UI clients that met different people's needs. You already mentioned that, but I think one of the critical pieces is that it allowed a lot of programmers to experiment with what made the system work for them, which made for far more innovation than would have happened with a centralized owner of "the UI".

* Local groups were supported using the same system as global groups.

* Although conversations were threaded, there was no assumption that the reader was able to immediately reference the previous message in the thread, and this evolutionary pressure led to a bottom-posting writing style where one explicitly quoted the text that one was replying to before writing the reply.

* More-generally, the time delay in posts was an evolutionary pressure that led to longer-form writing rather than single-sentence replies. This was also supported by the fact that it was possible to reply at any point in the thread -- there was minimal evolutionary pressure to respond quickly before the conversation moved on.

* Notification-wise, replies to threads were equal to "top-level" posts that created new threads. This avoids the problem that Dreamwidth often has, where if a conversation shows up under a week-old post, nobody who's not in the conversation will see it. More generally, although conversations were tree-threaded, the typical reading style was to just read the new posts from the last day, and Usenet readers made this reading style very easy to do. (This feels very rare among threaded forums.)

* Several of the newsgroups I was in died not because of the Inevitable Death of Usenet, but because of specific people who dominated conversations with their personal ideology. They weren't "toxic" in the obvious sense; they were simply deeply committed to an ideology that was at odds with other posters and also capable of writing large volumes of posts, and the result was that conversations in the group became more and more dominated by debating their ideology. All of this was nominally on-topic, and the "death" of the group wasn't visible in a decline in posts, but thoughtful discussion of any other topic got derailed enough that people stopped writing it.

* Groups typically had clear and relatively-easily-findable charters of what was considered on-topic and off-topic. Also, the lack of easy access to back archives meant that well-curated FAQs were common.

[brooksmoses]

One of the particular values of Usenet was that everything in general was public; there was no need to log in (and especially no need to log into a particular group!) to read the posts on a given group, which was a major benefit for discoverability.

Similarly, the lack of any sort of authentication except at the point of ingress of a message into the system had many benefits for simplicity and lack of gatekeeping on new users, but it also meant that spoofing was readily possible.

I'd note that "client-based killfiles" are an emergent result of "UI is separate from the transmission system"; there is no basis for killfiles in the transmission medium. (There was, however, support for filtering on the server side with regards to accepting messages from other servers.)

Pre-post moderation was also somewhat of an emergent result -- my impression is that the only requirement in the system was that messages on certain groups needed to be signed with specific keys. The whole mechanism of "to post to this group, you send your message to a moderator who then approves and posts it for you" was largely outside the core system, and functionally there was no limit on what moderators could do to a post before posting it.

Re: moderation

[brooksmoses]

There was a third form of moderation that was less-visible, because it likewise was primarily an anti-spam mechanism: Server-side filtering of incoming cross-server traffic. IIRC, most servers had extensive blocklists and filters, often based on the metadata of which other servers the message had passed through but also including things like "does the message contain binary data posted to a non-binaries group?"

My guess is that this filtering was also largely responsible for implementing the "moderated group" form of moderation, in that unsigned messages to moderated groups would also be filtered out. (Ideally, they would be rejected by the server where one tried to post them, but relying on that requires trusting everyone else's servers, which generally wasn't the way things were done.)

[brooksmoses]

How do you feel about photos and other multimedia content?

Personally, I feel that it's a very complicated question. Certainly photos can be of significant benefit to a lot of conversations, but they also can easily dominate conversations and be a mechanism for abuse. They also eat server space. IMO, the old process of linking to photos worked fairly well, but it's not at all ideal in that it doesn't work well for one-off images and requires people to have a place to host photos.

[brooksmoses]

This raises an observation of how groups were created. In the standard hierarchy, groups were only created when there was enough traffic to support them, and when a clear community had been developed (in other groups) to seed the new group.

I expect that this was fairly important for ensuring that groups had high-quality communities.

[brooksmoses]

On a different note, I think that while technology is part of the answer, there are other critical parts in the human element.

A particularly instructive example I would point to is https://board.spotlighthobbies.com/. It doesn't look like much, and the software is "some janky old Perl script that's incredibly fragile" (as the page footer now notes) that has some really horrible UI misfeatures, but that community has been there for 25 years and is one of the critical hubs of the hobby that it supports, so it is clearly doing a lot of things very well.

The big pieces that it gets right are obviously community. When Tom Carter started the board, he was a well-known figure in the hobby, and before launch he invited a dozen or two other respected people in the hobby to join and participate. That immediately gave the board credibility, so it attracted other people who had valuable things to contribute to the conversations, and it also meant that there were strong positive influences on the group culture from day one.

Also, at the time Tom was running full-page ads for his business in all the hobby magazines, so he put a blurb about the message board in those, which neatly addressed the "discoverability" problem in a way that simply couldn't be done online in that era (and likely not now either).

Tom actively moderated the board, quickly removing inflammatory or off-topic posts. (And, even though he has retired, the new board owners do similar moderation.) There isn't pre-post moderation, but after-post moderation happens pretty quickly, and I think there are IP filters for blocklisting problematic posters -- not ideal, but effective enough. That, manually-updated keyword filters, and exceptionally rudimentary password-based accounts for posting seem to keep the spam out. When an inflammatory or trolling post gets deleted, everything replying to it also gets deleted, and this plus reprimands means people generally don't engage the trolls.

I think it also matters that the board is attached to a retail company. It drives traffic to the business, which means that maintaining its health is a legitimate part of someone's day job. That's not a necessary condition for someone to be a high-touch moderator over multiple decades or for paying the bills to keep a centralized system running, as the Nielsen Haydens have demonstrated, but surely it helps.

There are a few pieces of the software that are notably useful, IMO. It uses a client-side cookie to record a "last read" time, so when you come back later it will tell you which messages are new so you can find them. The "index" view is compact, so you can scan conversations for interesting things easily -- and it has supported an emergent behavior of putting short replies in subject lines, so for a lot of conversations you don't even need to click through to the message. It supports linking to photos, which was really important for a visual-art medium. And it does an interesting sort of balance where the first page of the "index" view contains only recent posts, but within that set it threads things, so new messages in older threads don't get completely buried. This also provides a recency bias to things that people reply to, without restricting it.

Thinking about that, I think I have another item to add to my list of Usenet features:

* Although conversations are tree-threaded, topic drift is supported. Conversations have subject lines, and while the default is for the subject line to reflect the subject line of the previous message, you can also change it. Client UIs usually make this subject change visible in the index page so it's immediately visible when skimming, and some clients will even use it as a signal to consider the message to be the root of a new thread tree.

* Related to that, although thread trees are usually displayed as trees, they are usually presented to people with a bias towards the recent posts (perhaps only showing the unread messages, in a default view), rather than with an unchanging root. IMO, this promotes conversation expansion rather than contraction, which is valuable for promoting conversation in general.

[brooksmoses]

In summary, though:

The Spotlight Hobbies board shows that while there may be features that are important to a functional system, the necessary feature set is quite small. Also, the fact that this board thrived where no doubt hundreds of others using the same software never went anywhere or died out shows that there are critical human-factor elements involved that have little to do with the software.

More broadly, I think the number of things in these lists that are either emergent usage conventions or are things that client UIs have added on top of the base protocol show that supporting emergent behavior is, if not necessary, at least a very valuable component. I would argue that it's better to provide hooks for behavior to emerge (or at least to evolve) rather than to try to define everything in the software. Less is, in this case, better.

[brooksmoses]

Digging into this a bit more, I think there is a really critical tension between "conversations are generally discoverable" and "you can prevent particular people from seeing your posts". It is really difficult to produce an effective person-level blocklist for content, because fundamentally having a blocklist rather than an allowlist means the "everyone not otherwise specified" set of people is able to see content, and requiring people to have one unique specifier that they can't detach themselves from has a lot of really unfortunate effects.

It's also inevitably permeable, because people share information. Sometimes it's incidental that they're sharing it with the blocked person, as with quoted snippets in a reply, and sometimes it's intentional. Attempting to control that with a blocklist is a fool's errand; if you want something to be secret, don't spread it to almost everyone. (And, IMO, claiming to provide an effective blocklist for otherwise-world-readable content is socially irresponsible.)

A corollary of that, I think, is that delegating access control to servers that you don't trust or control is a bad idea, even if you are allowlisting rather than blocklisting. Use cryptography and access keys, or don't use a federated distribution system for limited-access content.

Interestingly, this sort of access-control system would have been entirely possible to implement on top of Usenet. Instead of putting the sensitive parts of your message in the message body, put a url from which it can be retrieved from an access-gated server. A client could present this entirely transparently, and could even implement message quoting by replacing quotes with urls to message-snippets when crafting reply messages.