Using technology to create a user community around the Sydney Morning Herald.

In case you aren't familiar with Tim Lambert's recent stunt - creating a not-really-a mirror of Tim Blair's site as a platform to put in an alternate comment system - read about it at Troppo and Tim Blair. The bits that are relevant to me are the alternate comment system that was ostensibly about routing around Andrea Harris, Blair's administrator; the bookmarklet that was offered after Lambert took down the mirror; and what it might mean to two of Cam's recent diaries.

I played with the bookmarklet that linked to the open comments more out of perverse interest than anything else; it was actually a pain in the arse because I had to re-insert the links every page view. But I've been playing with an extension for Mozilla Firefox, greasemonkey, in the last few days.

Again, if you are not familiar with greasemonkey, it allows you to supply targeted javascripts to perform any task you care to program. It comes with two scripts, one to convert underlines, non-link text to italicised text, and a second to make links of any non-text that resembles a link.

I decided to re-write the bookmarklet as a greasemonkey script so that it would be run automatically. Which is fine, but in the end, after all the kinks were worked out, I sat back and wondered why I had bothered. Sure, I got some practise is some of the more interesting bits of javascript, but really the whole thing is an exercise in perversity by Lambert. However, we could turn this into a force for Good rather than Evil. We could set up open comments using the comments same service, Haloscan, write a new greasemonkey script and voila, comments on Sydney Morning Herald.

Two issues. Well…three actually; because there is a certain level of screen-scraping going on, SMH - to pick an example from the air - could play silly buggers with the HTML and break the script. This is improbable though because it would have much more impact on them than it would on us. The two real issues are moderation - again - and trademark - not copyright - law. We wouldn't be infringing on SMHs copyright, but we probably would be infringing on their trademark.

The Open Tim Blair Comments script now becomes a proof of concept. My next step will be to write a new script for SMH and try a limited distribution to see what we see.
Cam Riley: South Sea Republic. Freedom, liberty, equity and an Australian Republic.

Comments
  • cam . # .
    Subversion of a closed system: .. always fun for the family. One of the better projects I did was an unofficial patch for a Dynamix game.

    I am not certain your script will gain wide use, Webdiary shows that there is a market for comments to be added directly to the newspapers, which your script would expand, but the majority of commentators like their comments to have their own colours and their own community, ie blogs.

    It will be interesting to see what happens when you release your script. If you need webspace, tell me, we can set something up on the ssr (mini-itx) *cough* server.

    cam
  • cam . # .
    I dont read Tim Blair\'s site: I had a look a while ago and my impression of it was that is was an Australian version of freerepublic.org; it also reminded me of Johnny\'s comment on dailykos.com where he said, \"dailykos is full of people violently agreeing with each other.\"

    I read through the comment thread link posted in the article. I havent changed my mind. Blair\'s site is still freepish .

    cam
  • avocadia . # .
    Widespread use:

    I don\'t know that I care if it gained wide spread use. Sure, that\'d be great, but at the same time it would probably be a hinderance - even if you could moderate the comments to get rid of the lamers and the content-free trollers, it would be a lot of comments. I think this, for me anyway, is more an exercise is seeing it it could be done; it would take a lot more than this to change the world.

    It\'s worth mentioning that what I am trying to do requires Mozilla Firefox, which currently holds (rule of thimb measurement) 5-10% of the browser market. This would have to be wildly popular and fulfill some colossal want in the public eye for it to drive people away from IE towards Firefox, and if that were the case, we\'d probably already have comments provided by SMH themselves.

    Besides, if SMH sent me a letter threatening legal action because of trademark violation, I would likely fold. Assuming it was legal action with merit.

    Thanks for the offer of space. To make life easier on me, I\'ll use a free comments host in the short term. I don\'t want to actually have to write a comment system to handle this; what with this, work, half a dozen other software projects (that I\'ll do when I can buy some more tuits), and actually being involved in my family life I think I might have quite enough to do right now :- )
  • Annotations: I think this links in nicely with your earlier link to the article on public EU Constitution annotation .  They\'re both about annotating a document in a public space from a different perspective.  With just a little more finessing over permissions and suchforth  it might fly.  Tim Blair\'s site seems a funny target though, since commenters on the open site are mostly going to be, as cam said, hysterically agreeing with one another, and disagreeing with him.

    Would it be possible (legal) to set up a headline and intro feed on a newspaper, with comments?  Click on the story and go to the newspaper, click on the comments and go to a scoopish comment system.

    I think that talkeuro site is a great idea.  For a start the content base is solid rather than the more ethereal blogosphere.  Secondly the domain is large - among 400 million EU citizens you\'d think there\'d be hundreds of thousands of political tragics.  The problem would be first to get people going to it, and second to stop it turning into the online flamefest equivalent of the Great War, with memetic trenches carved all over the Union\'s founding documents.

    There\'s some wikipedia-ish problems too.  Genuine technical commentary and explanation of the issues by experts in EU law would be extremely valuable.  I\'d like to be able to distinguish it from the layman ramblings of pseudonymous armchair pundits such as myself.  Laypeople have lots to contribute to the debate - that\'s the whole point of opening up comments - but sometimes it would be useful for an expert to just say \"there\'s no way Law X would get past the European Court of Justice with this clause in the constitution\".  Or even better, five experts saying that and one saying \"maybe\".
  • cam . # .
    Blogs and Specialists: But blogs do this -

    Would it be possible (legal) to set up a headline and intro feed on a newspaper, with comments? Click on the story and go to the newspaper, click on the comments and go to a scoopish comment system.

    - already. Most blogs consist of \"The NYT said this;\" then a three paragraph quote, and the blogger finishes with, \"George Bush is an idiot\". The alternative is, \"Ann Coulter says;\" then three paragraphs and finishes with, \"Liberals are traitors.\"

    I think the mass media has given up the innovation in that area as blogs now consist of the comments section for the newspapers and magazines. Blogs have the added advantage that the blogger gets to put up their own colours and have their friends comment too.

    Genuine technical commentary and explanation of the issues by experts in EU law would be extremely valuable. I\'d like to be able to distinguish it from the layman ramblings of pseudonymous armchair pundits such as myself.

    The benefits of the specialists talking on their subject of authority means that it permeates through to the laymen as well. Catallaxy, Quiggin etc are excellent for that. A lot of the threads bore me, but I read them anyway and know more of economics, social science, law etc for doing so.

    It makes knowledge more accessible and easier to access. Not to mention a friendlier way to absorb than the harsh concrete, cream paint and neon of a classroom.

    A scarce society requires specialists, an abundant society has project to project knowledge IMO. Not sure how that would fit in with a communal euro-law type site.

    cam
  • avocadia . # .
    Blogs as comments section:

    I think the mass media has given up the innovation in that area as blogs now consist of the comments section for the newspapers and magazines. Blogs have the added advantage that the blogger gets to put up their own colours and have their friends comment too.

    The problem with this is that blogs are largely a ghetto of webpages read by five people including the writer. Even if you cobble together working technology - and trackback/pingback/whateverback don\'t count cause they don\'t work very well at this problem - to gather many many blogs into a cloud of likes, threading conversations and all, then you haven\'t got a flashmob, you\'ve got a flash diaspora. It\'s been tried; I know of one very smart lady, burningbird who tried it and, last I heard, ran into a brickwall.

    Maybe del.icio.us tags will help, creating ad hoc folksonomies to pull together threads of many blogs into a cloud, and maybe comment feeds to thread it up, and then getting Blogger and Typepad/Movable Type and LJ (hell, even Scoop and Slash and PHP/Post Nuke) on board...well there\'s a lot of maybes and co-ordination there.

    Or we can get the MSM to play ball and stop being precious. And if not that, we circumvent them.

    Don\'t get me wrong, blogs play an important role in debate. It\'s just that I think it could be so much better.
  • cam . # .
    Google cobbles them together though: Do a search on a story and you get the same repeated AP lines from the MSM as well as the blog commentaries on things. As an example; Nasa Funding Cut by the third page blogs appear, by the fifth they start taking over.

    cam
  • cam . # .
    Another example that appears in SSR\'s stats: alot is Cornelia Rau , your article you published here appears on page 2 of the google search.

    cam
  • avocadia . # .
    Google has the same problem:

    The Cornelia Rau thing is probably an outlier. I\'d love to know who linked to that to push up it\'s Pagerank, because it would help me point out how much of an outlier it probably is. In general, 90% of blogs would have a hard time getting in the first half dozen pages of a google search because a) they have a tiny readership (to the best of my knowledge I can count my readership at avocadia.net on one-and-a-half hands), so they b) very rarely pick up links and thus Pagerank.
  • avocadia . # .
    Search depth: No threading though. And what are the stats on how many people get past the 5th, 10th and nth page of search results? I googled and didn\'t go past the fourth page - which probably says something, but I am at the office :- )
  • Precious: Or we can get the MSM to play ball and stop being precious

    I just had a strange vision of the editor of the Sydney Morning Herald as Gollum.

    This is a good point.  The mainstream papers could implement trackbacks, for example.  The natural centre of the commentary is an originating article, especially when it\'s big news, like Mr Lightfoot in the Oz this morning.
  • Specialists: On the blog side, I was envisioning something like an NYT-tracker site.  Scoop set up to take a feed of the headline and precis, say.  So you\'d read an article in the NYT and then say - I wonder if the guys on NYT-tracker have anything to say about this?

    I think all societies require specialists.  Reading specialists kicking about on their favourite subjects, and then desperately trying to fill in the gaps in your own model of the world, is a great way to learn.

    You read stuff by Virginia Postrel - some great stuff on the way solutions emerge given an opportunity - and she uses technocrat like a swearword.  I don\'t have that view, my view is closer to that of David Foster Wallace - a technocrat is someone putting her expertise at the service of a democratic society.

    To me, in a scarcity driven society a specialist is a magician who has to be certified from years of official study and promoted and underwritten by an institution.   I\'m not going to completely give up on this model.  If I go to a vet it will be one registered with the Queensland Board of Veterinary Surgeons.  

    I agree that an abundance driven society would be supportive of quick and unorthodox sequences of study.  And I\'m certain flocks of fast-moving interdisciplinarians would deliver a freer more prosperous society than squads of square eyed factory men.  But when a person spends 25 years studying the subject they love I\'ll give their opinion more weight, especially when I have little else to go on.

    In wikipedia, where people go for details on obscuria, this is a real problem, as Larry Sanger has pointed out .
  • cam . # .
    troppo, quiggin and surfdom link to us: Surfdom seems to the largest and most consistent of the referrers. Australia is also fortunate that it doesn\'t have a super-crowded MSM all shouting the same thing. And the AU blogs tend to be of very high quality. Again it is not a super-crowded space, and it is a small enough group of commenters that it can reward quality without hiding in undiscriminating tribalism.

    But that ain\'t gonna get us to abundance.

    So Scrymarch and yourself have both had the idea of scraping a MSM publication using their own RSS feed and setting it up for comments. \"SMHTracker.com\" would probably get me sued, but some other name wont.

    I can put up a scoop installation that does that. I woudl suggest we dont scrape the op-eds, and instead supply our own op-eds. lol. Use their AP/Rueters feed, but not the bad parts of their paper.

    We also need a meta-form of mojo that determines when a user is speakign with authority and when they are speaking out of their arse. Maybe it is better if that is a social effect rather than a technologicial one. People earn that over time, rather than with moderation clicks.

    So the cloud is the next thing. You suggested RSS for the comment threads to get it pageranked up. The inter-connected part is hard, but we already do it pretty viciously. People link in their blog posts, there are blogrolls, rss feeds, commenters sigs have links to their URL/blog.

    Maybe there is some social effect here. Maybe people arent embracing the cloud because they havent time or interest. Maybe they arent embracing the cloud because the technology is inhibiting them.

    Anyway I will set up an \"SMHTracker\" scoop site this weekend and we can start playing with, and trying out technologies, that may create - the cloud - the swarm - the flash-mob!

    cam
  • cam . # .
    Specialism: Abundance isnt to remove the specialist, nor diminish their role, but instead to remove the barriers of entry to becoming a specialist, and also stopping their entrenchment once they become reach a position of specialism.

    Kind of like the news cycle, it flits from subject to subject and gives big opinions on all. But the news cycle has static commentators, and doesnt incorporate flocking specialism in their process to any degree.

    The closest would be the talking head shows, like 7.30 Report which gets several specialists on that subject. Then again in the US you have celebrity driven talking head shows with a static groups of specialists who talk on subjects they have no knowledge about.

    It isnt specialism that is bad, but lack of generalism in that case. Or in other words a lack of subject to subject specialism. the celebrity specialists have become static, and their authority and legitimacy assumed as being pertinent to \"all\" subjects.

    I wandered ...

    cam
  • cam . # .
    SMH feeds: The rss feed for the SMH national has a headline, and that is it. No meat there, so any rss \"story\" will have a headline and that is all. From the RSS feeds anyway.

    cam
  • cam . # .
    Pagerank: I am also sigged on hulver and k5. That probably counts for a lot. I noticed that my googlebomb article ranks higher than Kick and Screams on the same topic.

    cam
  • Mojo: We also need a meta-form of mojo that determines when a user is speakign with authority and when they are speaking out of their arse.  Maybe it is better if that is a social effect rather than a technologicial one. People earn that over time, rather than with moderation clicks.

    Yeah, I dunno if that\'s as important on a newstracker site.  It\'s static obscuria like wikipedia or maybe that talkeuro site where it really starts to hurt.

    Also compare and contrast the BBC site which often has user comments at the bottom of the articles.  They can be a bit fatuous though.
  • cam . # .
    SMH/Age rss restrictions: There are a lot of restrictions on the SMH and Age feeds. From their rss channel page ;

    Please make sure that your reader is configured to look for updates no more frequently than every hour. These feeds update only twice a day, so looking for changes less frequently is recommended.

    These channels are for personal use and only in news reader applications. You may not publish headlines from these channels to a web page.

    Ironically scraping their page would probably be ok. If you want to publish their RSS feeds on your page you have to go through a click-through with the following terms and conditions ;

    3.5 You must not display or archive any of the Headlines for more than 48 hours.

    That rules the SMH/Age out. We cant do a haloscan like service over their headlines from the RSS feeds.

    I had a look at some other Australian papers (Australia, Telegraph, Bulletin), and the others didnt seem to have rss feeds. Is there another that has rss feeds?

    cam
  • cam . # .
    Parliament of Australia has an RSS feed: Maybe that would be a good target . They link to PDFS (blech), but that is something that does need a comments section.

    cam
  • cam . # .
    Alright it has changed again!: Now it is for posting comments on Australian Federal Parliamentary Bills and Research. Since the SMH/Age are being precious and other Auian mainstream media sources dont have RSS feeds. I pulled the RSS feed from aph.gov.au/library . I also changed the name to of the slogan to something less elitist/amibuous/confusing.

    I think commenting on bills is cooler than commenting on the mainstream media. I think we should run with this first and see where it takes us - Newcopia , a kind-of haloscan for parliamentry bills and research notes.

    cam
  • cam . # .
    Seeding the Cloud: So with newcopia as it stands, it potentially connects parliament house\'s output with the people. Those with accounts and the will to comment anyway.

    So now we have to connect the old and new media to parliament house\'s output. Since the MSM is precious about its data, through TOS or no rss feed - we will probably need to screen scrape the appropriate outlet/story. It could be posted as a \"scraped\" comment (as opposed to a normal or editorial comment). A word better than scraped would be better though.

    As to the new media, blogs have trackbacks, but that doesnt help media like crikey, which is email based. Maybe it will need to be scraped some other way. No link back to it either. Which would suck. It is an information dead end then.

    Trackbacks do collected too much spam, maybe a user (logged in) can make a \'trackback\' comment to their blog. It would be a manual process. It would also mean there wouldnt necessarily have to be a link in the originating article to the newcopia story.

    Alternatively we could require that sites join up to a trackback list, that will be the first filter (and bannination if anyone starts posting poker/casino spam)

    Since the trackbacks are a comment, people can comment directly off of it, if they so wish. Same with scraped comments from the MSM, that can be commented under directly as well.

    Any other ideas?

    cam
  • siento . # .
    Another target: Wouldn\'t the ABC\'s site be worth taking a go at?

    After all, we pay for it!

    And they have RSS feeds and all sorts of stuff.
  • siento . # .
    And the link: Sorry, a link to ABC politics
  • cam . # .
    ABC TOS: The ABC News RSS feed has this;

    ABC RSS feeds are protected by Australian and international copyright laws and are available for personal, non-commercial use only.

    Since the server is in the US, and the convention of \"fair use\" is judicially defined, I dont believe I will be violating US copyright law in posting them to a website.

    cam