Back in September, I had a blast speaking at Nickel City Ruby. My talk was entitled “Apprenticeship: Software Craftsmanship’s Missing Link” and included a lot of slides of sasquatches. The video has been posted to Confreaks, so I’m embedding it here. Also note that I have posted my notes and a few resources over on the microsite I created for this talk, http://blog.mattgauger.com/apprenticeship/
When I first starting talking about building a chording keyboard, both in person and online, people began to ask me about the possibility of building a computer foot pedal. With the Teensy microcontroller, a foot pedal would only need to watch a single digital input and output a few characters to the OS; much simpler than my chording keyboard project.
There are quite a few commercial products you can buy. They come with their own caveats:
- Kinesis makes several, but they can only be programmed on Windows. They are also very expensive.
- The Xkeys Foot Pedals look good, but they are also expensive.
- The DealExtreme Triple Action Foot Pedal provides three buttons and is relatively cheap; however, the reviews I’ve read say the reliably is poor. This product can only be programmed on Windows.
Building the foot pedal
My first idea for a foot pedal was to use DIY guitar pedal hardware. Such guitar pedals have nice, sturdy aluminum project boxes, robust foot switches, and look pretty good. But the ergonomics of such a pedal worried me: while a guitarist momentarily stomps on guitar pedals to turn them on or off while standing, a programmer would likely be tapping and/or holding a foot pedal for long periods of time, most likely from a seated position.
Such a pedal would look something like the above, but with a USB cable coming out of it. If you wish to use guitar pedal hardware, I suggest checking out Mammoth Electronics. (I receive no compensation for mentioning them; I have been a happy customer for several years.) For a two-to-three switch foot pedal, I suggest the 1590BB enclosures, and Mammoth can drill them in several ways for you.
The sustain pedal is designed to plug into a digital piano, so it has a 1/4” audio plug on its cable. In the past, I’ve built a lot of guitar effects pedals and some amps, and so I have a lot of 1/4” jacks around.
The wiring is simple: The Teensy (and most Arduinos) can do input pullup resistors for us. I mocked up the circuit with a little breadboard (ignore the weird angle of the Teensy here; it was required to get the pins into this small breadboard.) I downloaded some example button code to the Teensy and verified that it was working.
To assemble, we simply solder a digital IO pin to one side of the 1/4” jack, and the other wire gets soldered to ground pin. My button is going to be connected on pin 9.
I had a small project box, much bigger than the Teensy really needed, but suitable for the job. I used my Dremel to cut a round hole for the 1/4” jack, and a rectangular slightly bigger than the mini-USB cable plug for our USB cable.
Test fitting the Teensy in the project box:
Lastly, I used some velcro inside to attach the Teensy to the project box. All done with assembly!
The next step is to program the Teensy to send the key events we want. In this case, my coworker Josh suggested a vim clutch that enters insert mode when you press down on the pedal, and leaves insert mode when you release it. Since the Teensy’s Button class detects both button press and button release events, we can write code to do that.
Here’s what the code looks like:
(You can find the code in my chording keyboard repo on Github here.)
There are, of course, a few gotchas that I ran into:
KEY_ESC constant that is referenced by the Arduino documentation didn’t work. Similarly, sending the hex value and character code didn’t work. I couldn’t find anything online that suggested that the Mac has some different ASCII character for the Escape key, so I had to find another way to leave insert mode.
With the Arduino’s Keyboard class, we can build up a key combo by calling
Keyboard.press for each character in the combo, and then finally calling
Keyboard.releaseAll when we’re ready to send the key combo to the computer. Since vim also has
Ctrl-[ as a way of leaving insert mode, I created that key combo on lines 34-36.
Of course, this code could be made to be even more robust by guarding against, say, inserting an
i character when you are already in insert mode. Most likely, you’d send Ctrl-[ (and move building the key combo to a function we can reuse) and then send vim the command
:startinsert. I’ll leave implementing the more robust solution as an exercise for the reader.
Does it work? I used it while writing this blog post, but I must admit, I have a strong natural reflex to hit ESC to leave insert as soon as I finish typing a word or sentence. However, when consciously trying, it is quite natural to use the foot pedal to enter/leave insert mode. One downside I found was that the Ctrl-[ combo seems to back up the cursor one character, which can be annoying.
For my personal use, I am thinking about mapping the key it sends to
Left-Alt so that I can use the footpedal while in emacs’ Org Mode and not have to move my left hand down to hit Alt with my ring finger every time I want to adjust a heading or start a new heading.
Overall, this pedal is much cheaper than the high-end pedals mentioned above. The total cost came to around $37, because I had some of the parts on hand. Further, you can upload new code to the Teensy on all operating systems, a big win over the Windows-only pedals above.
However, this pedal only has one foot switch. You could easily add more M-Audio sustain pedals to the design, and keep adding 1/4” jacks to a project box. The Teensy has plenty more IO lines to use! Or you could go for it all in one enclosure with the guitar pedal hardware.
If you have any questions or comments, I’d love to hear about it over on Twitter, where I am @mathiasx.
Chording keyboards have been a computing peripheral for a long time. They’ve been in use at least since Douglas Engelbart gave what is known as The Mother of all Demos in 1968. Rather than try to sum up the history and potential of the device here, I recommend you read the excellent essay Engelbart’s Violin by Stanislav Datskovskiy on loper-os.org
I’ve always had an interest in building hardware. When I was younger, I dreamed of building robots some day that I could interact with, play games with, or maybe go to space with. Someday I hope that I’ll get that chance. But in the meantime, I’ve built all sorts of hardware, and I’ve always been interested in the low-level side of computing.
At Bendyworks right now, we have flurry of activity and interest focusing on low-level computing, digital logic, and hardware. We’ve had several discussions in off-hours about learning computing from first principles: digital logic, circuit design, and taking that knowledge to the point of designing and building CPUs and then computers around those novel CPUs. There’s interest in the Arduino and some of my coworkers are taking the so-called
nand2tetris course: Building a Modern Computer from First Principles.
We recently built a project at Bendyworks that we call concert_cam, which takes pictures at the Live on King Street concerts outside our office. You can see a gallery the pictures taken at the most recent concert on Facebook. At some point I’ll blog about the
concert_cam project and provide some lessons learned.
But for now, I’d like to talk about a project I’ve been working on for awhile: a chording keyboard. I don’t have a clever name for it, so the code currently lives at github.com/mathias/chording. The repo represents only some of the attempts I’ve made at getting it to work. (Code from several other attempts to code a chording keyboard died in a VMware Linux VM that ate its own disk image, and I didn’t commit any of that code. Luckily, that code was mostly things I’d ruled out as possible solutions.)
Building a chording keyboard in software is not trivial, so I have moved on to building one purely in hardware.
USB devices that we use for input, like mice and keyboards, implement something called USB HID, for Human Interface Device. Linux, Mac OSX and other operating systems have supported USB HID for a long time.
To build a chording keyboard that can work on any computer without special software installed, the device will have to implement USB HID and send the correct key events for a given chord.
There are lots of chording keyboard projects and commercial products out there. To name just a few:
- The Microwriter, which the author of loper-os.org discusses in the essay Engelbart’s Violin, and also covered reverse-engineering.
- The twiddler, which is commercially available. I have used my coworker Chris’s twiddler quite a bit, but couldn’t get over the TV-remote-control ergonomics.
- The tabspace layout for the twidder (PDF link), which is an optimized key map layout for Twiddler.
- The septambic keyer by Steve Mann.
- The chordite keyboard.
- The spaceman spiff layout / spiffchorder project.
- The gkos project to create a software chording keyboard for smartphones.
Ultimately, none of these projects or products really fit what I had in mind for a chording keyboard.
Why a chording keyboard, you might ask?
Well, I spend most of my work day editing text. While I am quite proficient at vim and slowly getting better at emacs, the kinds of key combos that a professional programmer uses daily are quite complex. There is a constant risk of RSI or carpal tunnel (which plagued me in my teens, but I have been free of for over a decade.)
With a chording keyboard, one could take a common key combo that you use all the time and put it under a much easier-to-type chord. This becomes especially attractive to me in replacing some of the key-combos required for operating emacs (especially those that use alt.)
Of course, such a device isn’t designed to completely replace having a keyboard on the desk. Even with a lot of practice, the reality is that I’ll probably continue to be faster at typing words on QWERTY. But having a chording keyboard off the left side, and a mouse on the right, seems to make sense. I am, after all, a software craftsman, and if I feel like I need to build a particular tool (even a physical piece of hardware) to augment my current toolbox, then it makes sense to do it.
For a long time, I have dreamed of a way of being able to record programming language keywords and idioms as macros, as well as methods to type faster. Since I haven’t found any way to train myself to type QWERTY faster, I started to look at alternatives. Learning Dvorak or Colemak might help, but it still doesn’t get me away from the legacy typewriter keyboard design.
1 2 3
(where the underscore shown is the cursor position)
And lastly, I wanted to build a chording keyboard because it’s cool, it’s different, and because we frequently accept the paradigms handed to us in computing without really thinking about how we might improve or replace them.
To understand chording keyboards best, it is worth noting that while regular keyboards detect the initial key being pressed down, a chording keyboard must detect when all of the keys in a chord are released simultaneously. Due to how fast electronics really are, no human could actually release all of the keys at exactly the same moment. For that reason, we need to implement chording in a way that takes time into account and looks to see if any chord pattern matches in the past n milliseconds.
Brief project history
After reviewing possible hardware to use to build a chording keyboard, I stumbled upon the Razer Nostromo (and its earlier incarnations from Belkin.) These gaming pads are popular with PC gamers because it puts the WASD keys under your left hand and frees your right hand up for the mouse.
Other features the Nostromo had over other devices were supporting N-key rollover — basically, the ability for the keyboard to know that multiple keys are being pressed at once, and send all of those keys to the computer — as well as a little 4-way directional pad, and blue lights (always a plus!)
The Nostromo also comes with a piece of software that you can install and remap the key mappings with. The key mappings get saved onto the device, meaning the device can operate without the software. The current mode is indicated by three LEDs near the thumb, which count up in binary, yielding 8 key maps. Additionally, any of the keys can trigger a pre-recorded macro of many key events.
Sadly, while the Nostromo software can have one key press trigger a sequence of buttons for you, for example, to macro a complicated action in a game, it did not support detecting a chord (multiple key presses simultaneously) into a single key event. So with that limitation in mind, I set about trying to write code to detect chords and turn them into key presses in software rather than at the hardware layer.
On Linux, there is a project called mxk.
mxk is basically a USB HID swiss army knife and HID event remixer, able to take input from any USB device, run it through various rules, and then output new USB HID events to the USB bus with a “virtual” USB HID device. Many example configuration scripts are provided, and one of the most common use seems to be to create a key press that turns a QWERTY keyboard into a DVORAK keyboard and back again, without having to change the Linux system settings. It seemed promising.
mxk even indicated that it had support for two different kinds of chording: braille keyboard chording and a simpler chord-matching function.
After banging my head against its confusing configuration syntax for about 2 weeks, I determined that
mxk just couldn’t do what I wanted. Two issues cropped up: in the braille chording function, only one row of keys was supported and two hands were assumed. (Basically, it wanted ASDF and JKL; keys to form chords, and didn’t support anything else.)
The regular chording functionality seemed like it might work, until I ran into a huge, glaring issue: A key used in one chord couldn’t be used in another. For example, let’s say I’m mapping keys for the left hand on a QWERTY keyboard to be used in a chord. So I might have a chord like RE which maps to ‘a’ and another chord like FE that maps to ‘k’. The E used in both chords didn’t work in
mxk. It simply couldn’t support it; it could only watch for unique chords. This limits the key map for a chording keyboard made of a 12-key grid from some 495 combinations to 12, gaining us nothing since individual key presses are just as efficient.
With these limitations discovered, I tried to hack the code for
mxk to support my use case better. But, I don’t have much in the way of C chops when it comes to device drivers and pointers, and implementing code against Linux’s libHID is kind of a pain. Further, the
mxk source isn’t very well documented, outside of very few comments littered around the code. Additionally, I found libHID’s docs unapproachable.
I realized that I was working at the wrong abstraction level, like so many projects that had frustrated me before. So I decided to move up to something at a higher level, with the hope that someone else had figured out the headaches of low-level libHID implementation. There are many high level libraries built on libHID. But since there isn’t a whole lot of need for them in most computing, they tend to be abandoned, very old, simplistic, or all of the above. I also was frustrated that my attempts so far would lock me into only using a Linux machine, as my normal work environment at Bendyworks is Mac OSX. We pair all of the time, so I would not be able to use my chording keyboard whenenver I paired with someone else.
Another project that I found that seemed promising was Plover. At first glance, Plover doesn’t seem to be the right kind of software at all. It is an open source stenographer program that allows transcriptionists and court stenographers to turn a regular QWERTY keybord with n-key rollover into a high-end steno tool. The specialized hardware that court stenographers use can cost thousands of dollars, so an open source project that implements it for free on a $100 keyboard is a great win for those in that field.
Despite coming mapped for stenography and transcription — a system which I find very confusing — at its heart, Plover is a cross-platform Python GUI app that supports turning chords into words or phrases. This seemed like exactly what I needed. Going through the source code, I noticed that different keyboards got different configuration classes, and set about trying to create a class to map the keys on the Nostromo to the key events that Plover was expecting.
I struggled with understanding the key event names and how they combined (since I don’t have a background in stenography) and eventually, lost the work due to the VMware Linux VM crash mentioned above. Taking it as a sign that I should not continue down the route of using Plover, I put aside the chording keyboard project for many months.
Teensy and a return to hardware
Originally I had tried to avoid hacking hardware so that I could focus on what I do day-to-day: write software. But in the end, I felt like I had exhausted my ability to get this project off the ground by only writing software.
When we built the
concert_cam project at Bendyworks, the first version of the button pedestal we built utilized a Teensy board — basically, a small Arduino-like microcontroller. The huge feature that the Teensy provides over other Atmel boards is that it comes out-of-the-box with a firmware that sets it up as a USB HID device.
The basic process for programming and using the Teensy is this:
- You program it in the Arduino IDE like other Arduino boards
- Anything you “println” will be sent as if someone had typed it on a USB keyboard.
- Additionally, there is a Mouse class that you can use to send mouse movement, click, and scroll wheel information.
- When you upload your code to the Teensy, it reboots, and immediately starts behaving like a USB HID device as far as the operating system is concerned.
concert_cam, it was easy enough to have a big arcade-style button send the keyboard event of “p” (for press) to the Raspberry Pi that the Teensy was attached to. A script on the Raspberry Pi watched for this keyboard input and told the camera to take a picture. It was very quick to get that working.
concert_cam button pedestal with Teensy and Raspberry Pi getting wired in:
Confident by my success with using the Teensy in the
concert_cam, I decided to put a Teensy in my Nostromo, replacing the normal USB board inside it and translating keyboard chords into key press events at the hardware level.
The first step is to open up the Nostromo. This is pretty easy; just unscrew all the visible screws, and find & unscrew the other screws underneath the rubber feet of the gamepad. The Nostromo comes apart into three pieces, and looks like this inside:
The first problem encountered is those little grey ribbons - they’re ZIF surface mount stuff, and everything else inside is surface mount here, too. For an electronics hobbyist, surface mount components can represent a big source of frustration. For me, I just didn’t want to deal with them. I can’t really solder to the cables with any precision and the spacing of the connectors was so small that there was no hope of attaching wires there. Further, the circuit boards had traces running inside but didn’t really have any pins that I could solder to. A real bummer.
I thought that maybe I could get more of the plastic cable connectors (the little white and black connectors on the main circuit board), but with a regular pin spacing, not surface mount. All attempts to find such a part were unsuccessful. So I had to get creative.
The Nostromo uses membrane keys, which most laptop and PC keyboards nowadays do. A long time ago, almost all keyboards used “mechanical” key switches – switches that had springs or other tension devices and sent a keypress when continuity was made between two contacts. The loud, much-loved IBM Model M keyboard counts as a mechanical keyboard. Gaming keyboards and a few hardcore programmers (like many of us at Bendyworks) are holdouts for using mechanical keyboards, and a few companies still make them. For mechanical key switches, the current winner is the Cherry MX line. (To learn more about mechanical keyboards and Cherry MX switches, I recommend this guide.)
Cherry MX key switches are not all that expensive, and I’d only need 15 to convert the Nostromo. I put in an order for the Cherry MX browns from wasdkeyboards.com and they shipped them out fast. I figured I could get away with using the keycaps from a number pad, which are also cheap, and so I threw those into the order as well.
However, by replacing the key switches, I was getting into a realm of hardware hacking that I was trying to avoid: fabbing physical things from scratch. For this project, I wanted the keyboard to be sturdy and reliable; I didn’t want it to fall apart or move under my hand or rattle. Making a mounting plate that they keys snap into is the typical way that hobbyists build their own mechanical keyboards. And lots of people do, over on sites like deskthority.net and geekhack.org, but without a really well stocked workshop or a Makerspace membership, I didn’t want to go down that road.
Luckily, some quick googling turned up a forum post in which the author replaces Nostromo key switches with Cherry MX keys. The mounting plate for the membrane keys in the Nostromo requires only a little dremeling to convert. Since I was also throwing out the existing USB circuit board for the Nostromo, I can get away with removing the membrane switch PCB and soldering a cable directly to the Teensy’s pin headers, getting rid of those pesky ZIF cables.
Currently, I’m waiting for the key switches to arrive so that I can cut down the white mounting plate for each switch. My plan is to wire up each key to an IO line on the Teensy. Typically, keyboards use multiplexing in a grid to let the computer know which key was pressed: essentially, rows of keys are connected to one set of pins and columns of keys are connected to another set of pins, and when a pin for a row and a column goes HIGH, you know which key was pressed based on it being the intersect of that row and column. The issue you run into here is, again, complicated by chording. While the microcontroller can figure out reasonably well individual keys, there are issues on some keyboards with ”ghosting” other keys when chording.
Since I only have about 15 keys and maybe another 4 or 5 input/outputs that I need, I can get away with using separate IO lines on the Teensy 2.0. The Teensy++ 2.0 has are even more IO lines, which would make it better suited to projects with more IO lines needed. Not needing diodes to ground, not having to wire up a matrix, and not having to detect two pins makes both the hardware and software here much simpler.
While waiting for the key switches to arrive, I began work on the one non-surface-mount component in the Nostromo that I could solder to: the scroll wheel. On the scroll wheel’s PCB, there is a mechanical encoder, a button (the click when you press down on a mouse scroll wheel) and an LED.
Here’s a quick link for reading up on how mechanical encoders work in mice, and particularly, how to interface one with an Arduino: Mouse Scroll Wheel Sensor and Arduino. There’s also some source code in that thread that will probably prove useful later. Note in the picture that the middle leg of the encoder is ground, and the two ends correspond to the signals sent when you rotate it — you’ll have to figure out whether the scroll direction matches their location yourself by experimenting.
I used part of my Friday Growth Day at Bendyworks to start wiring up and coding the firmware for the Nostromo’s scroll wheel. The basic idea is that the mechanical encoder has two outputs. You monitor both lines on separate IO pins on the Teensy, and whichever IO line goes HIGH first is the direction of the scroll. With the Teensy’s convenient Mouse class, you can then just send a scroll event to the OS. In a matter of minutes, I had my scroll wheel scrolling web pages. Not bad.
I’ll be implementing a variation on the sliding buffer for key events, so I will probably also use that code for handling scrolls (over a much shorter, separate buffer.)
Since I was trying to understand the mechanical encoder and could only really connect alligator clips to one pin at a time, I wrote up a quick Arduino script to detect only one direction of scrolling. Here’s that code:
In my repo on Github: scroll_wheel.ino
Note that this does not implement a sliding buffer or direction detection.
Eventually, I’d like to break my scroll wheel code out into an Arduino library that takes the two input pin numbers and a velocity as parameters to the constructor, and implements the scroll direction detection. The library would send the scroll event for you based on the velocity you passed in. But that will have to come later.
I’m going to be getting the key switches in the next week and beginning the hardware hacking to install them.
On the software end, I’d like to write a script (probably in Ruby) that takes a standardized keyboard mapping file and converts it into a C++ header file that the Arduino can utilize for its chord mappings. That way, I don’t have to maintain the header file manually or know what C++ constants map to what, I will simply edit my simple key mappings file and regenerate the code.
The twiddler_keymap.txt file in the repo represents the current plan for my keymap. It is based on the Tabspace layout mentioned above in the Background section. I haven’t found any reason to not use the Tabspace layout, as it seems sensible and leaves plenty of room in the unmapped chords for me to implement key combos, programming language idioms, and text editor movement.
One last decision I need to make is whether to replace the Nostromo’s thumb directional pad. The current thumbpad uses membrane switches similar to the keys, and is all surface mount with very little access to its signals. One thought is to replace the 4-way pad with a Playstation-controller style joystick from Sparkfun.
With this joystick, I’d be able to implement either mouse movements or arrow key movements, and could probably toggle which it behaves like with the small button located above the directional pad. However, securely mounting the joystick into the Nostromo case so that it is durable might prove an issue. I’ve put off making this decision until I have successfully hacked the mechanical switches in.
I’ll be blogging about each step of finishing this chording keyboard as I go. Stay tuned for the next post about installing up the key switches and wiring them to the Teensy.
I’ve got a lot of projects, and I put them down regularly to focus on something else. That list of projects continues to grow and grow. So I don’t feel very guilty that I’ve put down this project for awhile and am just now picking it up again. I’m making good progress, and find it interesting rather than frustrating, so I will continue work on it.
In truth, this project doesn’t really represent much value outside of the things I’m learning as I go. I have no ambitions of turning this into a commercial product, and while some people might find my notes here useful, I’m not trying to make a repeatable project that others can build part-for-part. Is it worth it? Definitely. Is it for everyone? Probably not.
One thing I’d like to do after completing this device is begin work on a custom computer. Specifically, I want to learn enough digital logic concepts to design a whole computer in a language like Verilog or VHDL, and then burn the design into a FPGA board.
Such a computer would not be intended to compete with your Core i7 quad core – I’m thinking of implementing a unique architecture with the overall processing capability of something like an early 80’s microcomputer. It would be interesting, but not entirely useful for day-to-day computing.
When the computer is built, I’d begin implement an operating system on top of that computer in Lisp, along the lines of what the author of loper-os.org has been working towards for many years.
Such a project may take a decade or more to realize, though, and so is not undertaken lightly or easily accomplished.
If you have any questions, comments, or your own chording keyboard project, I’d love to hear about it over on Twitter, where I am @mathiasx.
By “augmenting human intellect” we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by “complex situations” we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers–whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human “feel for a situation” usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.
Eli Pariser’s talk, Beware online “filter bubbles” recently hit the front page of the TED Talks website. I found it interesting because it discusses some of what I’ve seen happening online, and some of my fears for how we search and consume content. Watch the video if you haven’t to see just how much is already being filtered on the web for you.
Right after I saw the video, of course, Facebook announced that they’ll be rolling out a new version of the the news feed. They are quoted as making it more like a “personal newspaper” — a filter, essentially, for the things Facebook thinks you want to see. As the video above pointed out, they’ve already been filtering out the things they think you don’t want for awhile.
No doubt there will be some backlash as people adjust to changes on Facebook, but very few will abandon it. And if you are the kind of person that would abandon Facebook over filtering, or privacy, or concerns about owning your own data, you don’t have a lot of choices of where to go.
Sure, you can specifically seek out services that aren’t going to give you a filtered world view — DuckDuckGo comes to mind. But that’s not always possible. The problem is bigger than just the search engine you use and whether or not you’re logged into Facebook. Just about every site is run by someone else, and they are analyzing you constantly. They are in a brutal battle to keep you as their consumer and keep you from going to other sites. But are the things that they think you want to see what you really want or need to see?
As a developer, I’m capable of taking matters into my own hands, to some degree. Most of the sites that I use on a daily basis have an API and allow me to export or scrape my personal data. But the cost of switching off of the nice service, the well-designed UI/UX of an official mobile app, and so on has kept me from taking the plunge to export my data and go set up shop on my own version of various web services.
Recently I revamped a very old, empty repo that I had on Github. The point of this repo was code a way to export all of my data from Last.fm. But for whatever reason the repo has been empty for something like 4 years now. So the other week, I sat down, checked the Last.fm API docs, and wrote a very simple Ruby script to dump out all the JSON it could about my top tracks, artists, and albums over the past 7 years. That script is available as my birdsong repo on Github.
So far, I don’t have a real use for the data. I could try to use some visualization tools on it to make cool graphs or maps. Or I could try and get analytics out of it: genres listened to, how they’ve changed over the years. But for now, I am content to just have the JSON data.
All in all, it’s around 30 megabytes of JSON data, which is really just plain text with no compression, so it is really quite a lot of data. There’s more I can do, though. I plan to do to start taking control, to both aggregate my own data and filter it.
For a long time it has bothered me that Google Reader stopped receiving new features, and the features that existed only went so far. To be fair, Google Reader has been a rock solid web service for many, many years for me. It allows me to read my feeds quickly, reliably, and has parsed feeds well — all problems I’ve had with other feed readers. It’s always had some neat features like showing me analytics of what I read.
But as far as discovering new content and helping me to eliminate content I don’t read or don’t want to read, Google Reader is not so great. Reader is terrible at suggesting new feeds to read to me; it constantly suggests Dilbert and Lifehacker, and has for the past 5 years or so. I have no interest in either of those. It has never analyzed my reading patterns to such a degree that it suggested something new that blew me away. I am looking for those kinds of interesting suggestions: not just for feeds that match what I already read, but things that are outside of my normal bubble but would be interesting to me.
Luckily, Google Reader has an API, and rather than just exporting my data and building a new service, I can start to build off of it. I get to keep a lot of the features I enjoy while extending it with my own code.
In Pariser’s talk, he talks about encoding algorithms of filtering and recommendation with a sense of civic duty. To some degree, this means having some journalistic integrity. Such an algorithm needs to present both sides of the story. One issue of many feed readers and other content online is that, well, it only shows one view. The view of the article you’re on.
But imagine a feed reader that was more like the front page of Google News. It would not only show you the blog post you’re reading, but all recent blog posts from other authors about similar topics. Maybe, if they’re responding to a news event or writing about a known fact, the reader could do the work to track down the original source. To take it even further, without even really needing to be aware of motive, politics, and other factors, a dumb feed reader with good suggestions could probably present both sides of the story. Both sides of an argument. Both liberal and conservative takes on the same bill.
Dreaming up features like this can be a deep rabbit hole. Start considering the consequences of pulling up all past articles you’ve read about similar keywords or tags. Or performing searches for academic papers on the topic. Pulling in data from Wikipedia. Looking up books you’ve read or are planning to read on Goodreads that are related to the blog post you’re reading. Or any other number of ways to slice and dice content. And it doesn’t have to stop with Google Reader.
Opportunities to make better tools and more intelligently consume information are all around us. At the same time, consumer-consuming corporations on the web try to trap us into filter bubbles. They try to provide us with what they think we want, but can they ever really know? In the end, it’s control-your-own-filters or be filtered1.
1 The name of this article was lifted from Daniel Rushkoff’s Program or be Programmed, a book which I didn’t really enjoy. It was not what I thought it would be about: why we should learn to program so that we can control the complex technical systems around us rather than be controlled by them, or how programming’s problem solving skills can be applied elsewhere. Instead, I found the book to be some technology-fear-mongering and a bunch of diatribes about how things online or in computers are “less real.” Suffice to say, I did not enjoy it. ↩