Emptying “Deleted” Files

A few days ago, I woke to find my home partition 100% full and programs erroring out because of it.

“No problem,” I thought and fired up ncdu Install via APT to figure out what had eaten up the 12GiB I’d had free the night before. (Nothing beats ncdu for efficient workflow though FileLight Install via APT is a good alternative if you’re willing to trade keyboard shortcuts for a GUI)

Nothing showed up so, on a hunch, I tried using lsof Install via APT to see if it was a deleted file being held in existence by an open file handle. Bingo!

…it was the ~/.xsession-errors that I’d deleted weeks ago, it was being held open by half the apps on my desktop (including my session manager), and I had in-progress tasks that I couldn’t interrupt to restart my session. Here’s where it got tricky. I had to somehow get rid of a file that was already “deleted” when I no longer had a name to reference it by.

Thankfully, after a fair bit of searching, I found a solution. The key lies in how, on Linux, you can access all open files via /proc if you know the process’s PID, even if they’ve been deleted. (It’s the second column in the lsof output. You’ll also need the file descriptor number, but you can trial-and-error that using something like less)

Once you’ve got that, you can use a root shell to truncate the file to a length of zero.

you@host:~# > /proc/<PID>/fd/<FD>

(Yes, that’s using a non-appending redirect without providing a command to redirect output from)

I hope this helps if you ever find yourself in a similar situation.

Posted in Geek Stuff | Leave a comment

A little tool for command-line playlist building

For the last few years, I’ve been amassing a collection of little scripts I use every day to build playlists, both for Audacious and for MPlayer.

About a week ago, I realized that they’d started to duplicate each others’ functionality and refactored them into a single script which has different default behaviours depending on what it’s called as (but you can also use command-line flags).

So far, here are the behaviours it implements based on what you call it as (either via MPRIS [1] [2] or using a command provided via --exec such as mplayer):

aq
Add the given paths to the playlist.
ap
Like aq but start the first one playing too.
laq
Like aq but use locate -i to search for the first argument, filter for known media types and filter for the following arguments, then display a chooser.
lap
Like laq but start the first one playing too.
raq
Randomly select -n NUM songs (default: 10) from the paths provided (default: XDG_MUSIC_DIR) and add them to the playlist.
rap
Like rap but start the first one playing too.

…and all of these are just aliases to options listed in --help.

Until two days ago, the chooser was a simple little thing where you were given a numbered list of filenames and you typed in a sequence of numbers, optionally including Python-style start:stop slice syntax.

Now, if you’ve got urwid installed, you get a nice ncurses-style pseudo-GUI, complete with mouse interaction… though the scroll wheel support is preliminary and I’m still trying to figure out how to set up Tab-based focus switching.

Here’s a screenshot:

I haven’t had time to fully bring the codebase up to spec, write the last few new features, and write a test suite for it, but it’s good enough for my day-to-day use so, if you want to give it a try, it’s on GitHub in ssokolow/profile as home/bin/lap. (It has its own GitHub repo now.)

The queue option in the chooser can be toggled via the Q key or Meta+Q and, once you’ve selected all the tracks you want using Spacebar or the mouse, press Enter to commit.

In case I ever move it to its own repo and forget to update the link, you can also grab the revision at the time of this posting.

Posted in Geek Stuff | 6 Comments

Trifles Make Perfection

TL;DR: When you’re doing something creative, look for details which, with only minimal change, could greatly broaden your work’s appeal and staying power. (Also includes examples.)

Every now and then, I run across a creative work which frustrates me, not because it’s bad, but because the author made one tiny “unimportant”  decision that severely cripples either the story’s longevity or its appeal.

The most common example I run across is fanfiction where, aside from a few too many mentions of something like MySpace (as opposed to more generic terms), the story could easily last at least a decade or more without feeling strange. (Though breaking the atmosphere is also a common problem with mentioning things we recognize as contemporary brands in fanfiction since the source material generally occupies a different mental universe from what we encounter in day-to-day life.)

Hence, I appeal to people. When you’re doing something creative, please take a few minutes to think about what other people might care more about than you do. Ask friends to help if you can spare the time. I just don’t want to see good things fading into obscurity because of a minor oversight.

How about a more direct example. Two days ago, I was discussing musical tastes with a friend and, to clarify what she meant by “fan-song”, she sent me a YouTube link to a My Little Pony song.

Now, in order to get my point across, I need to be clear about my opinions. I have nothing against My Little Pony: Friendship is Magic but I’m not a fan either and neither of us approve of the Brony subculture. I treat it the same way I treat Care Bears… and some of my fondest childhood nostalgia relates to Care Bears but I don’t still watch it. MLP:FiM is a well-done show for people much younger than I am and, if my young self had access to it, I’d probably be nostalgic for it too.

Despite that, I’ve been listening to “Lullaby for a Princess” a lot. It’s a truly beautiful song and the lyrics are very cleverly phrased at times. (eg. “but such is the way of the limelight, it sweetly takes hold of the mind of its host”)

It’s relevant to this topic because, were it not for four uses of the word “pony” where other two-syllable words could readily have been substituted, it could easily transcend and outlive its fandom as a masterful expression of a common literary theme.

For those who haven’t had a need to google up the wikia page yet, here’s a little explanatory backstory: A thousand years before the first episode of MLP:FiM, Princess Celestia and Princess Luna ruled together, the elder sister with domain over the day and the younger over the night, both immortals.

Living in her sister’s shadow, Luna ends up feeling unappreciated and, one day, can take it no longer. Consumed by her pain and corrupted by how her emotions have fed into her magic, she refuses to allow the morning to come, determined to force the kingdom to appreciate the beauty of her domain. (as covered by “The Moon Rises“, which is also beautiful but was recorded before the fan group in question found a female member to do the vocals.)

Celestia is forced to choose her responsibility to her subjects over her love for her sister. With Luna refusing to back down, Celestia takes over dominion of night and banishes her sister to the moon for a thousand years to allow her emotionally charged self-transformation time to weaken. (“Lullaby for a Princess” is sung following this event, exploring Celestia’s guilt at her own blindness toward her sister’s pain and the responsibility she feels for how things turned out.)

Obviously, not all songs can be made more generally appealing in this manner. For example, “The Moon Rises” has a point where the lyrics include “the ponies of Equestria” and, even if I were to suggest a replacement for “ponies”, I can’t imagine an easy way to remove the kingdom’s name without weakening the lyrics.

However, in Lullaby for a Princess, the four uses of the word “pony” can be replaced very naturally. In the context of the song’s narrative, Celestia is just as much a “princess” as a “pony” and, with only a few seconds of thought, I can see that “no pony” could easily be replaced with “nobody”.

In fact, I think that such changes would make the lyrics more powerful for two reasons:

  • The song is about a sovereign regretting that she neglected her sister to the point where she had to banish her to protect the kingdom whose “limelight” she’d been blinded by. Referring to her as “a princess” rather than “a pony” supports the theme more strongly.
  • Compared to something like “nobody”, the one use of “no pony” feels a bit superfluous and impedes the immersiveness of the story for non-fans. (Sort of like fanfiction authors who include random bits of Japanese, not because they lack good English equivalents or because it aids comprehension, but because they think it’s cool.)

That’s not to say it’s not a beautiful song nonetheless. I’d still recommend it to anyone who can appreciate it despite its context, but it could have been so much more broadly appealing with such a small change.

To quote Michelangelo, trifles make perfection, and perfection is no trifle.

P.S. For anyone who likes those two songs, another fan did a cover of “Lullaby for a Princess” with modified lyrics named “Luna’s Reply“. Her voice isn’t quite as suited to the melody and, from what I read on wikia, I’m not sure if Luna’s un-corrupted personality would still exist in a state distinct enough from Nightmare Moon for it to be canon-compatible, but the lyrics are still excellent and the song is nice to listen to.

In case of dead links:

Update: Perfect Picture (MP3) by StormWolf is an excellent example of doing this well.

It’s nowhere near as deep as the ones I mentioned before, but it’s not supposed to be. It’s a catchy techno-styled musical summary of a famous fashion photographer who just happens to be a pony.

There’s nothing specifically restricting it to the MLP fandom because the lyrics focus on her character, rather than needlessly pointing out the fact that, like everyone else in the setting, she is non-human. From the lyrics to the musical styling, it’s a catchy musical summary of a character which also works as a catchy summary of a type of person if you have no idea who Photo Finish is.

That means it has a good shot at getting shared well beyond the fandom and surviving as “another catchy song” when MLP eventually wanes.

Posted in Otaku Stuff, Writing | 2 Comments

Making Coveralls work with PyPy and skip Python 2.5 on Travis-CI

I have a couple of projects that get tested on Travis-CI and I just discovered Coveralls, a tool which integrates with it to provide a code coverage badge to go alongside your build status badge. (and is also free for open-source projects)

Unit Test Status Coverage Status

Unfortunately, it seems that there’s a bug in the current Python client that causes it to error out when submitting coverage data from PyPy builds. (It tries to grab source for compiled PyPy modules with no source available and then dies when it can’t).

Here’s the boilerplate I use to skip it on Python 2.5 (which it doesn’t support), make it complete successfully on PyPy, and omit things like the unit test framework itself from the report:

Posted in Geek Stuff | Leave a comment

How To Invert Your X11 NumLock LED

TL;DR: numlockx off; xdotool key Num_Lock

I like to leave NumLock on all the time but, with my current keyboard, the indicator light is a blindingly bright blue. I noticed that the numlock light would sometimes get confused, so its behaviour was reversed (only on when NumLock was turned off) but I couldn’t figure out why until now.

Here’s the deal:

First, under modern versions of X.org with MPX support, the XInput system lets you have one or more master keyboard-pointer pairs, each which can be mapped to one or more physical keyboards and mice, called slaves. (You can examine this by running xinput in a terminal)

Here’s an example of what xinput outputs on my system:

⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ ATI Remote Wonder II                      id=12   [slave  pointer  (2)]
⎜   ↳ Logitech USB Gaming Mouse                 id=13   [slave  pointer  (2)]
⎜   ↳ Wacom BambooFun 6x8 stylus                id=14   [slave  pointer  (2)]
⎜   ↳ Wacom BambooFun 6x8 eraser                id=15   [slave  pointer  (2)]
⎜   ↳ Wacom BambooFun 6x8 cursor                id=16   [slave  pointer  (2)]
⎜   ↳ Wacom BambooFun 6x8 pad                   id=17   [slave  pointer  (2)]
⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
    ↳ Power Button                              id=6    [slave  keyboard (3)]
    ↳ Power Button                              id=7    [slave  keyboard (3)]
    ↳ Guest Barcode Reader                      id=8    [slave  keyboard (3)]
    ↳ CYKB23 USB Keyboard                       id=9    [slave  keyboard (3)]
    ↳ CYKB23 USB Keyboard                       id=10   [slave  keyboard (3)]
    ↳ zc3xx                                     id=11   [slave  keyboard (3)]

(For the curious, zc3xx is the button on the side of my USB webcam which triggers the Logitech software under Windows by sending a keycode that maps to the XF86WebCam keysym, same as on a multimedia keyboard)

Noteworthy implications of this design:

  • It’s possible to create more masters and move devices over to them. If you do, you get extra mouse pointers on your screen and, if your window manager supports it, you can have multiple people using several different applications at the same time on the same desktop.
  • Every master has a virtual slave called XTEST, which is used to inject events.
  • Each slave (physical keyboard) can have its own LEDs, but the mod2 modifier that actually controls NumLock behaviour is part of the master.
  • X11 doesn’t yet know how to force all the slave keyboards on a given master to share the same LED states. (Because, if you want to support all the different kinds of keyboards out there, it’s very complicated)

This means that, if you press NumLock on one keyboard, it will turn NumLock on/off for the whole master/slave grouping, but the light will only change on that one keyboard.

The problem I was having was that older versions of the numlockx command were faking a NumLock keypress on the XTEST virtual keyboard, rather than my physical one, and then pressing NumLock on the physical keyboard just toggled the LED and the modifier while assuming they were still in sync.

numlockx has now been fixed, but you can still trigger this mismatch manually if, like me, you want the light to come on when NumLock is turned off. Just use this command:

numlockx off; xdotool key Num_Lock

The first half puts both your NumLock modifier and LED into a known state (off) and then the call to xdotool presses NumLock on the virtual XTEST keyboard, which turns on the modifier and the invisible/imaginary LED on the virtual keyboard, while leaving the LED on the physical keyboard untouched.

As long as you only change your numlock state by pressing the button, they’ll stay out of sync until the end of your desktop session. Enjoy!

(Thanks to Peter Hutterer for explaining things)

Posted in Geek Stuff | 2 Comments

Error 400: The Web’s Most Under-Appreciated Response

TL;DR: You should respond with Error 400 if the URL is malformed or the parameters fail validation rather than with Error 404.

If you’ve ever run a website which reports 404ed requests to the admin, you’ve probably noticed that, every now and then, GoogleBot gets it into its head that it can do a better job if it tries guessed, nonsense URLs to try to tease hidden pages out of your site.

The big problem is that, most of the time, it’s guessing such utter nonsense that you have to wonder about the competence of whoever wrote it. (Not to mention that it’s effectively punishing the whole web because some sites are poorly structured)

That’s where HTTP Error 400 comes in. Error 404 is so overused as a generic “nope” response that it doesn’t say anything about why the request couldn’t be serviced.

By sending Error 400, you tell the requesting agent that the request couldn’t be serviced, not just because nothing matched, but because the URL given doesn’t meet the minimum requirements to be tested for a match. (Sort of like if you were to write “If-Modified-Since: pineapple” in your HTTP headers)

That means that, if someone or something is trying to guess URLs, it at least has enough information to learn the constraints for URL validity so it can only try to guess ones that have the potential to exist.

Given that, you might be wondering how to implement Error 400 responses. The simplest solution is just to tie them into your URL routing and input validation.

  1. Did the URL router fail to find a matching route? Return Error 400.
  2. Did the parameters fail to pass input validation (eg. “bike” in an integer field)? Return error 400.
  3. Did the URL request an object ID that doesn’t exist? Return Error 404.
  4. (optional) Did the URL request an object ID that has been flagged as deleted (with no replacement on record) but kept around to preserve relational integrity? Return Error 410 (Gone).

With modern web frameworks, this is often ridiculously simple to concisely retrofit and, as with Error 404, you can still return custom human-readable messages.

Posted in Geek Stuff | Leave a comment

Schedule Importer for TaskWarrior

I just wrote another useful little script. This one takes a CSV representation of any schedule in this form and imports it into TaskWarrior for easy time management

Week 1 Read pages 1-25
Week 2 Read pages 26-50
Do Exercise 1
Week 3 Read pages 51-75

The week column can contain anything. The script will warn you if it doesn’t recognize it but it’s really just using the presence of text in column 1 to mark the changeover to a new time period.

Using it is also quick and easy:

  1. Copy-paste the table from whatever format you got it in (HTML and MS Word tables work for me) into a spreadsheet tool like Excel or OpenOffice/LibreOffice Calc.
  2. If necessary, delete columns so column 2 holds task names.
  3. If necessary, alternate between Delete and the down arrow, rapid-fire, so only rows which mark a new time period have content in column 1.
  4. Save the file as CSV.
  5. Run the script like this:
    python taskwarrior_schedule_importer.py ~/schedule.csv \
        --start=2013-02-01 --deadline=2013-06-30           \
        --project=EXMPL101 > ~/schedule.json
    task import ~/schedule.json

All arguments are optional. By providing a deadline, you let script automatically adjust the definition of a week if there aren’t enough weeks before the given date.

If you want to use it for periods other than weeks, specify --increment=DAYS to override the default of 7.  The script will use the smaller of --increment and the value calculated from --deadline if both are provided. See --help for more details.

As usual, here’s the download link for if you don’t want to scroll past the embed. I’ve only tested it on Lubuntu Linux 12.04 LTS under CPython 2.7.3, CPython 3.2.3, and PyPy 1.9.0, but it should run anywhere.

(It was surprisingly complicated since TaskWarrior currently has a very primitive line-by-line JSON parser that reminds me of the parsers I wrote using things like sed and regexes when I was young and ignorant. Take a look at the last few lines if you want to see how I mixed json.dumps(indent=0) and string-formatting operators to securely and robustly get the newline use task import demands.)

Posted in Geek Stuff | Leave a comment