linux

Passphrase recovery from regex approximation

Due to the current memory-loss inducing holidays, you may end up forgetting your seldom-used (or recently-changed) login password, SSH or GPG passphrase. If you still have some recollection of what it looked like, one way back in is to generate a wordlist from a regular expression approximation, then feed it to a cracking tool like John the Ripper. Yes, this means cracking your own password.

Let's say your password was the hard for humans to remember Tr0ub4dor+3. But you don't remember the various capitalizations, which letter did you l33t, which punctuation mark and suffix you used in a feeble attempt to slow down a potential attacker. You can generate a plausible wordlist using regdlg, the regular expression grammar language dictionary generator

regldg -m 15 -us 255 '[tT]r[0o]ub[@a]d[0o]r.[0-9]!?' >/tmp/mywl.txt

Here -m 15 sets the maximum length, while -us 255 enables the regexp period to match all alphanumeric characters plus punctuation and symbols (the regexp universe). Then, download and install a “jumbo”-patched John the Ripper:

git clone https://github.com/magnumripper/JohnTheRipper
cd JohnTheRipper/src
make linux-x86-native
cd ../run

Now generate a password file using unshadow, ssh2john or gpg2john (you may want to delete irrelevant lines from the output)

./gpg2john ~/.gnupg/secring.gpg >/tmp/gpwd.txt

and crack the password:

./john --show --wordlist=/tmp/mywl.txt /tmp/gpwd.txt

There's usually no point in enabling the --rules john option (or writing custom John rules) as they don't deal with the kind of variation pertinent to approximate password recall.

Tips

  • Be sure to delete your cracking script (if you saved the above commands to one) and your .bash_history, or better yet change your passphrase after recovering it.
  • Of course, there's also the BOFH method of recalling passwords by recreating the scenario which made you dream them up in the first place, but it may not always applicable (or definitive in recovering all details).

Gstreamer video player with variable speed support: pb2player

I've recently had the chance to play with an older Nokia Maemo phone. Having a full Linux (Debian, no less) distro on a phone is quite a treat, which explains why N900 has somewhat of a cult following. While trying to watch some video lectures, I discovered that the default media player (which is GStreamer-based) doesn't have any variable speed support. On the other hand, the players that do (vlc / mplayer and their front-ends) don't necessarily have hardware acceleration, because only GStreamer can access the DSP. As is the case with many embedded platforms, Gstreamer enjoys special support.

As PyGST / PyGTK are friendly platforms, I ended up coding pb2player, a variable-speed Gstreamer video player that works both on the desktop and on Maemo devices. I had to restrict myself to Python 2.5, which in practice wasn't too bad: there's no str.format(), and I had to use optparse instead of argparse (which is aggravated by gstreamer's buggy takeover of optparse once pygst is imported — one has to do all CLI processing before the import). The only cross-platform issue I've encountered is the appearence of GTK FileChooserDialog, which leaves too little room for long file names and sometimes makes it difficult to navigate to the parent directory.

The player is basic, but quite useful and solid in my experience. It supports m3u playlists (a priority given the FileChooserDialog woes), faster / slower playback and random seeking within the current video. The variable speed playback is nice for skimming the boring parts of a clip. Pitch control (a la mplayer's scaletempo, so that the voices in your head-phones don't start squealing at high playback rates) would be even nicer.

If you're going to try out pb2player, I suggest grabbing one of the releases or the stable github pb2player v1.0.x branch. After finishing the first workable version I decided to test-drive Qt, which seems to have a much more usable file dialog on Maemo; Therefore I refactored the app according (loosely) to MVA in order to be able to keep both front-ends (in the short term, because we all know how bridging different philosophies works out in the end). That experiment is not yet finished.

Once downloaded, you can run pb2player from its source folder, or you can build a Debian package to be installed elsewhere (but not on Maemo — see below). I put the Debian instructions in a mydebian folder, as having an actual debian/ in your source tree makes the package "native Debian", which gets in the way of real packagers most of the time. The make debuild target creates a .deb in /tmp based on mydebian.

To build and install a Maemo-suitable .deb, two more steps are required:

  1. Maemo wants .desktop files in a non-standard location (/usr/share/applications/hildon), so mydebian/rules must patch the default $(desktopdir)
  2. What is now called python-gobject-2 on a Ubuntu Precise system used to be called python-gobject in the past (which now refers to version 3). This is a result of Debian's moving target naming policies. The PC-generated playbin2player requires python-gobject-2. As a workaround, we can build a dummy package using equivs that depends on pgo-1 and provides pgo-2.

I've uploaded debian/rules and equivs control file for building python-object-2 (as well as the two resulting .deb packages) to this talk.maemo.org thread, so I won't include the details here. Upload and install python-gobject-2_1.0_all.deb and playbin2player_1.0.1-1_all.deb to your Maemo device and you can run pb2player.

Prevent overheating (and thermal shutdowns) with cputhermalfreqd

For some of us, relief from noisy fans, overheating laptops and thermal shutdowns is a can of compressed air away. For the hardware-impaired, though, there's now cputhermalfreqd, which I've written over the summer as the prudence required to keep my laptop online (stop video player every so often, suspend for a while before burning a CD etc) began to take its toll on my patience.

cputhermalfreqd is a daemon that controls the CPU speed using the cpufreq drivers. Unlike other similar daemons, it is
focused on slowing down your CPU once the system gets too hot, thus preventing inopportune thermal shutdowns (and thermal
shutdowns are never opportune).

cputhermalfreqd starts with a list of temperature thresholds (e.g. 23 18 12) which represent degrees below critical temperature. At every iteration (you can specify how often the daemon checks the sensors), cputhermalfreqd checks all temperature sensors and spots the sensor that is closest to its critical temperature (e.g. one of the CPU sensors or the hard disk). That remaining “thermal headroom” is used to locate a particular interval between the initialization thresholds and dial down the CPU to the corresponding speed step (e.g. if the headroom is 20, that would be the second-highest speed available in the system in our example.)

Internally, the program discovers the available sensors by strace-ing the sensors command (from the lm-sensors package) as it boots up. It then regularly uses cpufreq-set to adjust the speed as necessary. I know, it's not necessarily classy, but it works and it's portable.

So, anyway, grab cputhermalfreqd from github. Or wait for the summer, if you're in the northern hemisphere and are currently using your laptop as a heating device...

Tags: 

Memtest86+ from a USB stick, the easy way

My MSI Wind U100 has finally arrived (a few days after Christmas when I was expecting it, but still in 2008 luckily) and it has been exhibiting several strange Windows crashes. Since the U100 version I ordered comes with a "bonus" 1024M of RAM, which (by my understanding) are not installed by the OEM, but by the online store that sells the netbook, I naturally suspected memory problems and reached for Memtest86+.

Unfortunately Memtest86+ does not run from Windows or Linux as "normal" software does, because it needs to replace whatever OS exists and trash the memory as part of its job. For most folks the easiest way to run Memtest86+ is to burn the distributed ISO image and boot from the CD (or boot it off a floppy for the few that still have such a peripheral). But netbooks don't have CD/DVD drives. The only workable option then is to run Memtest86 from a bootable USB disk.

There are many tutorials on how to create bootable USB disks using things like syslinux, isolinux, makebootfat, but most of these are boring to even skim, let alone put in practice. After looking around for a while I found a simpler solution:

  • Download unetbootin
  • Run it and install a FreeDos image on the USB stick (unetbootin has a specific menu option for this — it will download the FreeDos setup files for you automatically)
  • Download Memtest86+ — use the "Pre-Compiled EXE file for USB Key (Pure DOS)" version
  • Unpack the zip file (it should contain a single executable) to the root directory of your USB disk.
  • Ensure legacy USB support is enabled in BIOS
  • Boot off the USB stick and choose one of the LiveCD options (not "Install"!)
  • At the DOS prompt, change drives by typing "c:" (this will take you to the USB disk) and run the Memtest86+ executable (e.g. mt211.exe).

Memtest86+ will then run on your system for an hour or so (hopefully telling you nothing is wrong). At the end, you will need to reboot the computer, because (as mentioned above) Memtest86 completely replaces any running OS. But you weren't likely to stick around in FreeDOS any longer anyway.

This still takes several steps, but it's light, non-error-prone GUI work and doesn't require handling a lot of disparate components like other methods. You can use this method for running other DOS-only executables (e.g. legacy software or BIOS flashing programs.)

Tags: 

Catch up with the audio and music tools on Ubuntu Hardy

Updates: per readers' request, I have added Hardy backports to all packages mentioned in this article (except Pulseaudio where the appropriate section isn't clear) in my PPA. I have also added newer ALSA packages.

Many Ubuntu users have chosen to stick out with Hardy, the Long Term Support (LTS) release, supported until April 2011 (or 2013 for the server edition). But support mostly means security fixes, not necessarily updated versions of popular software; the Hardy Backports project doesn't necessarily keep pace with our favorite packages. In my case, I wanted the latest version of Rosegarden (1.7.2), the audio / MIDI / score editor and sequencer.

Fortunately, there is a way to enjoy fresh software while still postponing the dreaded dist-upgrade (or clean reinstall) marathon for as long as Hardy remains supported. The solution is to download source packages from later distributions, compile them on your system, and install the resulting binary packages.

You can either add entries to /etc/apt/sources.list and then run apt-get source ... (disruptive because it enables "future" versions for all packages), or use dget manually:

dget http://ftp.debian.org/debian/pool/main/p/pulseaudio/pulseaudio_0.9.10-3.dsc
dpkg-source -x pulseaudio_0.9.10-3.dsc

Either way, to build the package you then

cd pulseaudio-*
debuild -uc -us -b
cd ..; dpkg -i # your packages here

(if the build fails you may need to install extra -dev libraries). Some good places to scout for updated versions include:

In all cases, you can go directly to a package by appending its name to the above URL's; and once you got to a package page, you will find a link to the .dsc file you need to dget (as explained above) on the right side. Keep in mind that not all packages will compile on Hardy — you will have to experiment.

Back to audio packages — the reason I mentioned the Debian Sid pulseaudio package is because (unlike Ubuntu) it includes module-jack-sink, which allows Pulseaudio to run on top of the low-latency jackd daemon. This means you can have Jack and some music applications (like a soft synth and Rosegarden) running perfectly, and still be able watch YouTube videos (without having to kill and later restart Jackd). As with any Pulseaudio on Hardy setup, you will still need libflashsupport in order for Mozilla to be able to connect to Pulseaudio (same with Opera).

I also recommend the following: jackd 0.1.116 (again from Debian Sid); Fluidsynth 1.0.8 (from Intrepid); qjackctl 0.3.4 (from Ubuntu Jaunty); and finally, rosegarden 1.7.2, for which I couldn't find any distribution to leech from — use the version in my PPA (obtained by updating the Intrepid package to the latest upstream release). The PPA page contains the instructions for enabling it in APT.

Tags: 

Resuming a file copy operation

If you ever need to interrupt (and then resume) a slow cp operation (e.g. from a USB stick or over NFS), you will appreciate cURL’s support for the file:// scheme:

curl -C - -O file:///media/memstick/file.avi

will resume the copy (and display a nice progress report as well).

Here are some alternatives that don't quite work:

  1. The more famous (and widely-installed) wget doesn't grok file:// URIs
  2. rsync reads both the source and destination files (in order to compute checksums), so there is no speedup
  3. Mr. Hartvig's clever recp script uses dd with a block size of 1, which is slow and CPU-intensive

Of course, you can use curl instead of cp to begin with, if you like the progress bar and don't mind the extra keystrokes.

Tags: 

X, with and without an external monitor

As a laptop user, I often find myself switching between LCD-only, external-monitor-only, and dual-screen setups. Read below for a summary of how to achieve this flexibility under X (more specifically Xorg), both statically (via multiple configuaration files, requiring X restarts) and dynamically (while X is running) — but also some of the gotchas you will run into.

  1. Some static configurations
    You can have multiple xorg.conf configuration files, but they all must reside in /etc. To start Xorg with a specific configuration file, use, for example:
    startx -- -config xorg.conf.external

    If you've already started X, you can also start a distinct X session by specifying a new display number:

    startx -- :1 -config xorg.conf.external
    • A configuration that disables the laptop screen: in the Device section of xorg.conf.*, add
              Option "monitor-LVDS" "LVDS"
      

      Also add a Monitor section:

      Section "Monitor"
              Identifier "LVDS"
              Option "Ignore" "True"
      EndSection
    • The same effect can be achieved using TwinView for NVIDIA cards:
      Section "Screen"
          Option         "TwinView" "True"
          Option         "MetaModes" "nvidia-auto-select, off"
      EndSection
      
    • To enable both screens, you can use a vanilla xorg.conf (as generated for example by sudo Xorg -configure); xrandr can then configure dual-head, as described in the next section. However, I have noticed that under this setup X disables the XVideo support (meaning, for example, a slower mplayer); I don't know if there's a way to avoid this problem.
  2. Dynamic configuration
    #disable laptop screen
    xrandr --output LVDS --off
    # switch back to laptop screen
    xrandr --output VGA --off
    xrandr --output LVDS --auto
    # dual-head (laptop + external)
    xrandr --output VGA --above LVDS
    # --left-of, --below etc. also work

    For the last xrandr command (dual-head), your combined external + laptop virtual screen resolution must not exceed the virtual desktop size. If not specified in xorg.conf, the X server pre-computes it at startup as the highest resolution of all monitors connected to your computer (i.e. if you start with your external monitor disconnected, the laptop's resolution; if the external monitor is connected at start-up, it will most likely dictate the virtual). Therefore you will most likely want to specify the virtual desktop size:

    Section Screen
      Subsection Display
        Depth 32
        Virtual 2048 2048
      EndSubsection
    EndSection

    However, as a further twist, some cards lose graphics acceleration capabilities when the virtual size is too high. If you notice your browser scrolling a page slower than normal, for example, this may be to blame.

Tags: 

Debian / Ubuntu packaging: Zorba XQuery

Today I uploaded Ubuntu source and binary (Gutsy and Hardy) packages for Zorba, the new C++ streaming XQuery processor. The Ubuntu PPA system (Personal Package Archives) is a great service; without it, you'd need to host an APT repository in order to conveniently distribute packages that are not (yet) part of Debian or Ubuntu (especially since a Debian source package is actually three files).

In fact, my source package works in Debian unstable too; as there is no custom Debian Sid APT repository (Ubuntu PPA only serves Ubuntu distros), here's what you need to do to build and install it:

  • dget the .dsc file (which pulls the original tarball and a .diff.gz as well)
  • run pbuilder zorbaxquery_0.9.1-3.dsc (apt-get install and set up pbuilder if you don't have it)
  • retrieve the .deb's from /var/cache/pbuilder/results/

It would be really nice if someone set up a PPA-like service for Debian, at least for repositories of source packages. I realize that setting up a cluster of build boxes is possible only with someone like Canonical behind. But the required storage for source packages could be quite small: if the *.orig.tar.gz "link" would dynamically retrieve an archive hosted elsewhere (a webapp could do this, trading space for bandwidth), such repositories could be quite compact (the .dsc and .diff.gz files are usually tiny). Alternatively, this scheme might work with a modified apt that could recognize HTTP redirects.

Pages