Saturday, April 16, 2011

gnome terminal unlimited scrollback

A recent, and very handy feature that came out in gnome terminal (my currently preferred terminal program, basically because I use gnome), was unlimited scrollback. I thought I'd try it for a while, to see if it would have any negative effects on performance (particularly memory use), as it'd be a useful thing to have, as there have been a number of times I've lost output that I would've liked to have kept off the top of the buffer.

The first thing I did was to fill the terminal with a whole lot of output, to see what would happen. My preferred way of doing this was simply $ base64 < /dev/urandom, which avoids the problem of strange things happening due to the non-printable characters. After running that for a while, I didn't notice the memory usage increasing, but also couldn't find any new large files in my home directory, or /tmp. Which left me a little confused.

They're probably using some kind of fancy compression was my next thought (despite random data being impossible to compress). So I changed my command to run as $ base64 < /dev/urandom | tee tmp to make a temporary file of the output as well as spitting it out. Compressing that with xz, bzip2 and gz all shrunk it to about 70% of the file size (remember, it's base64 encoded random data, so we can actually compress it). As a sidenote, the compressed base64 random data was about 5% larger than just the pure random data - which itself can't be compressed. So this led me to think that they couldn't be compressing it all away to virtually nothing, it's got to be somewhere. (Meanwhile, I left the command running, filling up my terminal).

My next port of call was to have a look at the source code. So I git cloned the gnome-terminal repo, which it turns out uses another gnome library, vte, which I also cloned. I could find where the setting was defined, and it turns out "unlimited" just means there are a max G_MAXLONG in the buffer (which is effectively infinite). I also found mention in a comment that it was saving it to disk, but I couldn't find where it was saving the data.

But then I saw the light. I could just look at what files gnome-terminal had open, so lsof to the rescue. Then I found it was being sneaky, it had a number of files called /tmp/vteXYZ1tv open, but it had already deleted them. Thus you can't see them when browsing, and they will be removed when the program closes. This makes sense, it means that when the process is closed, it doesn't matter how (at least I think), the space of the files can be reclaimed, i.e. we won't get leftover files on a program crash, or a kill -9. They can be restored though, my way (there are probably others), was to do a ls -l /proc/<gnome-terminal pid>/fd to see what they point to. Then you can cat these to make a new file. These are just a verbatim copy of the terminal output. No compression. No nothing. As it turns out, one of my terminal history's was almost 900 MB! But that was only after random data being spat out very quickly for quite a while, unlikely to happen in ordinary usage. For the foreseeable future though, I think I'll keep this option checked, as I can live with a bit of disk usage for the handiness.

4 comments:

  1. Amazing, this is super useful, thanks!

    ReplyDelete
  2. FYI, a security perspective: http://www.climagic.com/bugreports/libvte-scrollback-written-to-disk.html

    ReplyDelete
  3. Years later... vte-0.40 adds compression and encryption to these files.

    ReplyDelete