There’s probably a better way to do this. (If so, leave a comment!) However, I run a debian server for a small task and use the default MTA Exim for mail delivery of status updates to my main mail accounts. Usually I run Postfix or Sendmail. I even have to support Qmail on occasion. However Exim is unfamiliar territory for me — I’m an Exim n00b.

Recently I decided I wanted to remove the “frozen” messages seen by running the mailq command. For example:


# mailq
48h 1.6K 1NUQLB-0007BC-QL <> *** frozen ***
user@domain.tld

48h 1.6K 1NUQO9-0007CW-1u <> *** frozen ***
user@domain.tld

48h 1.6K 1NUQOG-0007CY-1f <> *** frozen ***
user@domain.tld

48h 1.6K 1NUQOG-0007CZ-1v <> *** frozen ***
user@domain.tld

48h 1.6K 1NUQP4-0007DH-Af <> *** frozen ***
user@domain.tld


I know they’ll be purged eventually, I just wanted to clear the queue for some quick testing I was performing. The documentation I found on-line mentioned running “exim -Mrm <id>” for each message id in the mailq output. Being lazy, I don’t want to run that each time — I want a simple one-line command to purge frozen messages. So, I wrote the following BASH script that parses the mailq output and executes the command for each message id:

for i in `mailq | awk '$6 ~ /^frozen$/ {print $3}'`; do exim -Mrm $i; done

Worked like a charm! Message queue emptied of frozen messages.

This is a hard post to write since it is absolutely embarrassing to me and may cause me to lose serious geek-cred. However, I hope this is a useful mea culpa for others’ benefit. This morning I spent a while revising a script that accepted external input which I borrowed (and modified) from someone else. The original script used the external input safely – my modifications made it a true security nightmare. Basically, the script took content from the Twitter API and rerouted it through Jabber (XMPP) protocol. My change also sent the content through email.

If I had used the built-in SMTP features of ruby this post would have been unnecessary. However, as I was in a hurry (the excuse of a scoundrel) I simply piped the output through BASH to the mail command via ruby’s system() function. Also, I took no precautions to escape the external input from being executed by BASH itself. System admins and top-notch developers are now crying with laughter.

Why did I do this? Besides being an idiot, I did this because this script was a “one-off” and not something I was paid to develop. I needed a feature and, in the spirit of a developer (not a system administrator or security professional), I looked for the path of least resistance. With the system() call I could use my knowledge of the Linux command-line to accomplish my task. Since the original script was written in ruby, a language I have not used much at all, previously, I was able to add my needed feature in minutes. A couple more minutes would be all that was necessary to find out how to use ruby itself to accomplish this result. Laziness won over thoroughness and correctness. Result: a *huge* security hole.

Here’s what prompted me to invest in learning to add this feature in the Right Way™: a friend of mine who is an überGeek realized that, based on a clue I gave out about my script’s performance, he could have his way with my server via Twitter. I was Twitter-pwn’ed. Here’s what he sent me (knowing I watch my tweets via this script):

@rjamestaylor $(touch /tmp/test)

@rjamestaylor $(ls -la /tmp/test)

After he sent these two tweets my script sent me an email with this:

-rw-r--r-- 1 root root 0 Dec 27 03:39 /tmp/test

Yes. BASH executed his tweet (!) and as ROOT! So, if someone had tweeted “@rjamestaylor $(rm -rf /)” my server would be a barren husk of hardware. Worse, someone could have used my script — via Twitter! — to setup privileged user accounts, send spam (though small), attack other systems, etc. Why root (or as my geek friends would ask, WHY IN GOD’S NAME “root”?)? Simple: I was in a hurry. Stupid.

Lesson: spending the little bit of extra time necessary to add features without violating security protocols is always worth it. Sure, I could have just bought some CodeOffsets to assuage my guilt, but it was actually better to (1) shut down my script until (2) I fixed the security issues.

Turns out adding net::smtp to a script in ruby is painless. Now I have my feature and no exciting side-effects from rushed, careless development. In fact, most security holes I encounter in others’ code is due to the same thing: rush to add features without a review of implications. Anytime one takes external input from a user or the Internet (or Twitter) one must sanitize the input from unintended effects. This goes for SQL injection (parameters, people!) as well as simple command expansion attacks, among many other possible vectors.

So, go ahead and tweet BASH-expandable tweets to me, my code is not just offset, it’s fixed.

Thanks everyone for reading along. I hope you’ve enjoyed reading these
tips as much as I’ve enjoyed writing them.

For the last installment, I wanted to share a very cool feature in VIM
that I am just beginning to learn how to use. VIM provides support for
Perl, Python and Ruby so that you can use these languages to create
functions in VIM.

I’m most familiar with Python myself, so here are a couple of examples.
In VIM, try the following:

:python print "Hello world"

You’ll see ‘Hello world’ show up in the status line at the bottom of the
screen. Cool, but not all that handy (though you can use this as a
quick calculator, e.g. :py print 256 * 8 ).

To actually get Python to do something interesting with the contents of
your editor, you can define a VIM function that uses Python to do the
heavy lifting.

Here’s how:
:function! PySort()
python << EOF
import vim
b = vim.buffers[0]
x = b[:]
x.sort()
b[:] = x
EOF
endfunction

The :function! line begins a function definition. VIM has its own
internal scripting language, which is swell and all, but if you already
have familiarity with one of the other supported languages, you can use
that language to get a jump start on seriously tricking out VIM.

The line:

python << EOF

tells VIM that we’re defining a block of Python code. The block will
end with the line “EOF”. The enclosed lines are pure Python.

First, we import the vim module:

import vim

Now we have access to that module’s components, like the buffers[]
list. Just like in normal Python, buffers[] is zero-indexed, so
buffers[0] is the first buffer.

Next, we copy the contents of b as a list into the variable x:

x = b[:]

Then we sort that list alphabetically:

x.sort()

Then we replace the current buffer with the sorted context of x:

b[:] = x

And voila – you can now sort the current buffer alphabetically by
calling the PySort() function:

:call PySort()

This is just a trivial example, but hopefully it gives you some ideas of
what can be done with Python (or Perl, or Ruby) inside of VIM.

To get more information about using scripts in VIM, try the following
help commands in VIM:

:help python-vim
:help perl
:help perl-using
:help ruby-vim

Thank you all for reading. Anyone interested in exploring any of the
topics discussed in this or previous tips should feel free to contact me.

Happy VIMming!

- David Roth

VIM’s got a lot of helpful commands that start with g. Not sure why g,
of all letters, but there you are.

One that I find really helpful when running through Apache
configurations is gf. If you put your cursor over the path to a file
and type:

gf

VIM will open up the file that’s under your cursor. So, given a vhost
like the following:

<VirtualHost *:80>
DocumentRoot /var/www/html
ServerName example.com
</VirtualHost>

If you put your cursor over /var/www/html and type gf, VIM will open up
a directory listing of that directory. Or, if you’ve piped the output
of httpd -S to a file and you put your cursor over the path to a
particular virtual host’s configuration file, you can open it up in the
same way. Note that if you’ve updated the current file and it hasn’t
been written to disk yet, VIM will complain that the current file hasn’t
been saved yet and gf will fail.

Another handy use of g is when you’re editing a file with wrapped
lines. Ever find yourself on a line that is wrapped, and you want to
make a change that’s right below the cursor, so you type j or hit the
down arrow, but the cursor skips right past what you want and goes to
the next line? Don’t you just HATE that? Well, if you type gj or
g<down arrow> instead, the cursor will move according to displayed
lines, not logical lines. Much better.

Finally, my favorite stupid VIM trick. g? will ROT-13 text. Use the
ggVG sequence to highlight your entire file. Then type:

g?

And presto – every alphabetic character is shifted 13 places to the
right in the alphabet. Quick and easy obfuscation.

Happy VIMming!

- David Roth

We went over creating VIM macros in a previous tip. If you save your
macros to a file, you can have VIM run the macro against a file,
effectively creating a batch mode for executing VIM macros.

Let’s say that, for whatever reason, you want to convert a file to all
lower case. Create a file called lower.vim and put in the following lines:

ggVGu
:wq!

Make sure there are carriage returns at the end of each line. Those
commands will jump to the first line (gg), turn on visual highlighting
mode (V), then jump to the last line (G). The ‘u‘ command at the end of
the first line will translate all highlighted characters to lower case.
Then the next line writes and quits the file.

Now you can apply this script to any file with:

vim -s lower.vim /path/to/file

Maybe you want to translate all your XML files under the current
directory to lower case. In that case, you can combine VIM with ‘find’
to do some quick batch processing:

find . -type f -name \*.xml -exec vim -s lower.vim {} \;

Note that when you use VIM in this way, VIM does not operate in the
background. An instance of VIM is started for each file being
processed, and the commands in the script file are run like a macro. If
your script file makes a lot of complicated changes, this can be really
amusing to watch. I run this on my computer at home instead of having a
fireplace.

Happy VIMming!

- David Roth

VIM ships with a number of plugins, which are scripts that enhance VIM’s
basic functionality. You’ll find the plugins that shipped with your
copy of VIM in $VIMRUNTIME/plugin.

(Sub-tip: don’t know where VIM is installed? In vim, type “:echo
$VIMRUNTIME“.)

We talked about syntax highlighting in an earlier tip. Syntax
highlighting is great and it makes code more readable, but if you take
your text file that’s all fancy in VIM and publish it on a web page
somewhere, it goes back to dreary black text.

But it doesn’t have to. With a syntax-highlighted file open, type:

ggVG

(that highlights the entire file)

:TOhtml

(that creates an HTML document of your highlighted file)

Cool, huh?

Another fun pair of plugins are gzip.vim and tarPlugin.vim. These are
great because you don’t have to do anything to activate them – just use
VIM to open a .gz or .tar file (or, for that matter, a .tar.gz file).
VIM will uncompress your gzipped file on the fly and re-compress it when
you’re done editing it. If you’re opening up a .tar file, VIM will give
you a list of the files in the tarball.

Happy VIMming!

- David Roth

VIM has a bunch of built-in event handlers which can be used to automate
certain tasks. These are known as autocommands. You can define actions
that get executed whenever one of these autocommands is fired to do
things like read in a template file or write a note to a log file. In
today’s tip, I’m going to show you how to use template files for certain
file types.

Let’s say you write a lot of HTML code, and you always find yourself
creating the following structure:

<HTML>
<HEAD><TITLE></TITLE></HEAD>
<BODY>

</BODY>
</HTML>

Instead of having to type that every single time you create a new .html
file, you can write that structure into a template file. In your home
directory, there should be a .vim directory. If there isn’t go ahead
and create one, then create a directory called templates and write your
template file under the name html.tpl.

Now, add the following line to your .vimrc:

:autocmd BufNewFile *.html 0r ~/.vim/templates/html.tpl

That line tells VIM that you’re defining a new autocommand that is to be
executed when the BufNewFile action is triggered. BufNewFile gets
triggered every time a new file is created in vim. In this case, if the
new file’s name matches the pattern “*.html”, it’s going to read in the
contents of ~/.vim/templates.html.tpl. So now, if you type:

vim brandnewfile.html

VIM is going to create the new file and read in the contents of your
HTML template.

One note about this tip – I have no idea what the 0 in “0r” is for. If
anyone knows why the zero is needed, please let me know.

Happy VIMming!

- David Roth

This is why ninjas are scared of VIM – it allows you to see that which
cannot be seen.  This is a trick that I learned when editing a lot of
Python code in a massive project.  As you may or may not know,
in Python, whitespace is syntactically significant.  Indented lines are
used to indicate the bodies of functions, classes, and loops.  Problems
can arise when bothspaces and tabs are used for indentation, because
Python doesn’t necessarily consider eight spaces to be the equivalent
of two tabs, even though they may look the same to a human eye.

So when you have a file open in VIM and you want to view non-printing
characters, you can turn on list mode to see non-printing characters:

:se list

With list mode enabled, tabs will appear as ^I, carriage returns will
appear as ^M, etc.  To turn list mode off, type:

:se nolist

If you see a character and you’re not sure what it is, you can use
another couple of tricks to view the ASCII value of the character.  Put
your cursor on top of it and type:

ga

(Think “get ASCII”).  If you’re editing a UTF-8 file, the command is:

g8

Then you can look up the value at http://www.asciitable.com/ or
http://www.utf8-chartable.de/ to see what it is. (Unless you’ve memorized the ASCII and UTF-8 tables, in which case you’re probably an emacs user and not interested in these VIM tips anyway.)

Happy VIMming!

- David Roth

Like most modern applications, VIM has an undo/redo feature.  As you’re
typing along, you might realize that you’ve made a mistake and you want
to undo it.  This is easy.  In command mode, type:

u

and VIM will undo your latest changes.  If you decided that you really
did mean to make those changes, type:

<CTRL>+r

or

:redo

And the last change will be restored.

But VIM doesn’t just store a linear stack of changes.  Often, the
changes you make will cause branches in the document’s change history,
and VIM can help you navigate these changes.  If you type:

:undolist

You’ll see a not-very-user-friendly list of information in three
columns.  The first column is the change number, which is the branch’s
ID; the second is the number of changes that occurred in that branch,
and the third is the time the branch was created.

The branch IDs are sequential, so if you remember the order in which
changes were made, you can type:

:undo <ID>

to jump back to a specific branch.  But because the time stamps are
included for each branch, you can also use the commands

:earlier

and

:later

:earlier and :later take an argument of minutes, seconds, or hours.  So
if you made a change ten minutes ago and you want to revert to it, you
can type:

:earlier 10m

And from there,

:later 5m

You can also use the following commands to scroll backwards and forwards
through all change branches:

:g-
:g+

Happy VIMming!

- David Roth

This is one of the most flexible and powerful features of VIM: macros.

On day 8 I mentioned registers. To recap: VIM provides 26 registers
named a-z which can hold arbitrary text. This text can include VIM
commands. When you are in command mode, typing q followed by the name
of a register (a-z) will put you in to macro recording mode. Once this
starts, every keystroke you type will be recorded into the register that
you named.

When working with large blocks of structured text, this is insanely
productive. When you’re done typing the keystrokes that you need, hit
to go back in to command mode and hit q again. Now, all your
keystrokes are in the register and ready for playback.

Type @+ to play back the keystrokes. VIM will play back
exactly the commands you typed in while recording the macro, so if you
need the cursor to drop to the next line, or if you need to search
forward for some text, make sure that you end the macro with the
appropriate keystrokes.

Since the macro is recorded to a register, the same rules of register
use apply – if you want to append keystrokes to an existing macro that
was recorded in register a, you can add more commands by typing qA
(capital names append to registers).

Go crazy, and happy VIMming.

- David Roth