Skip to content

So you want to write a Shell script

So some people, companies even, have guidelines that describe how to write shell scripts, or even unit tests for shell scripts, as if “UNIX Shell” was a programming language. That’s wrong.

“Modern Shells” are based on a language that has been written without a formal language specification. The source looked like this, because somebody didn’t like C and wanted Algol, abusing the preprocessor. The original functionality and language rules had to be reverse engineered from that source, and original shell has a lot of weird rules and quirks:

  • You can use the caret, ‘^’, as replacement for the pipe symbol, ‘|’.
  • Check out the section »Consider a variable which has been picked up by the shell from the environment at startup. Modifying this variable creates a local copy.« in that document, especially the part where they explain this:
    If you call a script directly from a bourne shell (“./script” without shebang),  then the shell only forks off a subhell and reads in the script.
    The split between original and local copy of the variable is still present in the subshell.But if the script is a real executable with #! magic, or if another sh is called, then fork and exec is used and only the original unmodified variable will be visible.

And it gets better if you go down the entirety of that particular document.

If you think Unix Shell is a survivable programming environment, good luck, and please take your code with you while you leave.

Published inComputer ScienceErklärbär

10 Comments

  1. AndreasLobinger

    So we’re preparing for “go away or I will replace you with a simple script” ?

  2. Sven Dirks

    As with everything else it depends what you want to use such scripts for. I tend to write short scripts of several lines only in order to automate simple, repetitive tasks. For example setting a bunch of variables on a number of ESXI-hosts in a new cluster. The second benefit of this being that I don’t have to memorize oodles of options for certain commands. I “man” them once and put them in the script, so next time I just need to take a peek at the script in order to know what I will be doing.

    Being an engineer but no programmer this still allows me to speed up my work considerably.

    • AndreasLobinger

      Being an engineer, too, my assumption is, that yes, you speed up your work working with something that is ‘just’ available, while at the same time you slow down maintenance by someone else who is not aware of all subtle points of your specific shell.

    • Andre

      It’s basically a problem of “when to get off”..
      While it’s quite ok to write a couple of lines of shellscript to write less boilerplate code, as soon as you’re trying to parse some data file or need access to more complex data structures like nested lists and/or hashes or you’re going to pass variables through multiple programs/ssh sessions/scripts, you will want a “proper” programming language.
      It can be done in (a “modern”) shell, but it’s usually pretty messy in regard to readability and sooner or later, you’ll be in quotation hell.

      And if you’re working in a bigger environment, that “This needs to stop!” decision often isn’t yours alone and refactoring scripts that are already pretty crummy takes time that nobody has.. Hence: Use a better tool to begin with.

      Addendum: Do not use a new language for every new problem.. Agree on a common platform, because every single one is going to suck at something, but switching hundreds of times between nodejs, python, ruby, bash and a couple of other languages per day sucks even more.

      • Sven Dirks

        I agree, standardization ist the way to go. However, I get a bunch (like, 30, 40, 50) servers, having the basic setup of our internal IT-gurus. The customer requires specific settings and I have a very long list of “to dos” to cross off one after the other.

        This is where my own little scripts (no one else wants or gets them ;-) ) come in extremely handy. I just change the site specific parameters, fire off the script, check if everything is set up the way it should be by using the official tool (Zabbix) for that, saving me hundreds or thousands of mouse clicks while being less error prone at the same time.

        The scripts do nothing else but issue a handful of power cli commands to esxi or some settings to the literally hundreds of embedded linux devices we use.

        If the management decides to go with a different sort of tools and provides both the tools and the time to get acquainted with them I’d be happy to use them. :-)

  3. Jörg Baumgartner

    “then the shell only forks off a subhell ” – sounds like infernal programming.

  4. Hartmut Holzgraefe

    ““Modern Shells” are based on a language that has been written without a formal language specification.”

    Gets even worse when you enter autoconf/automake land: now you’re stuck with three languages without formal specification being intermixed (sh, m4, make … extra pain points for having to avoid bashisms).

    But then CMake replaces two of these “languages” with known quirks (sh, m4) with its own custom language that’s not used anywhere else, and *still* doesn’t have any real formal specification.

    At first it looks as if it is at least based on a machine readable flex/bison grammer that you could eventually extract a formal language specification from, but at a closer look that “grammer” only describes how function definitions and function arguments are parsed while all higher (or lower?) level constructs like control structures are are implemented in hand-written parsing code, hiding somewhere else in the code base …

    So we still can’t have nice things … (and don’t even get me started on “Why CMake doesn’t generate a ‘make uninstall’ target
    by default)

  5. Devdas Bhagat

    From IRC a few minutes ago, handles obfuscated:
    bofh1: Right now, python and perl and the mess of conflicting versions and code that works in one and not another is making those not my friend.
    “Oh, this group of systems has no version of python installed, and uses perl 5.8.8.”
    and that’s what C is for :)
    “Oh, this box has Python 3.3, but these others only get Python 2.5”
    bofh1: Then I have to find a compiler on each OS and architecture …
    bofh1, then you get to deal with systems with no C compiler installed
    (ohai solaris)
    bofh2: Or HP-UX…
    bofh2: Or worse, “this box’s C compiler is from the 1990s, so your nice pretty C code won’t compile, uses features that weren’t added to C yet”
    heheh
    There’s a reason I’ve been writing a lot of POSIX awk and POSIX sh. They’re reliably available.
    this is true, but at some point you have to decide whether it’s easier/more maintainable to do something in shell, or a more advanced language
    bofh1: What is it that you think makes shell “unmaintainable”? The fact that it isn’t $popular_language_du_jour?
    Bad code is bad code, and good code is good code, almost regardless of language
    shell, for instance, doesn’t have named file handles, or lexically-scoped file descriptors
    probably that most people don’t understand advanced shell scripting these days — at least in part because bash’s proliferation of unfortunate features has given
    advanced shell scripting a bad name

    • kris kris

      Yes. That styleguide is literally the first link in the article above.

Leave a Reply

Your email address will not be published. Required fields are marked *