6. Shell basics¶

6.1. Job Control¶

6.2. POSIX¶

Using POSIX compatible syntax allows shell scripts to run on other systems, where no bash(1) is available (NAS, various embedded systems, old systems with proprietary unixes).

So, use #!/bin/sh instead of #!/bin/bash.

Use The Open Group Base Specifications Issue 7, 2018 edition for general reference. See sh - shell, the standard command language interpreter and Shell Command Language for specific shell reference.

program [ { key=[value] | -key | [!]key } ..]

program [key=[value]].. [-key].. [[!]key]..

program [key=[value]] [-key] [[!]key]

  program [OPTIONS] [MODE OPTIONS]

MODE OPTIONS
  --one  perform action 1
  --two  perform action 2

6.3. Special Purpose Language vs. Generic Programming Language¶

The bourne shell sh(1) is a special purpose language. It is not a generic programming language.

Making the shell more like C, with e.g. csh(1), are misguided experiments.

Making the excution of the test program look like a condition in a programming language is a very special brain dead example of syntactic obfuscation.

The standard syntax shows quite clearly, what happens, when the program test is executed:

if test arg1 arg2
then
    :
fi

The alternate program name [ requires an extra argument ] for closing the fake opening bracket, just so the command execution resembles a mathematical condition:

if [ arg1 arg2 ]
then
    :
fi

Avoid arithmetic expansion $(( ... )), if expr(1) can do the job.

printf "variable: %s, arg count: %d, args: %s\n" "${variable}" "${#}" "${*}"

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Mai  8  2018 /bin/sh -> dash

$ /bin/dash -c 'echo "hello\nnext line"'
hello
next line

$ /bin/dash -c 'echo -e "hello\nnext line"'
-e hello
next line

$ /bin/bash -c 'echo "hello\nnext line"'
hello\nnext line

$ /bin/bash -c 'echo -e "hello\nnext line"'
hello
next line

printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "arg_count" "${arg_count}"

function func_name
{
    :
}

func_name ()
{
    :
}

6.4. WRF loop - single line processing in shell¶

Emulating single line processing like sed(1) and awk(1) with read in a while loop. WRF stands historically for while/read/file.

while read -r in_line

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# setup some line records
in_records="
# ::fillme:: this comment is skipped, as are blank lines
some words on

a line
varying count of words
"

printf "%s\n" "${in_records}" \
| (
while read -r in_line
do
    # skip blank lines and comments
    case "${in_line}" in
    ''|"${comm-#}"*) continue;;
    esac

    # print in_line
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "in_line" "${in_line}"
done
)

while read -r in_word0 in_word1 rest

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# setup some line records with
# standard IFS whitespace separator
in_records="
# ::fillme:: this comment is skipped, as are blank lines
some words on

a line
varying count of words
"

printf "%s\n" "${in_records}" \
| (
# use standard IFS to split line
while read -r in_word0 in_word1 rest
do
    # skip blank lines and comments
    case "${in_word0}" in
    ''|"${comm-#}"*) continue;;
    esac

    # print parts
    printf >&2 "# --------------------------------------------------\n"
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "in_word0" "${in_word0}"
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "in_word1" "${in_word1}"
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "rest" "${rest}"
done
)

while IFS=: read -r in_word0 in_word1 rest

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# setup some line records with
# standard IFS whitespace separator
in_records="
# ::fillme:: this comment is skipped, as are blank lines
some words:on

a line
varying:count of:words:and:fields
"

printf "%s\n" "${in_records}" \
| (
# use IFS=: to split line
while IFS=: read -r in_word0 in_word1 rest
do
    # skip blank lines and comments
    case "${in_word0}" in
    ''|"${comm-#}"*) continue;;
    esac

    # print parts
    printf >&2 "# --------------------------------------------------\n"
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "in_word0" "${in_word0}"
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "in_word1" "${in_word1}"
    printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "rest" "${rest}"
done
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

BEGIN {
    if (!line) {
        line = "line";
    }
    if (!varbase) {
        varbase = "col";
    }
    if (!callback) {
         callback = "col_process";
    }
    if (!max_count) {
         max_count = 10;
    }
}
function single_quote_enclose (str) {
    gsub(/'/, "'\\''", str);
    return "'" str "'";
}
{
    printf("%s=%s;\n", line, single_quote_enclose($0));
    printf("%s_count=%d;\n", varbase, NF);
    for (_i=1;_i<=NF;++_i) {
        printf("%s%d=%s;\n", varbase, _i, single_quote_enclose($_i));
    }
    for (_i=NF+1;_i<=max_count;++_i) {
        printf("%s%d=;\n", varbase, _i);
    }
    if (callback) {
        printf("%s;\n\n", callback);
    }
}

1
2
3
4
5

#!   split_and_process_lines [-F " *: *"] [-v line=line] [-v varbase=col] [-v callback=col_process] [-v max_count=10]
split_and_process_lines ()                                                 # ||:fnc:||
{
    ${AWK__PROG-awk} ${1+"$@"} "${AWK_SCRIPT_SPLIT_AND_PROCESS_LINES}"
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

in_records="
# comment
in the : city
and : over : the : mountains

som'e where : over ' the : rainbow
some wh''ere : over the : rainbow
"

col_process ()
{
case "${line}" in
'#'*|'')
    printf >&2 "#  |"":WRN:|  warning: comment or blank line [%s]\n" "${line}"
    return
    ;;
esac

printf "line: col1 [%s] col2 [%s] col3 [%s] col4 [%s] col5 [%s]\n" "${col1}" "${col2}" "${col3}" "${col4}" "${col5}"

_indx=1
while test ${_indx} -le ${col_count}
do
    _var="col${_indx}"
    eval _value=\"\${${_var}}\"

    printf "      %s=%s\n" "${_var}" "${_value}"

    _indx="$( expr ${_indx} + 1 )"
done
}

_script="$(
  printf "%s\n" "${in_records}" \
  | split_and_process_lines -F ' *: *' -v varbase='col' -v callback='col_process'
  )"

printf >&2 "# --------------------------------------------------\n"
printf >&2 "#   "":DBG:   %-${dbg_fwid-15}s: [%s]\n" "_script" "${_script}"

printf >&2 "# --------------------------------------------------\n"
eval "${_script}"

# --------------------------------------------------
#   :DBG:   _script        : [line='';
col_count=0;
col1=;
col2=;
col3=;
col4=;
col5=;
col6=;
col7=;
col8=;
col9=;
col10=;
col_process;

line='# comment';
col_count=1;
col1='# comment';
# ...
col_process;

line='in the : city';
col_count=2;
col1='in the';
col2='city';
# ...
col_process;

line='and : over : the : mountains';
col_count=4;
col1='and';
col2='over';
col3='the';
col4='mountains';
# ...
col_process;

line='';
col_count=0;
# ...
col_process;

line='som'\''e where : over '\'' the : rainbow';
col_count=3;
col1='som'\''e where';
col2='over '\'' the';
col3='rainbow';
# ...
col_process;

line='some wh'\'''\''ere : over the : rainbow';
col_count=3;
col1='some wh'\'''\''ere';
col2='over the';
col3='rainbow';
# ...
col_process;

line='';
col_count=0;
# ...
col_process;]

# --------------------------------------------------
#  |:WRN:|  warning: comment or blank line []
#  |:WRN:|  warning: comment or blank line [# comment]
line: col1 [in the] col2 [city] col3 [] col4 [] col5 []
      col1=in the
      col2=city
line: col1 [and] col2 [over] col3 [the] col4 [mountains] col5 []
      col1=and
      col2=over
      col3=the
      col4=mountains
#  |:WRN:|  warning: comment or blank line []
line: col1 [som'e where] col2 [over ' the] col3 [rainbow] col4 [] col5 []
      col1=som'e where
      col2=over ' the
      col3=rainbow
line: col1 [some wh''ere] col2 [over the] col3 [rainbow] col4 [] col5 []
      col1=some wh''ere
      col2=over the
      col3=rainbow
#  |:WRN:|  warning: comment or blank line []

6.5. Single quoting¶

1. Starting with an unquoted string, assume that it is part of a single quoted string already:

   

2. Find next embedded single quote and insert a single quote before to terminate single quoting ' -> '':

   

3. Insert a backslash to escape the embedded single quote ' -> '\':

   

4. Add a single quote after the escaped single quote to continue quoting ' -> '\'':

   

5. Repeat steps 2 through 4 for all remaining embedded single quotes ' -> '\'':

   

6. Enclose string in single quotes ' … ' to complete single quoting:

   

Activity diagram for algorithm:

6.6. Construct correctly quoted shell script¶

A shell script is assigned to the variable _script to be executed for different purposes, e.g.

• at a later time:

  eval "${_script}"
  

• in different shell process, e.g.:

  sh -c "${_script}"
  printf "%s\n" "${_script}" | sh
  

• as different user:

  sudo -u user sh -c "${_script}"
  

• on remote host:

  ssh user@host "${_script}"
  printf "%s\n" "${_script}" | ssh user@host
  

6.7. Command execution¶

For bash(1), four types of commands are defined:

• aliases
• shell functions
• builtin commands
• external programs

A POSIX shell like dash(1) does not support aliases.

From the man page of bash(1):

COMMAND EXECUTION

After a command has been split into words, if it results in a simple command and an optional list of arguments, the following actions are taken.

If the command name contains no slashes, the shell attempts to locate it. [If the shell is interactive or shell option expand_aliases is set and an alias by that name is found, it is expanded.] If there exists a shell function by that name, that function is invoked as described above in FUNCTIONS. If the name does not match a function, the shell searches for it in the list of shell builtins. If a match is found, that builtin is invoked.

If the name is neither a shell function nor a builtin, and contains no slashes, bash searches each element of the PATH for a directory containing an executable file by that name. Bash uses a hash table to remember the full pathnames of executable files (see hash under SHELL BUILTIN COMMANDS below). A full search of the directories in PATH is performed only if the command is not found in the hash table. If the search is unsuccessful, the shell searches for a defined shell function named command_not_found_handle. If that function exists, it is invoked with the original command and the original command’s arguments as its arguments, and the function’s exit status becomes the exit status of the shell. If that function is not defined, the shell prints an error message and returns an exit status of 127.

If the search is successful, or if the command name contains one or more slashes, the shell executes the named program in a separate execution environment. Argument 0 is set to the name given, and the remaining arguments to the command are set to the arguments given, if any.

If this execution fails because the file is not in executable format, and the file is not a directory, it is assumed to be a shell script, a file containing shell commands. A subshell is spawned to execute it. This subshell reinitializes itself, so that the effect is as if a new shell had been invoked to handle the script, with the exception that the locations of commands remembered by the parent (see hash below under SHELL BUILTIN COMMANDS) are retained by the child.

If the program is a file beginning with #!, the remainder of the first line specifies an interpreter for the program. The shell executes the specified interpreter on operating systems that do not handle this executable format themselves. The arguments to the interpreter consist of a single optional argument following the interpreter name on the first line of the program, followed by the name of the program, followed by the command arguments, if any.

figure 6.4 shows an activity diagram for the command execution process.

6.8. . command¶

The . command is an include mechanism for script files (much like the preprocessor command #include in C). Note, that the standard definition of . ignores all arguments, which means, that no arguments are allowed to avoid inconsistent behavior for different shells..

All variable assignments in the included file are incorporated into the shell environment.

From man page of bash(1):

. filename […] Read and execute commands from filename in the

current shell environment and return the exit status of the last command executed from filename. If filename does not contain a slash, filenames in PATH are used to find the directory containing filename. The file searched for in PATH need not be executable. When bash is not in posix mode, the current directory is searched if no file is found in PATH. If the sourcepath option to the shopt builtin command is turned off, the PATH is not searched. […] The return status is the status of the last command exited within the script (0 if no commands are executed), and false if filename is not found or cannot be read.

6.9. Subshell and compound commands¶

From man page of bash(1):

Compound Commands

A compound command is one of the following. In most cases a list in a command’s description may be separated from the rest of the command by one or more newlines, and may be followed by a newline in place of a semicolon.

(list)

list is executed in a subshell environment (see COMMAND EXECUTION ENVIRONMENT below). Variable assignments and builtin commands that affect the shell’s environment do not remain in effect after the command completes. The return status is the exit status of list.

{ list; }

list is simply executed in the current shell environment. list must be terminated with a newline or semicolon. This is known as a group command. The return status is the exit status of list. Note that unlike the metacharacters ( and ), { and } are reserved words and must occur where a reserved word is permitted to be recognized. Since they do not cause a word break, they must be separated from list by whitespace or another shell metacharacter.

Builtin commands in a subshell do not affect the shell environment in the parent shell, e.g.:

VAR='value'
( VAR='something'; echo "${VAR}"; )
echo "${VAR}";

results in output of:

something
value

Builtin commands in a group command do affect the shell environment outside the group, e.g.:

VAR='value'
{ VAR='something'; echo "${VAR}"; }
echo "${VAR}";

results in output of:

something
something

var=outer
echo world | { var=inner; echo hello; cat - }
echo "${var}"

var=outer
echo world | ( var=inner; echo hello; cat - )
echo "${var}"

unexported='internal value'
export exported='external value'
( echo "[${unexported}]"; echo "[${exported}]" )

[internal value]
[external value]

unexported='internal value'
export exported='external value'
cat <<'EOF' | sh
echo "[${unexported}]"; echo "[${exported}]"
EOF

[]
[external value]