.. -*- coding: utf-8 -*-
.. role:: sref(numref)
.. role:: xref(numref)
.. Copyright (C) 2019, Wolfgang Scherer, <Wolfgang.Scherer at gmx.de>
..
.. This file is part of Development.
..
.. Permission is granted to copy, distribute and/or modify this document
.. under the terms of the GNU Free Documentation License, Version 1.3
.. or any later version published by the Free Software Foundation;
.. with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
.. A copy of the license is included in the section entitled "GNU
.. Free Documentation License".
.. inline comments (with du_comment_role)
.. role:: rem(comment)
.. role:: html(raw)
:format: html
.. role:: shx(code)
:language: sh
.. highlight:: sh
.. rst-class:: narrow xmedium xlarge xhuge xultra
##################################################
:rem:`|||:sec:|||`\ Shell basics
##################################################
.. >>CODD See `the components of a doctoral dissertation and their order <http://site.uit.no/english/writing-style/bookstructure/>`_
.. >>CODD Dedication
.. >>CODD Epigraph
.. >>CODD Abstract
.. compound::
.. \|:here:|
.. >>CODD Introduction
.. >>CODD Chapter
==================================================
:rem:`|||:sec:|||`\ Job Control
==================================================
+----------------------+----------------------------------------------------------------------------------------------------------------------+
| Command | Description |
+======================+======================================================================================================================+
| :samp:`jobs` | show jobs (background tasks) |
+----------------------+----------------------------------------------------------------------------------------------------------------------+
| :kbd:`C-z` | send signal TSTOP to foreground job, which stops, is sent to the background job queue and becomes the current job |
+----------------------+----------------------------------------------------------------------------------------------------------------------+
| :samp:`bg [job no.]` | run current job in job queue in background |
+----------------------+----------------------------------------------------------------------------------------------------------------------+
| :samp:`fg [job no.]` | run current job in foreground |
+----------------------+----------------------------------------------------------------------------------------------------------------------+
| :samp:`cmd &` | run cmd as background job |
+----------------------+----------------------------------------------------------------------------------------------------------------------+
==================================================
:rem:`|||:sec:|||`\ POSIX
==================================================
Using POSIX compatible syntax allows shell scripts to run on other
systems, where no bash(1) is available (NAS, various embedded systems,
old systems with proprietary unixes).
So, use `#!/bin/sh` instead of `#!/bin/bash`.
Use `The Open Group Base Specifications Issue 7, 2018 edition`_ for
general reference. See `sh - shell, the standard command language
interpreter`_ and `Shell Command Language`_ for specific shell
reference.
.. attention:: Be aware that there are very old shells out there that
do not conform to POSIX. Mainly because POSIX was not around at
their conception. So not everything allowed by POSIX is necessarily
failsafe for all shells.
-----------------------------------------------------
:rem:`||:sec:||`\ Conventions for Syntax Descriptions
-----------------------------------------------------
The conventions for syntax descriptions are covered in
`Chapter 12. Utility Conventions`_ of `The Open Group Base
Specifications Issue 7, 2018 edition`_.
An informal description can also be found in man-pages(7).
The syntax for alternative arguments ``{ arg1 | arg2 | arg3 }`` is not
part of POSIX, but is mentioned in `syntax - Is there a specification
for a man page's SYNOPSIS section? - Stack Overflow`_
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:rem:`|:sec:|`\ Specific Conventions in Templated Shell Scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Single letter options preceded by a single dash ``-`` must not be
grouped together. The additional effort is not worth the conceived
advantage. Readability is much better, when separating single letter
options.
In addition to POSIX chapter 12, a shortened ellipses ``..`` may be
used in place of a full ellipses ``...``.
Clarifying POSIX section 12.8, the vertical bar ``|`` is only used
within braces ``{``, ``}`` to indicate exclusive
alternatives. Together with brackets ``[``, ``]`` This allows
specifying optional syntax variants without an implied order. E.g.:
.. code-block:: sh
program [ { key=[value] | -key | [!]key } ..]
This notation emphasizes that each of these options overrides the
effect of a previous occurence. The following syntax description is
equivalent, but the exclusive nature of the alternatives is not so
clear:
.. code-block:: sh
program [key=[value]].. [-key].. [[!]key]..
This example can be further shortened, if the option description
mentions, that the option can be specified multiple times:
.. code-block:: sh
program [key=[value]] [-key] [[!]key]
In addtion to specifying multiple synopsis lines according to POSIX
section 12.8, mutually exclusive option may be given summarily as
[MODE OPTIONS], which are understood to override any previous mode
options. E.g.:
.. code-block:: sh
program [OPTIONS] [MODE OPTIONS]
MODE OPTIONS
--one perform action 1
--two perform action 2
.. _`syntax - Is there a specification for a man page's SYNOPSIS section? - Stack Overflow`: https://stackoverflow.com/a/8716112/2127439
.. _`Shell Command Language`: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
.. _`sh - shell, the standard command language interpreter`: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html
.. _`The Open Group Base Specifications Issue 7, 2018 edition`: https://pubs.opengroup.org/onlinepubs/9699919799/
.. _`Chapter 12. Utility Conventions`: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html
=============================================================================
:rem:`|||:sec:|||`\ Special Purpose Language vs. Generic Programming Language
=============================================================================
The bourne shell sh(1) is a special purpose language. It is **not** a
generic programming language.
Making the shell more like C, with e.g. csh(1), are misguided
experiments.
Making the excution of the :program:`test` program look like a
condition in a programming language is a very special brain dead
example of syntactic obfuscation.
The standard syntax shows quite clearly, what happens, when the
program :program:`test` is executed:
.. code-block:: sh
if test arg1 arg2
then
:
fi
The alternate program name :program:`[` requires an extra argument `]`
for *closing* the fake opening bracket, just so the command execution
resembles a *mathematical* condition:
.. code-block:: sh
if [ arg1 arg2 ]
then
:
fi
.. warning:: Using this abomination in a shell scripts results in
immediate deletion.
Avoid arithmetic expansion :samp:`$(( ... ))`, if :program:`expr`\ (1)
can do the job.
--------------------------------------------------
:rem:`||:sec:||`\ Variable expansion
--------------------------------------------------
To be safe and to make replacemnts simpler, always use curly braces
for variable expansion:
.. code-block:: sh
printf "variable: %s, arg count: %d, args: %s\n" "${variable}" "${#}" "${*}"
Emacs support in Shell-script mode:
+--------------+-----------+
| Shortcut | Expansion |
+==============+===========+
| :kbd:`C-c v` | ${} |
+--------------+-----------+
| :kbd:`C-c q` | "${}" |
+--------------+-----------+
--------------------------------------------------
:rem:`||:sec:||`\ echo (1) , printf(1)
--------------------------------------------------
Do not use echo(1), since it is not portable, use printf(1)
instead.
Emacs support in Shell-script mode:
+------------------+--------------------+
| Shortcut | Expansion |
+==================+====================+
| :kbd:`C-c p` | printf "%s\n" |
+------------------+--------------------+
| :kbd:`C-u C-c p` | printf >&2 "%s\n" |
+------------------+--------------------+
Especially dash(1) (ubuntu system shell) and bash(1) differ extremely.
.. code-block:: sh
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Mai 8 2018 /bin/sh -> dash
$ /bin/dash -c 'echo "hello\nnext line"'
hello
next line
$ /bin/dash -c 'echo -e "hello\nnext line"'
-e hello
next line
$ /bin/bash -c 'echo "hello\nnext line"'
hello\nnext line
$ /bin/bash -c 'echo -e "hello\nnext line"'
hello
next line
Emacs support in Shell-script mode for debug output of variables:
:kbd:`arg_count C-c d v v`
expands to
.. code-block:: sh
printf >&2 "# "":DBG: %-${dbg_fwid-15}s: [%s]\n" "arg_count" "${arg_count}"
--------------------------------------------------
:rem:`||:sec:||`\ Avoid special bash syntax
--------------------------------------------------
Do not use:
.. code-block:: sh
function func_name
{
:
}
but use POSIX compatible syntax instead:
.. code-block:: sh
func_name ()
{
:
}
--------------------------------------------------
:rem:`||:sec:||`\ Do not use arrays
--------------------------------------------------
Shell arrays are not POSIX compatible! If you think you need to use
arrays, you should probably not use the shell but a generic script
programming language like awk(1), perl(1) or python(1).
See :sref:`sec:WRF loop - single line processing in shell` for single
line processing with splitting into fields.
See also `bash - How to mark an array in POSIX sh? - Stack Overflow`_
See also `GitHub - krebs/array: a POSIX-compliant implementation of
arrays`_, for a POSIX compliant implementation of arrays (untested).
.. _`bash - How to mark an array in POSIX sh? - Stack Overflow`: https://stackoverflow.com/questions/6499486/how-to-mark-an-array-in-posix-sh
.. _`GitHub - krebs/array: a POSIX-compliant implementation of arrays`: https://github.com/krebs/array
.. _`sec:WRF loop - single line processing in shell`:
==============================================================
:rem:`|||:sec:|||`\ WRF loop - single line processing in shell
==============================================================
Emulating single line processing like sed(1) and awk(1) with
:command:`read` in a :command:`while` loop. WRF stands historically
for while/read/file.
--------------------------------------------------
:rem:`||:sec:||`\ WRF loop
--------------------------------------------------
A file is parsed as single lines with the :command:`read` command (see
:xref:`lst:WRF loop`, line 14):
.. code-block:: sh
while read -r in_line
See :xref:`fig:WRF loop` for activity diagram.
.. _`fig:WRF loop`:
.. uml:: _static/wrf_loop-a0.puml
:caption: WRF loop
.. _`lst:WRF loop`:
.. literalinclude:: _static/wrf_loop_x_wrf_loop.sh
:caption: WRF loop
:emphasize-lines: 12
:language: sh
:linenos:
--------------------------------------------------
:rem:`||:sec:||`\ WRF loop with standard IFS split
--------------------------------------------------
Instead of reading an entire line, the :command:`read` command parses
the line into several variables (see :xref:`lst:WRF loop with standard IFS
split`, line 14):
.. code-block:: sh
while read -r in_word0 in_word1 rest
The standard IFS is used which splits the line on whitespace.
See :xref:`fig:WRF loop with standard IFS split` for activity diagram.
.. _`fig:WRF loop with standard IFS split`:
.. uml:: _static/wrf_loop-a1.puml
:caption: WRF loop with standard IFS split
.. _`lst:WRF loop with standard IFS split`:
.. literalinclude:: _static/wrf_loop_x_wrf_loop_with_standard_ifs_split.sh
:caption: WRF loop with standard IFS split
:emphasize-lines: 14
:language: sh
:linenos:
--------------------------------------------------
:rem:`||:sec:||`\ WRF loop with special IFS split
--------------------------------------------------
Instead of reading an entire line, the :command:`read` command parses
the line into several variables (see :xref:`lst:WRF loop with special IFS
split`, line 14):
.. code-block:: sh
while IFS=: read -r in_word0 in_word1 rest
IFS is set to ``:`` for the :command:`read` command only, which splits
the line on a ``:`` character.
See :xref:`fig:WRF loop with special IFS split` for activity diagram.
.. _`fig:WRF loop with special IFS split`:
.. uml:: _static/wrf_loop-a2.puml
:caption: WRF loop with special IFS split
.. _`lst:WRF loop with special IFS split`:
.. literalinclude:: _static/wrf_loop_x_wrf_loop_with_special_ifs_split.sh
:caption: WRF loop with special IFS split
:emphasize-lines: 14
:language: sh
:linenos:
-----------------------------------------------------
:rem:`||:sec:||`\ split and process lines with awk(1)
-----------------------------------------------------
.. literalinclude:: _static/wrf_loop.sh
:caption: AWK script to split and process lines via callback
:start-after: .:lst:. awk_script_split_and_process_lines
:end-before: .:lst:. awk_script_split_and_process_lines -
:language: awk
:linenos:
.. literalinclude:: _static/wrf_loop.sh
:caption: Function split and process lines via callback
:start-after: .:lst:. split_and_process_lines
:end-before: .:lst:. split_and_process_lines -
:language: sh
:linenos:
.. literalinclude:: _static/wrf_loop.sh
:caption: Example for split and process lines via callback
:start-after: .:lst:. split_and_process_lines_example
:end-before: .:lst:. split_and_process_lines_example -
:language: sh
:linenos:
.. code-block:: text
:caption: Script generated by example for split and process lines via callback
# --------------------------------------------------
# :DBG: _script : [line='';
col_count=0;
col1=;
col2=;
col3=;
col4=;
col5=;
col6=;
col7=;
col8=;
col9=;
col10=;
col_process;
line='# comment';
col_count=1;
col1='# comment';
# ...
col_process;
line='in the : city';
col_count=2;
col1='in the';
col2='city';
# ...
col_process;
line='and : over : the : mountains';
col_count=4;
col1='and';
col2='over';
col3='the';
col4='mountains';
# ...
col_process;
line='';
col_count=0;
# ...
col_process;
line='som'\''e where : over '\'' the : rainbow';
col_count=3;
col1='som'\''e where';
col2='over '\'' the';
col3='rainbow';
# ...
col_process;
line='some wh'\'''\''ere : over the : rainbow';
col_count=3;
col1='some wh'\'''\''ere';
col2='over the';
col3='rainbow';
# ...
col_process;
line='';
col_count=0;
# ...
col_process;]
.. code-block:: text
:caption: Output from example for split and process lines via callback
# --------------------------------------------------
# |:WRN:| warning: comment or blank line []
# |:WRN:| warning: comment or blank line [# comment]
line: col1 [in the] col2 [city] col3 [] col4 [] col5 []
col1=in the
col2=city
line: col1 [and] col2 [over] col3 [the] col4 [mountains] col5 []
col1=and
col2=over
col3=the
col4=mountains
# |:WRN:| warning: comment or blank line []
line: col1 [som'e where] col2 [over ' the] col3 [rainbow] col4 [] col5 []
col1=som'e where
col2=over ' the
col3=rainbow
line: col1 [some wh''ere] col2 [over the] col3 [rainbow] col4 [] col5 []
col1=some wh''ere
col2=over the
col3=rainbow
# |:WRN:| warning: comment or blank line []
==================================================
:rem:`|||:sec:|||`\ Single quoting
==================================================
#. Starting with an unquoted string, assume that it is part of a
single quoted string already:
.. uml::
:html_format: png
:latex_format: png
:scale: 100%
@startditaa
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| | | | | | | | | | | | | | | | | | | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| e | m | b | e | d | ' | e | d | ' | s | i | n | g | ' | l | e | | | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
@endditaa
#. Find next embedded single quote and insert a single quote before to
terminate single quoting ``'`` -> ``''``:
.. uml::
:html_format: png
:latex_format: png
:scale: 100%
@startditaa
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| | | | | | ↓ | | | | | | | | | | | | | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| e | m | b | e | d | ' | ' | e | d | ' | s | i | n | g | ' | l | e | | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
@endditaa
#. Insert a backslash to escape the embedded single quote ``'`` -> ``'\'``:
.. uml::
:html_format: png
:latex_format: png
:scale: 100%
@startditaa
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| | | | | | | ↓ | | | | | | | | | | | | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| e | m | b | e | d | ' | \ | ' | e | d | ' | s | i | n | g | ' | l | e | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
@endditaa
#. Add a single quote after the escaped single quote to continue
quoting ``'`` -> ``'\''``:
.. uml::
:html_format: png
:latex_format: png
:scale: 100%
@startditaa
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| | | | | | | | | ↓ | | | | | | | | | | | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| e | m | b | e | d | ' | \ | ' | ' | e | d | ' | s | i | n | g | ' | l | e | | | | | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
@endditaa
#. Repeat steps 2 through 4 for all remaining embedded single quotes ``'`` -> ``'\''``:
.. uml::
:html_format: png
:latex_format: png
:scale: 100%
@startditaa
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| | | | | | | | | | | | ↓ | ↓ | | ↓ | | | | | ↓ | ↓ | | ↓ | | | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| e | m | b | e | d | ' | \ | ' | ' | e | d | ' | \ | ' | ' | s | i | n | g | ' | \ | ' | ' | l | e | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
@endditaa
#. Enclose string in single quotes ``'`` ... ``'`` to complete single quoting:
.. uml::
:html_format: png
:latex_format: png
:scale: 100%
@startditaa
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| ↓ | | | | | | | | | | | | | | | | | | | | | | | | | | ↓ |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| ' | e | m | b | e | d | ' | \ | ' | ' | e | d | ' | \ | ' | ' | s | i | n | g | ' | \ | ' | ' | l | e | ' |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
@endditaa
Activity diagram for algorithm:
.. uml::
@startuml
start
while (for each single quote) is (do)
:replace single quote ""'""
with ""'\''"";
endwhile
:enclose escaped string
in single quotes ""'"" ... ""'"";
stop
@enduml
=======================================================================
:rem:`|||:sec:|||`\ Construct correctly quoted shell script
=======================================================================
A shell script is assigned to the variable `_script` to be executed
for different purposes, e.g.
- at a later time:
.. code-block:: sh
eval "${_script}"
- in different shell process, e.g.:
.. code-block:: sh
sh -c "${_script}"
printf "%s\n" "${_script}" | sh
- as different user:
.. code-block:: sh
sudo -u user sh -c "${_script}"
- on remote host:
.. code-block:: sh
ssh user@host "${_script}"
printf "%s\n" "${_script}" | ssh user@host
--------------------------------------------------
:rem:`||:sec:||`\ Preparations
--------------------------------------------------
#. Update snippets to latest version:
.. code-block:: sh
cd /srv/ftp/pub && ./sync.sh --restore && ./xx-sync-ftp-pub.sh -l 0
#. Create test shell script with template::
snn x_quoted_script.sh
#. Expand snippet (at end of line (:kbd:`C-e`) enter key sequence
:kbd:`C-x C-e`)::
## (progn (forward-line 1) (snip-insert "sh_f.single-quote" t t "sh" " --key single_quote_minimal") (insert "\n"))
#. Add example environment setup in body::
set -- arg1 arg2 'arg with spaces'
TEMP_DIR='/tmp/some-rndajom-stuff'
#. Add example command::
( cd "${TEMP_DIR}/" || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "${_arg}"; done
echo 'hello' | cat -
#. Execute and study output::
x_quoted_script.sh: 2: cd: cannot cd to /tmp/some-rndajom-stuff/
arg1
arg2
arg with spaces
hello
--------------------------------------------------
:rem:`||:sec:||`\ Single quoted string
--------------------------------------------------
#. Single quote entire command::
_script='
( cd "${TEMP_DIR}/" || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "${_arg}"; done
echo '\''hello'\'' | cat -
'
add some execution tests::
printf "%s\n" "${_script}"
printf "%s\n" '--------------------------------------------------'
eval "${_script}"
printf "%s\n" '--------------------------------------------------'
sh -c "${_script}"
and observe output::
( cd "${TEMP_DIR}/" || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "${_arg}"; done
echo 'hello' | cat -
--------------------------------------------------
x_quoted_script.sh: 2: cd: cannot cd to /tmp/some-rndajom-stuff/
arg1
arg2
arg with spaces
hello
--------------------------------------------------
/
hello
#. Interrupt quoting to insert expanded variables.
#. Use :func:`single_quote_enclose` as necessary::
_script='
( cd '"$( single_quote_enclose "${TEMP_DIR}/" )"' || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "${_arg}"; done
echo '\''hello'\'' | cat
'
#. Use :func:`single_quote_args` as necessary::
_script='
( cd '"$( single_quote_enclose "${TEMP_DIR}/" )"' || exit 1; pwd )
for _arg in '"$( single_quote_args ${1+"${@}"} )"'; do echo "${_arg}"; done
echo '\''hello'\'' | cat -
'
and observe output::
( cd '/tmp/some-rndajom-stuff/' || exit 1; pwd )
for _arg in 'arg1' 'arg2' 'arg with spaces'; do echo "${_arg}"; done
echo 'hello' | cat -
--------------------------------------------------
x_quoted_script.sh: 2: cd: cannot cd to /tmp/some-rndajom-stuff/
arg1
arg2
arg with spaces
hello
--------------------------------------------------
sh: 2: cd: cannot cd to /tmp/some-rndajom-stuff/
arg1
arg2
arg with spaces
hello
--------------------------------------------------
:rem:`||:sec:||`\ HERE document
--------------------------------------------------
#. Enclose entire command in ``cat <<EOF`` ... ``EOF``, escape as
necessary::
cat <<EOF
( cd "${TEMP_DIR}/" || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "\${_arg}"; done
echo 'hello' | cat -
EOF
and observe output::
( cd "/tmp/some-rndajom-stuff/" || exit 1; pwd )
for _arg in arg1 arg2 arg with spaces; do echo "${_arg}"; done
echo 'hello' | cat -
#. Use :func:`single_quote_enclose` and :func:`single_quote_args` as
necessary::
cat <<EOF
( cd $( single_quote_enclose "${TEMP_DIR}/" ) || exit 1; pwd )
for _arg in $( single_quote_args ${1+"${@}"} ); do echo "\${_arg}"; done
echo 'hello' | cat -
EOF
and observe output::
( cd '/tmp/some-rndajom-stuff/' || exit 1; pwd )
for _arg in 'arg1' 'arg2' 'arg with spaces'; do echo "${_arg}"; done
echo 'hello' | cat -
#. Enclose in subshell expansion ``"$(`` ... ``)"`` for assignment to
variable::
_script="$(
cat <<EOF
( cd $( single_quote_enclose "${TEMP_DIR}/" ) || exit 1; pwd )
for _arg in $( single_quote_args ${1+"${@}"} ); do echo "\${_arg}"; done
echo 'hello' | cat -
EOF
)"
add some execution tests::
printf "%s\n" "${_script}"
printf "%s\n" '--------------------------------------------------'
eval "${_script}"
printf "%s\n" '--------------------------------------------------'
sh -c "${_script}"
and observe output::
( cd '/tmp/some-rndajom-stuff/' || exit 1; pwd )
for _arg in 'arg1' 'arg2' 'arg with spaces'; do echo "${_arg}"; done
echo 'hello' | cat -
--------------------------------------------------
x_quoted_script.sh: 1: cd: cannot cd to /tmp/some-rndajom-stuff/
arg1
arg2
arg with spaces
hello
--------------------------------------------------
sh: 1: cd: cannot cd to /tmp/some-rndajom-stuff/
arg1
arg2
arg with spaces
hello
#. Enclose entire command in here document specifying a quoted
end-of-file marker ``cat <<'EOF'`` ... ``EOF``, no escaping is
necessary:
.. code-block:: sh
cat <<'EOF'
( cd "${TEMP_DIR}/" || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "\${_arg}"; done
echo 'hello' | cat -
EOF
and observe output:
.. code-block:: text
( cd "${TEMP_DIR}/" || exit 1; pwd )
for _arg in ${1+"${@}"}; do echo "\${_arg}"; done
echo 'hello' | cat -
The type of quotes (single or double) does not matter.
==================================================
:rem:`|||:sec:|||`\ Command execution
==================================================
For bash(1), four types of commands are defined:
- aliases
- shell functions
- builtin commands
- external programs
A POSIX shell like dash(1) does not support aliases.
From the man page of bash(1):
**COMMAND EXECUTION**
After a command has been split into words, if it results in a
simple command and an optional list of arguments, the following
actions are taken.
If the command name contains no slashes, the shell attempts to
locate it. [If the shell is interactive or shell option
expand_aliases is set and an alias by that name is found, it is
expanded.] If there exists a shell function by that name, that
function is invoked as described above in FUNCTIONS. If the name
does not match a function, the shell searches for it in the list
of shell builtins. If a match is found, that builtin is invoked.
If the name is neither a shell function nor a builtin, and
contains no slashes, bash searches each element of the PATH for a
directory containing an executable file by that name. Bash uses a
hash table to remember the full pathnames of executable files (see
hash under SHELL BUILTIN COMMANDS below). A full search of the
directories in PATH is performed only if the command is not found
in the hash table. If the search is unsuccessful, the shell
searches for a defined shell function named
command_not_found_handle. If that function exists, it is invoked
with the original command and the original command's arguments as
its arguments, and the function's exit status becomes the exit
status of the shell. If that function is not defined, the shell
prints an error message and returns an exit status of 127.
If the search is successful, or if the command name contains one
or more slashes, the shell executes the named program in a
separate execution environment. Argument 0 is set to the name
given, and the remaining arguments to the command are set to the
arguments given, if any.
If this execution fails because the file is not in executable
format, and the file is not a directory, it is assumed to be a
shell script, a file containing shell commands. A subshell is
spawned to execute it. This subshell reinitializes itself, so
that the effect is as if a new shell had been invoked to handle
the script, with the exception that the locations of commands
remembered by the parent (see hash below under SHELL BUILTIN
COMMANDS) are retained by the child.
If the program is a file beginning with #!, the remainder of the
first line specifies an interpreter for the program. The shell
executes the specified interpreter on operating systems that do
not handle this executable format themselves. The arguments to
the interpreter consist of a single optional argument following
the interpreter name on the first line of the program, followed by
the name of the program, followed by the command arguments, if
any.
.. note:: **DO NOT** set the shell option expand_aliases in
scripts. Generally, **DO NOT** write bash(1) scripts. Stick
to **POSIX**.
:xref:`fig:Shell command execution process` shows an activity
diagram for the command execution process.
.. _`fig:Shell command execution process`:
.. uml::
:caption: Shell command execution process
@startuml
partition "Shell command execution process" {
start
if (command is a simple command (no slashes)?) then (yes)
if ((shell is interactive or shell\n option expand_aliases is set)\nand an alias by this name is defined?) then (yes)
:expand alias;
elseif (a //shell function//\nby this name exists?) then (yes)
:execute //shell function//;
elseif (a //builtin command//\nby this name exist?) then (yes)
:execute //builtin command//;
else
while (for path_element in PATH) is (do)
if (//path_element/command// exists?) then (yes)
:execute //path_element/command//;
break
endif
endwhile
endif
elseif (//command// file exists?) then (yes)
:execute //command// file;
else
:error;
end
endif
stop
}
@enduml
==================================================
:rem:`|||:sec:|||`\ **.** command
==================================================
The **.** command is an include mechanism for script files (much like
the preprocessor command `#include` in C). Note, that the standard
definition of **.** ignores all arguments, which means, that no
arguments are allowed to avoid inconsistent behavior for different
shells..
All variable assignments in the included file are incorporated into
the shell environment.
From man page of bash(1):
. filename [...] Read and execute commands from filename in the
current shell environment and return the exit status of the last
command executed from filename. If filename does not contain a
slash, filenames in PATH are used to find the directory
containing filename. The file searched for in PATH need not be
executable. When bash is not in posix mode, the current
directory is searched if no file is found in PATH. If the
sourcepath option to the shopt builtin command is turned off,
the PATH is not searched. [...] The return status is the status
of the last command exited within the script (0 if no commands
are executed), and false if filename is not found or cannot be
read.
==================================================
:rem:`|||:sec:|||`\ Subshell and compound commands
==================================================
From man page of bash(1):
Compound Commands
A compound command is one of the following. In most cases a
list in a command's description may be separated from the rest
of the command by one or more newlines, and may be followed by
a newline in place of a semicolon.
(list)
list is executed in a subshell environment (see COMMAND
EXECUTION ENVIRONMENT below). Variable assignments and
builtin commands that affect the shell's environment do not
remain in effect after the command completes. The return
status is the exit status of list.
{ list; }
list is simply executed in the current shell environment.
list must be terminated with a newline or semicolon. This is
known as a group command. The return status is the exit
status of list. Note that unlike the metacharacters ( and ),
{ and } are reserved words and must occur where a reserved
word is permitted to be recognized. Since they do not cause
a word break, they must be separated from list by whitespace
or another shell metacharacter.
Builtin commands in a subshell do not affect the shell environment in
the parent shell, e.g.:
.. code-block:: sh
VAR='value'
( VAR='something'; echo "${VAR}"; )
echo "${VAR}";
results in output of:
.. code-block:: text
something
value
Builtin commands in a group command do affect the shell environment outside the group, e.g.:
.. code-block:: sh
VAR='value'
{ VAR='something'; echo "${VAR}"; }
echo "${VAR}";
results in output of:
.. code-block:: text
something
something
.. note::
A command in a pipeline is implicitely executed in a subshell.
I.e.:
.. code-block:: sh
var=outer
echo world | { var=inner; echo hello; cat - }
echo "${var}"
is equivalent to:
.. code-block:: sh
var=outer
echo world | ( var=inner; echo hello; cat - )
echo "${var}"
.. note::
Generally avoid :samp:`{{ list }}` grouping. Especially the side effect
of shell environment manipulation.
.. note::
A subshell is not equivalent to execution of an external shell script.
A subshell can access all variables of the parent shell
environment, whether they are exported or not. E.g.:
.. code-block:: sh
unexported='internal value'
export exported='external value'
( echo "[${unexported}]"; echo "[${exported}]" )
results in output
.. code-block:: text
[internal value]
[external value]
Whereas an external shell script can only access exported variables of the parent shell
environment. E.g.:
.. code-block:: sh
unexported='internal value'
export exported='external value'
cat <<'EOF' | sh
echo "[${unexported}]"; echo "[${exported}]"
EOF
results in output
.. code-block:: text
[]
[external value]
.. >>CODD Conclusion
.. >>CODD Appendix A
.. \|:here:|
.. >>CODD Notes
.. ==================================================
.. :rem:`|||:sec:|||`\ Footnotes
.. ==================================================
:html:`<hr>`
.. \[#]
.. >>CODD Reference List/Bibliography
.. ==================================================
.. :rem:`|||:sec:|||`\ References
.. ==================================================
.. include:: doc_defs.inc
.. include:: doc_defs_combined.inc
..
.. \||<-snap->|| doc_standalone
.. include:: doc/doc_defs_secret.inc
.. \||<-snap->|| doc_standalone
.. \||<-snap->|| not_doc_standalone
.. include:: doc_defs_secret.inc
.. \||<-snap->|| not_doc_standalone
.. _`Wolfgang Scherer`: wolfgang.scherer@gmx.de