.. -*- coding: utf-8 -*-












.. role:: sref(numref)
.. role:: xref(numref)
.. Copyright (C) 2018, Wolfgang Scherer, <Wolfgang.Scherer at gmx.de>
..
.. This file is part of Documentation Standard.
..
.. Permission is granted to copy, distribute and/or modify this document
.. under the terms of the GNU Free Documentation License, Version 1.3
.. or any later version published by the Free Software Foundation;
.. with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
.. A copy of the license is included in the section entitled "GNU
.. Free Documentation License".

.. inline comments (with du_comment_role)
.. role:: rem(comment)
.. role:: html(raw)
   :format: html
.. role:: shx(code)
   :language: sh

.. rst-class:: xnarrow xmedium large xhuge xultra
.. _`sec:Relevance of Documentation`:

##################################################
:rem:`|||:sec:|||`\ Relevance of Documentation
##################################################
.. >>CODD See `the components of a doctoral dissertation and their order <http://site.uit.no/english/writing-style/bookstructure/>`_
.. >>CODD Dedication
.. >>CODD Epigraph

.. epigraph::

   `A map is not the territory it represents`_, but, if correct, it
   has a similar structure to the territory, which accounts for its
   usefulness. If the map could be ideally correct, it would include,
   in a reduced scale, the map of the map; the map of the map, of the
   map; and so on, endlessly, a fact first noticed by Royce.

   -- Alfred Korzybski, `Science and Sanity`_, `Chapter IV`_, p. 58.

.. _`A map is not the territory it represents`: https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation
.. _`Chapter IV`: http://esgs.free.fr/uk/art/sands-ch04.pdf

.. >>CODD Abstract
.. compound::

   .. \|:here:|

   The relevance of documentation, :math:`R_{doc}` deteriorates, whenever
   the source code is changed, but the documentation is not updated.

   The rate of deterioriation for a given piece of documentation
   increases with

   - the distance to the source code, :math:`D_{source} \ge 0`
   - the inavailability of a documentation editor, :math:`I_{edit} \ge 0`
   - the necessary effort for synchronization with the master document,
     :math:`E_{sync} \ge 0`
   - the inertia of the human components :math:`H_{inertia} \ge 1`
     involved.

   The |lrd|, |LRdoc| is therefore:

   .. math::
      :label: eq_lrd_likeliness_relevance_documentation

      & L_{R_{doc}} = \left( \left( \frac{1 + ( {D_{source}} + {I_{edit}} + {E_{sync}} + {H_{inertia}} )}{2}\right)\cdot{H_{inertia}} \right)^{-1}\\

.. \|:here:|

.. >>CODD Introduction

.. _`sec:doc-relevance:Introduction`:

==================================================
:rem:`|||:sec:|||`\ Introduction
==================================================

The relevance of documentation is a measure of how similar the
structure of the documentation is to the structure of the source code.

This is closely connected to the `map–territory relation`_ of `General
semantics`_. A thorough understanding of this principle really helps
manage the expectations of what documentation of source code is, is
not and can be. The fact, that documentation can never be accurate in
the sense of a bijective mapping, does not mean it can not be helpful
-- even essential -- for a human brain to construct an appropriate
model.

.. _`fig:they might be having sushi`:
.. figctr:: I think they might be having sushi

    .. only:: html

       .. image:: _static/i-think-they-might-be-having-sushi.png
          :align: center
          :scale: 70%

    .. \|:todo:| here is an option to wrap figures in LaTeX https://stackoverflow.com/questions/16463051/how-to-create-floating-figures-in-restructuredtext-sphinx/30849577#30849577

    .. only:: latex

       .. image:: _static/i-think-they-might-be-having-sushi.png
          :align: center
          :scale: 50%

A major problem of software documentation arises from the fact, that
the documented source code is often not available to the
reader. Imagine, being given the following instructions for obtaining
an item from a certain location:

.. hlist::
   :columns: 1

   - go straight
   - go left
   - look at the sign, that says "Menu"
   - order the second item
   - bring it back

However, you are only given access up to a fence with a locked gate,
where you can see the straight part of the road and the left turn, but
you cannot see the menu. At the locked gate, there is only a sign with
yesterday's menu. Now you are supposed to predict what the second item
on today's menu will be without being able to examine the actual menu
(see :xref:`fig:they might be having sushi`). It is pretty obvious,
that this cannot be reliably done, especially if the menu is changing
daily to whatever the chef feels like.

This describes the situation developers often find themselves in when
working with proprietary software. But even when source code access is
available -- since they are incidentally the developer, who works on
it -- they are still expected to produce documentation for people that
cannot understand the code; i.e., these people -- who may just as well
be future versions of the developer himself -- choose to stay at the
fence and refuse to walk through the gate. So developers may write
things like: "And then look around in the source code, to find out
what the real state of affairs is at any given moment." However, they
cannot expect any particular reader to make the effort. See the
`Sphinx`_ and `Doxygen`_ document generators in |chapter-sphinx| for
excellent examples, how the entire source is included for
cross-referencing\ [#]_.

So, how much interest can you muster for the utterly boring
description of a treasure hunt for an unspecified treasure in a
country you don't know and don't have the means to ever get to?

For me, that is one trivially self-evident answer to questions like
"`What's with the aversion to documentation in the industry?`_".  The
rapid deterioration of relevance for documentation during the
development phase being another.

However, having no documentation at all is not a viable option and
there are certainly ways to reduce the effort and increase the
relevance of documentation.

.. _`What's with the aversion to documentation in the industry?`: https://softwareengineering.stackexchange.com/a/202171/258871

.. >>CODD Chapter

.. _`sec:Source and documentation management`:

==========================================================
:rem:`|||:sec:|||`\ Source and documentation management
==========================================================

Derived from the above equation, the best |lrd| is achieved when all
deterioration factors, |Dsource|, |Iedit|, |Esync|, |Hinertia| are at a
minimum, which makes :math:`L_{R_{doc}} = 1`.

To discuss the |lrd|, it is necessary to define a model, which
explains the parameters in their respective context (see
:xref:`fig:Source and documentation management`).

.. _`fig:Source and documentation management`:
.. uml:: _static/source-doc-management.puml
   :caption: Source and documentation management

.. note:: The required editor for a change is just one of a possible
   multitude of appropriate editors. It is not necessarily the
   preferred editor of a specific programmer (see
   :xref:`fig:Guaranteed and preferred editors`).

.. _`fig:Guaranteed and preferred editors`:
.. uml::
   :caption: Guaranteed and preferred editors

   @startuml

   skinparam padding 1

   /' |:here:| '/

   class "Source\nEditor" as SEditor << (E,aqua) >> {
   }

   object "ed(1)" as ED
   object "vi(1)" as VI
   object "notepad" as NOTEPAD
   object "nano(1)" as NANO
   object "**emacs(1)**" as EMACS

   SEditor <|.. ED
   SEditor <|.. VI      : **<<guaranteed:**\n**unix>>**
   SEditor <|.. NOTEPAD : **<<guaranteed:**\n**windows>>**
   SEditor <|.. NANO
   SEditor <|.. EMACS   : **<<preferred>>**

   @enduml

===================================================
:rem:`|||:sec:|||`\ Minimum Distance to Source Code
===================================================

A distance to source code, |Dsource| of |0| corresponds to the
principle "**the source is the documentation**", i.e., there is no
additional documentation for the source code. Substituting |Dsource|
by |0| in equation :eq:`eq_lrd_likeliness_relevance_documentation`
results in:

.. math::
   :label: eq_lrd_d_source_zero

      & \left( \left( \frac{1 + ( {I_{edit}} + {E_{sync}} + {H_{inertia}} )}{2}\right)^{H_{inertia}} \right)^{-1}\\

In this case, the source editor and the documentation editor are
necessarily the same and therefore |Iedit| can be eliminated from
equation :eq:`eq_lrd_d_source_zero`:

.. math::
   :label: eq_lrd_ds0_i_edit_zero

   & \left( \left( \frac{1 + ( {E_{sync}} + {H_{inertia}} )}{2}\right)^{H_{inertia}} \right)^{-1}\\
   & L_{R_{doc}} = ( 1 + ( {D_{source}} + {E_{sync}} )^{H_{inertia}} )^{-1}\\

When the source code changes are made to the master source file,
|Esync| is also |0| and equation :eq:`eq_lrd_ds0_i_edit_zero` is reduced
to:

.. math::
   :label: eq_lrd_ds0_ie0_e_sync_zero

    & \left( \left( \frac{1 + {H_{inertia}}}{2}\right)^{H_{inertia}} \right)^{-1}\\

Without automatic documentation generators that have to be triggered
manually, human inertia |Hinertia| is |1| and equation
:eq:`eq_lrd_ds0_ie0_e_sync_zero` is reduced to |1|:

.. math::
   :label: eq_lrd_minimum

   & \left( \left( \frac{1 + 1}{2}\right)^{1} \right)^{-1} = 1\\

So it seems, that the only time, documentation is guaranteed to be
entirely relevant, is if there is none. In all other cases,
documentation and source strictly abide by the properties of the
`map–territory relation`_\ [#]_.

However, there are tools like grep(1), ctags(1), etags(1), cscope(1),
even `Sphinx`_ and `doxygen`_, also (reluctantly) IDEs like Eclipse,
which extract meta-information from undocumented source code to
provide hyper linking facilities, which is in itself an extremely
useful part of any documentation.

Other Generators produce Nassi-Shneiderman diagrams (flowcharts,
activity diagrams), dependency graphs, etc.

Therefore the principle "**the source is the documentation**" is
actually not so blunt or ridiculous as it may sound at first.

However, since automatically generated documentation requires a
compilation step that may require manual triggering, |Hinertia| may
quickly become a signifanct factor.

==================================================
:rem:`|||:sec:|||`\ Inertia
==================================================

In the eternal sunshine of the spotless mind, each relevant source
modification triggers a documentation update (see :xref:`fig:Sunny day
documentation update`).

.. _`fig:Sunny day documentation update`:
.. uml::
   :caption: Sunny day documentation update
   :scale: 50%

   @startuml
   skinparam padding 1
   /' |:here:| '/
   state "Editing Sources in the Sun" as SM {
     state "Source\nediting" as Editing
     state "Documentation\nupdate" as DocUpdate

     [*] --> Editing
     Editing -> DocUpdate : relevant\nsource\nchange
     DocUpdate --> Editing : updated
     Editing -> [*] : finished
   }
   /' |:here:| '/
   @enduml

But this is only true, when no human components are involved in the
process of deciding, whether a source change is relevant for a
documentation update. And this can only be the case when no manually
created documentation parts have to be maintained.  An example for
this is the initial documentation produced by generators like
`Sphinx`_ and `Doxygen`_.

However, there is usually a more or less complex update decision
process involved, which potentially results in increases of |Iedit|
and human inertia |Hinertia| (see :xref:`fig:Rainy day documentation
update`).

.. _`fig:Rainy day documentation update`:
.. uml::
   :caption: Rainy day documentation update
   :scale: 50%

   @startuml
   skinparam padding 1
   !define I_EDIT **I<sub>edit</sub>**
   !define H_INERTIA **H<sub>inertia</sub>**
   /' |:here:| '/
   state "Editing Sources on a Rainy Day" as SM {
     state "Editing\nsource" as Editing
     ' state "Source and\nDocumentation\nsync'ed" as Synced
     state "Documentation\nupdate" as DocUpdate
     state "Update\ndecision" as Deciding
     state "Increment\nInertia" as IncInertia
     IncInertia : H_INERTIA += rand()
     state "Increment\nInavailability" as IncInavail
     IncInavail : I_EDIT += rand()

     [*] --> Editing
     Editing -> Deciding : source\nchanged
     Deciding --> Editing : not required
     Deciding --> IncInavail : not possible
     IncInavail --> Editing
     Deciding --> IncInertia : no update
     IncInertia --> Editing
     Deciding --> DocUpdate : do update
     DocUpdate --> Editing : updated
     Editing -> [*] : finished

   }
   /' |:here:| '/
   @enduml

The decision process in state `Update Decision` looks somewhat like
the activity diagram in :xref:`fig:Documentation update
decision`. Feel free to add an infinite number of excuses for not
updating the documentation or postponing the update indefinitely.

.. _`fig:Documentation update decision`:
.. uml::
   :caption: Documentation update decision

   @startuml
   skinparam padding 2
   /' |:here:| '/
   partition "Update Decision (Procrastination in the 9th Circle of Hell)" {
     start
     :res = undecided;

     repeat
       if (manual upate\n//not// required?) then (yes)
         :res = not required;
       elseif (deadline is near and\nedit takes too long?) then (yes)
         :res = no update;
       elseif (do you just //not//\nfeel like it?) then (yes)
         :res = no update;
       elseif (documentation editor\nis available?) then (yes)
         if (do you know\nhow to use it?) then (yes)
           :res = do update;
         else (no)
           :res = no update;
         endif
       elseif (editor is\ninstallable?) then (yes)
         if (you feel like\ninstalling?) then (yes)
           :install editor;
         else (no)
           :res = no update;
         endif
       else (no)
         :res = not possible;
       endif
     repeat while (res == undecided?) is (yes)

     if (do you no longer feel like it?) then (yes)
       :res = no update;
     endif

     -> return res;
     stop

     floating note right
     res is one of
     * not required
     * not possible
     * no update
     * do update
     end note
   }

   /' |:here:| '/
   @enduml

==============================================================
:rem:`|||:sec:|||`\ Single Editor for Source and Documentation
==============================================================

When the documentation editor is the same as the source editor, the
decisions about using or installing a separate documentation editor
are removed. Therefore the result "**not possible**" is no longer
returned (see :xref:`fig:Documentation Update Decision With Single Editor`).

.. _`fig:Documentation Update Decision With Single Editor`:
.. uml::
   :caption: Documentation Update Decision With Single Editor

   @startuml
   skinparam padding 2
   /' |:here:| '/
   partition "Documentation Update Decision With Single Editor (The Dog Ate My Homework)" {
     start
     :res = undecided;

     repeat
       if (no manual upate\nrequired) then (yes)
         :res = not required;
       elseif (deadline is near\nedit takes too long) then (yes)
         :res = no update;
       elseif (do not feel like it) then (yes)
         :res = no update;
       else (no)
         :res = do update;
       endif
     repeat while (res == undecided) is (yes)

     if (no longer feel like it) then (yes)
       :res = no update;
     endif

     -> res;
     stop

     floating note right
     res is one of
     * not required
     * no update
     * do update
     end note
   }

   /' |:here:| '/
   @enduml

Consequently, the state machine for editing sources is also reduced.
The state "**Increment Inavailability**" is removed and therefore
|Iedit| is no longer incremented (see :xref:`fig:Editing with single
editor`).

.. _`fig:Editing with single editor`:
.. uml::
   :caption: Editing with single editor

   @startuml
   skinparam padding 1
   /' |:here:| '/
   !define H_INERTIA **H<sub>inertia</sub>**
   state "Editing with Single Editor" as SM {
     state "Source\nediting" as Editing
     state "Documentation\nupdate" as DocUpdate
     state "Update\ndecision" as Deciding
     state "Increment\nInertia" as IncInertia
     IncInertia : H_INERTIA += rand()

     [*] --> Editing
     Editing -> Deciding : source\nchanged
     Deciding --> Editing : not required
     Deciding --> IncInertia : no update
     IncInertia --> Editing
     Deciding --> DocUpdate : do update
     DocUpdate --> Editing : updated
     Editing -> [*] : finished
   }
   /' |:here:| '/
   @enduml

Therefore |Iedit| is always |0| and equation
:eq:`eq_lrd_likeliness_relevance_documentation` is reduced to:

.. math::
   :label: eq_lrd_i_edit_zero

   & L_{R_{doc}} = \left( \left( \frac{1 + ( {D_{source}} + {E_{sync}} + {H_{inertia}} )}{2}\right)\cdot{H_{inertia}} \right)^{-1}\\

.. note:: Please, note, that in addition to a documentation editor,
   there may be separate documentation generators required to produce
   some form of final documentation (e.g., SVG, PNG, JPEG, HTML, PDF).
   However, the lack of these does not contribute to the deterioration
   of relevance, as long as the documentation source is updated.

   If it is found, that the update of a piece of documentation without
   a visual representation created by the documentation generator is
   not possible, this decision may have to be revisited.

==================================================
:rem:`|||:sec:|||`\ Synchronization
==================================================

When there is a single file on just one computer, there is obviously
no synchronization necessary and therefore :math:`E_{sync} = 0`.

Sometimes it is better to copy a project directory and work on the
copy so that others are not disturbed by the intermediate states of
development. That is called a `branch` or `working on a branch`. The
original project file is the `master`.

As long as nothing leaves the branch directory, the documentation is
not affected by the effort necessary for synchronizing the branch with
the master. It is only, when e.g. a binary version from the branch is
delivered to a production system, that the master documentation
becomes less relevant until the branch with all documentation changes
is synchronized (`merged`) with the master. This is the situation when
:math:`E_{sync}` becomes :math:`> 0`.

Human forgetfulness and general sloppiness are major factor for
discrepancies, where branches are left to rot on flash drives and,
newer versions are overwritten by older versions, etc.

This is why the process is nowadays highly automated by distributed
version control systems (DVC).

DVCs are quite good, but sometimes you don't want to check in changes,
that need some final touch. In that case, you are back to copy or
remote copy (scp(1), rsync(1)). So all other stages are still useful:

- backup file
- diff(1), patch(1), diff3(1)
- scp(1), rsync(1)
- VCS
- DVCS

--------------------------------------------------
:rem:`||:sec:||`\ qs-gen-sync.pl (.sync.rc)
--------------------------------------------------

.. role:: shx(code)
   :language: sh

There are two aliases for `backup` and `restore`:

bsy
  send stuff to remote host :shx:`./sync.sh --backup`

rsy
  get stuff from remote host :shx:`./sync.sh --restore`

Generate initial `.sync.rc`:

.. code-block:: sh

   isy -n

--------------------------------------------------
:rem:`||:sec:||`\ diff3
--------------------------------------------------

|:todo:| finish description of diff3(1)

::

  diff3 doc/index.rst README.txt doc/overview.rst

==================================================
:rem:`|||:sec:|||`\ Battling Human Inertia
==================================================

The only way to defeat human inertia, |Hinertia|, is automation.

See also :xref:`fig:Geeks and repetitive tasks`

|:todo:| finish description of scripting sh(1), make(1)

- make/scripts are memory banks for knowledge

.. _`fig:Geeks and repetitive tasks`:
.. figctr:: Geeks and repetitive tasks

   .. image:: _static/alternate-view-of-automation.png

   Source: `Another way to think about geeks and repetitive tasks - Jon Udell`_

.. _`Another way to think about geeks and repetitive tasks - Jon Udell`: https://blog.jonudell.net/2012/01/09/another-way-to-think-about-geeks-and-repetitive-tasks/

..
    .. image:: _static/geeks-and-repetitive-tasks.png

    Source: `Bruno Oliveira`_

    .. _`Bruno Oliveira`: https://plus.google.com/102451193315916178828/posts/MGxauXypb1Y

.. _`sec:doc-relevance - Integrated Development Environment`:

----------------------------------------------------
:rem:`||:sec:||`\ Integrated Development Environment
----------------------------------------------------

An `IDE`_ strives to be an *integrated* solution for cross-referencing,
build automation, generated documentation.

In that sense, emacs(1) can also be classified as an `IDE`_.

`Eclipse`_ is ridiculously hard to extend out of the box with plug-ins
written in Java. E.g., in a forum question from 2016 about `how to
extend the default Java editor`_, somebody just wants to add a few
keywords to the Java editor and gets links to a `PDF slideshow about
JDT`_ and to `Xtext`_. Neither are anywhere near trivial.

.. _`Xtext`: https://eclipse.org/Xtext/index.html
.. _`PDF Slideshow about JDT`: https://www.eclipse.org/eclipse/platform-text/eclipseCon/talk.pdf
.. _`How to extend the default Java editor`: https://www.eclipse.org/forums/index.php/t/1075624/

`EASE`_ does provide integration of script engines into `Eclipse`_
giving access to the workbench. This engine may be able to handle
couple of keywords, if it can be added to `Eclipse`_ extension points
(hooks), which needs to be researched |:todo:|. So in 2017 eclipse
announces another `convoluted overstructured object oriented
integration`_, which still does not reach easily into an editing
buffer (see also `Internal and External Editors`_).

.. _`convoluted overstructured object oriented integration`: https://www.eclipse.org/community/eclipse_newsletter/2017/october/article3.php
.. _`Internal and External Editors`: https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fextension-points%2Forg_eclipse_ui_editors.html

See also
https://stackoverflow.com/questions/26912785/eclipse-jdt-syntax-highlighting-of-constants
from 2014, which is still unanswered. The answers in
https://stackoverflow.com/questions/13802131/in-an-eclipse-plugin-how-can-i-programmatically-highlight-lines-of-codes-in-the,
give a variety of strategies to follow, but none of the answers gives
a step-by-step example.

Here is another failed attempt
https://www.eclipse.org/forums/index.php/t/489221/ for adding some
highlighting to the `Eclipse`_ editor.

All in all `Eclipse`_ has still not reached the efficient state of the
art provided by emacs(1) for decades.  emacs(1) can be extended
ridiculously simple with a multitude of tools glued together by Elisp.

As discussed in the article `Using Automation to Assist – not replace
– Manual Testing`_, automating entire workflows is very much
ineffective (see :xref:`Using Automation to assist`).

.. _`Using Automation to assist`:
.. figctr:: Using Automation to assist

   .. image:: _static/reality-of-automation.png

   Source: `Using Automation to Assist – not replace – Manual Testing`_

.. _`Using Automation to Assist – not replace – Manual Testing`: https://smartbear.de/blog/test-and-monitor/using-automation-to-assist-not-replace-manual-test/?l=ua

In that context, a tightly integrated `IDE`_ -- like `Eclipse`_ or
Visual Studio -- is the ineffective attempt to provide an entirely
automated workflow in a one-size-fits-all manner.

In contrast, the loosely integrated approach with emacs(1) and tools
provides a framework for task automation which can adapt to a variety
of workflows.

A monolithic `IDE`_ usually has a plethora of cryptic configuration
settings to provide some sort of rudimentary flexibility
(customization), whereas the tool based `IDE`_ allows full
modification of each automation step down to the code level.

.. \|:here:|

.. >>CODD Conclusion
.. >>CODD Appendix A

.. \|:here:|

.. >>CODD Notes
.. ==================================================
.. :rem:`|||:sec:|||`\ Footnotes
.. ==================================================

:html:`<hr>`

.. [#] `Sphinx`_ even includes the source code for the documentation
       itself. However, the `map–territory relation`_ still holds,
       since there is no description of the process that transforms
       the documentation source code into the final output.

.. [#] Considering that the source code of a program is itself a map
       of an executable, requiring a processor (interpreter) and
       possibly a compiler to produce another representation, there
       really is no simple escape from the `map–territory relation`_.

.. include:: doc_defs.inc
.. include:: doc_defs_combined.inc
.. include:: abbrev_defs.inc
..
  .. \||<-snap->|| doc_standalone
  .. include:: doc/doc_defs_secret.inc
  .. \||<-snap->|| doc_standalone
  .. \||<-snap->|| not_doc_standalone
  .. include:: doc_defs_secret.inc
  .. \||<-snap->|| not_doc_standalone

.. _`Wolfgang Scherer`: wolfgang.scherer@gmx.de