NEWS for version 1.12

   Release date: 2019-04-10
   Intended compatibility: WSUS Offline Update 11.6.1+ (r1032)

   Internal changes

   Indirect addressing of virus definition files with "LinkIDs"

      The four virus definition files are now referenced with URLs,
      with refer to LinkIDs first. The filename on the server can only
      be retrieved after several redirections.

      Several functions had to be adjusted to work with these LinkIDs:
      calculate_static_downloads_wddefs8, download_single_file,
      download_single_file_failsafe, and cleanup_client_directory.

      Note: Some debug messages in these functions have been left
      enabled, to see, if anything changes. They will be removed in the
      next versions.

        * Forum: https://forums.wsusoffline.net/viewtopic.php?f=6&t=9098
        * Trac:  http://trac.wsusoffline.net/trac.fcgi/changeset/1032


NEWS for version 1.11

   Release date: 2019-04-04
   Intended compatibility: WSUS Offline Update 11.6.1 and later

   New features

   The script update-generator.bash remembers the last used settings

      The script update-generator.bash writes the current settings to a
      settings file update-generator.ini, if the external utility
      “dialog” is used. On the next run, the settings are reloaded and
      the previously selected options are checked again.

      Note, that this does not work with the internal command “select”
      of the bash.

   Revised method for calculating superseded updates

      An optional, new method for the calculation of superseded updates
      removes one possible cause for missing updates: It automatically
      corrects the list of superseded updates for updates, which are
      only superseded by full quality update rollups, but not by
      security-only updates. This may prevent some rare problems with
      missing updates.

      The current implementation for calculating superseded updates in
      both Windows and Linux is depicted in the forum article:

        * https://forums.wsusoffline.net/viewtopic.php?f=5&t=5676

      One problem with this implementation is, that superseded updates
      may be missing, if the superseding updates are excluded from
      download.

      This is an old problem, which was observed a few times:

      It was first found with Windows XP: The embedded Windows XP
      POSReady was supported longer than the regular desktop versions.
      Updates for the embedded version can supersede older updates for
      the desktop versions. The newer updates for the embedded version
      are not downloaded by WSUS Offline Update, because they cannot be
      installed on the desktop versions. But the older updates for the
      desktop versions are still treated as superseded and not
      downloaded either. This was solved by adding the missing updates
      to the file ExcludeList-superseded-exclude.txt.

      When monthly quality update rollups and security-only updates were
      first introduced, the quality update rollups superseded the
      security-only updates. WSUS Offline Update needed two steps to
      support security-only updates:

        * The quality update rollups had to be excluded from download
          and installation.

        * The security-only updates had to be re-enabled for download.

      At this point, I first suggested a new method, which could
      reschedule superseded updates for download, if the superseding
      updates are excluded from download. The problem with this method
      was, that it needed an initial block list of excluded downloads to
      start with, and this list didn't exist yet. So it was only a
      partial solution.

        * https://forums.wsusoffline.net/viewtopic.php?f=5&t=6141

      The initial release of the Linux download scripts, version
      1.0-beta-1, actually included an implementation of this new method
      in the subdirectory available-tasks.

      The differentiation of quality update rollups and security-only
      updates was finally solved in WSUS Offline Update by creating new
      configuration files in the client/static and client/exclude
      directories.

      In the meantime, Microsoft changed the way, how quality update
      rollups and security-only updates depend on each other, after just
      one month:

        “UPDATED 12/5/2016: Starting in December 2016, monthly rollups
        will not supersede security only updates. The November 2016
        monthly rollup will also be updated to not supersede security
        only updates.”

        https://techcommunity.microsoft.com/t5/Windows-Blog-Archive/More-on-Windows-7-and-Windows-8-1-servicing-changes/ba-p/166783

      A more subtle problem is, that the quality update rollups
      sometimes seem to include older updates, while the security-only
      updates only include new updates for the current month. Then these
      older updates will be missing, if security-only updates are
      selected. Again, this was solved by adding the missing updates to
      a new configuration file
      ExcludeList-superseded-exclude-seconly.txt.

      The problems with the files ExcludeList-superseded-exclude.txt and
      ExcludeList-superseded-exclude-seconly.txt is, that they can only
      be updated, after some updates have been found missing.

      When I retested the new method, it could detect two missing
      updates, before they were reported as missing:

        * https://forums.wsusoffline.net/viewtopic.php?f=4&t=7085
        * https://forums.wsusoffline.net/viewtopic.php?f=2&t=8697

      So, this method may still be useful, and the script
      60-main-updates.bash now includes an optional implementation. It
      uses the file HideList-seconly.txt as an initial block list, to
      automatically correct the list of superseded updates for updates,
      which are only superseded by the monthly quality update rollups,
      but not by security-only updates.

      To select the new file
      ExcludeList-Linux-superseded-seconly-revised.txt for the
      calculation of dynamic updates, both options prefer_seconly and
      revised_method must be set to “enabled” in the preferences file.

   New script open-support-pages.bash

      This script opens the Microsoft support pages for a series of kb
      numbers.

      It tries a series of Linux “open handlers”, to open the URLs with
      the preferred application of the desktop environment. Suitable
      open handlers are:

       Open handler      Package name        Desktop environment
       gio open          libglib2.0-bin      GNOME 3.30 in Debian 10
       gvfs-open         gvfs-bin            GNOME 3.22 in Debian 9
       gnome-open        libgnome2-bin       GNOME 2
       kde-open5         kde-cli-tools       KDE 5 (untested)
       kde-open          kde-runtime         KDE 4 (untested)
       exo-open          exo-utils           Xfce
       xdg-open          xdg-utils           others

       (The package names are for Debian and related distributions.)

      In addition to these open handlers, sensible-browser, firefox-esr
      and firefox are also tried, because this script only needs to
      handle http or https URLs. The script /usr/bin/sensible-browser is
      part of the update-alternatives system in Debian. It uses
      gnome-www-browser, x-www-browser or www-browser, depending on the
      context.

      If none of the above can be found, then the script recommends the
      installation of xdg-open as a general open handler, which is not
      tied to a particular desktop environment.

      Note: Neither gvfs-open nor Firefox can handle multiple URLs on
      the command line. Calling xdg-open repeatedly usually means that
      the application should be launched multiple times, but this will
      fail with Firefox. To have Firefox open the URLs in multiple tabs,
      it should be launched first, before running this script.

   Bug fixes

   Workaround for broken cabextract in Debian 10 Buster/testing

      As of 2019-03-26, the package cabextract is still broken in Debian
      10 Buster/testing, long after two relevant bug reports have been
      marked as fixed and closed.

        * libmspack0: Regression when extracting cabinets using -F
          option fixed upstream, needs to be patched
          https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=912687

        * cabextract: -F option doesn't work correctly
          https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914263

      But so far, the fixed packages only arrived in Debian
      Sid/unstable. They are not yet available in Debian Buster/testing.

      So I added some more tests and a workaround, which does not use
      the cabextract option -F. Then the file wsusscn2.cab must be
      completely unpacked, which may take slightly longer.

        * Forum: https://forums.wsusoffline.net/viewtopic.php?f=9&t=8706

   User visible changes

   Better support for terminal colors

      Version 1.4 of the Linux download scripts introduced text
      formatting of the output, using bold text and terminal colors.
      Such text formatting should only be used, if the output is written
      to a terminal emulator window or to a virtual console. It should
      not be used, if the output is redirected to a file or piped to
      another application, for example if the script is running as a
      cron job.

      The library messages.bash uses two tests, to make sure that text
      formatting can used safely:

        * First, the script tests, if standard output and error output
          are attached to a terminal, using the test -t of POSIX shells.

        * Then all escape sequences are determined with the utility
          tput, rather than hard-coding them. tput checks again, if text
          formatting is safe to use.

      But tput is overly restrictive in the use of terminal colors: It
      only uses colors, if the environment variable TERM is set to
      “xterm-256color”, “rxvt-256color” or “rxvt-unicode-256color”.
      Otherwise, only bold text is used.

      Many terminal emulators simply set TERM to “xterm”, and they don't
      provide any means to change this environment variable. Still, all
      tested terminal emulators support colors, including xterm itself.
      Then it should be safe to change TERM from xterm or xterm-color to
      xterm-256color, to get the expected results.

      The same adjustments could be done with rxvt, but Debian already
      provides different builds with different settings for rxvt. The
      urxvt from the package rxvt-unicode-256color sets the environment
      variable TERM to “rxvt-unicode-256color”, which is recognized as a
      color-capable terminal by tput.

      Notes:

      There are other ways to set or change the environment variable
      TERM. This may benefit other applications as well. For example,
      some themes for the Midnight Commander also require 256 colors.

      Within a shell, environment variables can be defined before the
      script or application to run:

      ~$ TERM=xterm-256color ./update-generator.bash

      Some terminal emulators like the MATE Terminal allow to run a
      custom command instead of the standard shell. This can be used to
      set environment variables with:

      /usr/bin/env TERM=xterm-256color bash

      The environment variable TERM could also be set in files like
      ~/.profile or ~/.bashrc, but this may give unexpected results:
      Using TERM=xterm in the Linux console will mess up the output of
      the external utility “dialog”, because the box drawing characters
      are different.

   Support for ExcludeListForce-all.txt

      The custom file
      wsusoffline/exclude/custom/ExcludeListForce-all.txt was already
      applied to static and dynamic updates for Windows, Office and .NET
      Frameworks. It is now applied to the .NET Framework installation
      files as well.

        * Forum: https://forums.wsusoffline.net/viewtopic.php?f=2&t=8901
        * Trac: http://trac.wsusoffline.net/trac.fcgi/changeset/1015

   Support for ExcludeList-superseded-exclude-seconly.txt

      The custom file
      wsusoffline/exclude/custom/ExcludeList-superseded-exclude-seconly.txt
      will be used for the calculation of superseded updates, if
      security-only updates are selected.

        * Forum: https://forums.wsusoffline.net/viewtopic.php?f=2&t=8697
        * Trac: http://trac.wsusoffline.net/trac.fcgi/changeset/1002

   Virus definition files are tested with cabextract

      The four virus definition files are basically self-extracting
      cabinet files. They can be at least partially tested with
      cabextract -t.

   The scripts compare-integrity-database.bash and
   compare-update-tables.bash are configured with command-line arguments

      These scripts are used for development: They compare the
      directories md and ofc on Windows and Linux. Previously, these
      directories had to be edited directly in the scripts, but they are
      now passed as command-line arguments.

   Internal changes

   Reordered the sections at the top of the scripts update-generator.bash
   and download-updates.bash

      The new order in these files is:

       1. Shell options

       2. Environment variables

       3. Configuration: script version and release date

       4. Global variables: script name and home directory, other
          directories and log file

       5. Preferences: default values for the settings in the optional
          preferences file

       6. Traps

       7. Functions

       8. Commands

   The directories cache, log and timestamps are stored with absolute paths

      The directories cache, log and timestamps used to be defined with
      the relative paths ../cache, ../log and ../timestamps. This does
      not work well with external utilities like hashdeep and curl,
      which require constant changes to the current working directory
      and thereby make relative references invalid.

      After revealing the current working directory with readlink, these
      directories are now stored with absolute paths.

   The function seconly_safety_guard calculates the patch days for two
   months

      The function seconly_safety_guard tries to make sure, that some
      configuration files in the directories client/exclude and
      client/static have been updated after each patch day, before
      downloading security-only updates. Without this configuration,
      WSUS Offline Update would default to download and install the full
      quality update rollups, or it might even download and install both
      sets of updates.

      The first implementation only calculated the patch day of the
      current month, and compared the modification date of the
      configuration files to this date.

      Now the function calculates the official patch days for the last
      two months, and selects the right one for comparison:

        * The last patch day is the second Tuesday of the current month,
          if today is on the same day or later.

        * Otherwise, the last patch day is the second Tuesday of the
          last month.

      These calculations also use an integer value for the day of the
      week, instead of comparing the weekday names literally.

   The function log_message duplicates messages to the terminal and the log
   file unchanged

      The function log_message does not prefix the message with the
      current date anymore. It is now used to duplicate the output of
      hashdeep to the terminal and to the log file.

   The function apply_exclude_lists now skips empty lines in input files

      The function apply_exclude_lists now reads the input files
      line-by-line. Empty lines are ignored. This should prevent errors
      with files, which only consist of one empty line.

   New function name_to_description and changed function
   language_name_to_locale

      The new function name_to_description checks, that the specified
      update, language or option name exists in a table and returns the
      description. This replaces some awkward constructs with grep.

      The function language_name_to_locale was rewritten in the same way
      to return the locale for a language name, e.g. deu → de, enu → en.

   The file sh/exclude/ExcludeListUSB-w60.txt was simplified

      The different filters vcredist* were replaced with a single filter
      *_x64*.

        * Forum: https://forums.wsusoffline.net/viewtopic.php?f=5&t=8258
        * Trac: http://trac.wsusoffline.net/trac.fcgi/changeset/989

   Added Cygwin to known systems and unzip to needed packages, thanks to
   "slycordinator"

      Cygwin is basically a Linux environment on Windows. All needed GNU
      utilities and other packages should be available. trash-cli can be
      installed from the Python repository with "pip install trash-cli".

      unzip is needed for the self-update of the WSUS Offline Update
      installation and to unpack the Sysinternals utilities Autologon and
      Sigcheck. unzip is often installed with graphical archive managers
      like Xarchiver, but it may be missing on a basic command-line
      system.

   Documentation

      The new file compatibility.txt lists the distributions, on which
      the Linux scripts were at least briefly tested, and the
      corresponding Bash versions.

      Some typos were corrected.

      The E-Mail address was replaced.


NEWS for Version 1.10

  Release date: 2018-08-09
  Intended compatibility: WSUS Offline Update 11.4 and later

  Changes in this version

  - Bug fix: USB filters for w61 and newer erroneously excluded ndp46-*
    patches.

    The private copies of the files ExcludeListUSB-*.txt
    were patched according to Changeset 981:
    http://trac.wsusoffline.net/trac.fcgi/changeset/981

    Actually, in the files ExcludeListISO-*.txt, only
    the lower-case filter *ndp46-* was removed, while the
    upper-case filter *NDP46-* is kept. This excludes the two
    installation files NDP46-KB3045557-x86-x64-AllOS-ENU.exe and
    NDP46-KB3045557-x86-x64-AllOS-DEU.exe, but not the dynamic updates
    for .NET Framework 4.6.

    A possible reason for this distinction is to keep the size of ISO
    images below 4.7 GB. Excluding large installers may be needed to
    reach this goal.

    This distinction is not used for the ExcludeListUSB-*.txt files,
    because the filters for xcopy.exe are not case-sensitive, and copying
    to an external USB device does not have the same limitations for
    the overall file size as creating ISO images.

  - The script 10-show-selection-dialogs-with-dialog.bash was made
    more configurable, by moving the variable parts of the dialogs to
    a configuration section.

  - Small corrections to the installation guide.


NEWS for Version 1.9

  Release date: 2018-07-30
  Intended compatibility: WSUS Offline Update 11.4 and later

  Changes in this version

  - The script update-generator.bash now uses the external utility
    "dialog", to display nicely formatted dialogs in the terminal window.

    All three dialogs for updates, languages and optional downloads
    allow multiple selections. This allows to get all needed updates
    with a single call of the download script download-updates.bash.

    The existing script, which uses the internal command "select" of
    the bash, is kept as a fallback, if dialog is not installed. This
    command only allows single selections.

    The new script 10-show-selection-dialogs-with-dialog.bash is based
    on a mockup, which I once created for the now obsolete script
    DownloadUpdates.sh:
    http://forums.wsusoffline.net/viewtopic.php?f=9&t=4061


NEWS for Version 1.8

  Release date: 2018-07-27
  Intended compatibility: WSUS Offline Update 11.4 and later

  Changes in this version

  - The documentation of the Linux download scripts is organized more
    like the Linux documentation in the directory /usr/share/doc.

    The file NEWS.txt replaces the former release_notes_[version].txt. It
    contains detailed information about the last five versions in
    reverse order.

    The file changelog.txt replaces the former version-history.txt. It
    contains a summary of the changes for all versions in reverse order.

  - The file 70-synchronize-with-target.bash was replaced with a more
    elaborate script copy-to-target.bash.

    The usage is:

    ./copy-to-target.bash <update> <destination-directory> [<option> ...]

    The <update> can be one of:

    all           All Windows and Office updates, 32-bit and 64-bit
    all-x86       All Windows and Office updates, 32-bit
    all-win-x64   All Windows updates, 64-bit
    all-ofc       All Office updates, 32-bit and 64-bit
    wxp           Windows XP, 32-bit                    (ESR version only)
    w2k3          Windows Server 2003, 32-bit           (ESR version only)
    w2k3-x64      Windows XP / Server 2003, 64-bit      (ESR version only)
    w60           Windows Vista / Server 2008, 32-bit
    w60-x64       Windows Vista / Server 2008, 64-bit
    w61           Windows 7, 32-bit
    w61-x64       Windows 7 / Server 2008 R2, 64-bit
    w62           Windows 8, 32-bit                     (ESR version only)
    w62-x64       Windows 8 / Server 2012, 64-bit
    w63           Windows 8.1, 32-bit
    w63-x64       Windows 8.1 / Server 2012 R2, 64-bit
    w100          Windows 10, 32-bit                (current version only)
    w100-x64      Windows 10 / Server 2016, 64-bit  (current version only)

    The available options for the update parameter are determined by
    the installed files wsusoffline/exclude/ExcludeListUSB-*.txt, which
    are meant for the Windows script CopyToTarget.cmd. There is a strict
    one-to-one relationship:

    Update        Used exclude list
    ------        -----------------
    all           ExcludeListUSB-all.txt
    all-x86       ExcludeListUSB-all-x86.txt
    all-win-x64   ExcludeListUSB-all-x64.txt
    all-ofc       ExcludeListUSB-ofc.txt
    wxp-x86       ExcludeListUSB-wxp-x86.txt   (ESR version only)
    w2k3          ExcludeListUSB-w2k3-x86.txt  (ESR version only)
    w2k3-x64      ExcludeListUSB-w2k3-x64.txt  (ESR version only)
    w60           ExcludeListUSB-w60-x86.txt
    w60-x64       ExcludeListUSB-w60-x64.txt
    w61           ExcludeListUSB-w61-x86.txt
    w61-x64       ExcludeListUSB-w61-x64.txt
    w62           ExcludeListUSB-w62-x86.txt   (ESR version only)
    w62-x64       ExcludeListUSB-w62-x64.txt
    w63           ExcludeListUSB-w63-x86.txt
    w63-x64       ExcludeListUSB-w63-x64.txt
    w100          ExcludeListUSB-w100-x86.txt  (current version only)
    w100-x64      ExcludeListUSB-w100-x64.txt  (current version only)

    The Windows script CopyToTarget.cmd uses xcopy.exe, and the exclude
    lists had to be edited to work with rsync on Linux. Therefore,
    the Linux script now uses its own set of these files. Some files
    are also renamed to match the names of the command-line parameters.

    The destination directory is the directory, to which files are copied
    or hard-linked. It should be specified without a trailing slash,
    because otherwise rsync may create an additional directory within
    the destination directory.

    The options are:

    -includesp         Include service packs
    -includecpp        Include Visual C++ Runtime Libraries
    -includedotnet     Include .NET Frameworks
    -includewddefs     Include Windows Defender virus definitions for
                       the built-in Defender of Windows Vista and 7.
    -includemsse       Include Microsoft Security Essentials. The virus
                       definitions are also used for the built-in Defender
                       of Windows 8, 8.1 and 10.
    -cleanup           Tell rsync to delete obsolete files from included
                       directories. This does not delete excluded files
                       or directories.
    -delete-excluded   Tell rsync to delete obsolete files from included
                       directories and also all excluded files and
                       directories. Use this option with caution,
                       e.g. try it with the option -dryrun first.
    -hardlink <dir>    Create hard links instead of copying files. The
                       link directory should be specified with an
                       absolute path, otherwise it will be relative to
                       the destination directory. The link directory
                       and the destination directory must be on the same
                       file system.
    -dryrun            Run rsync without copying or deleting
                       anything. This is useful for testing.

    The operation "per language" is not supported, because it is not
    needed anymore. It was useful for Windows XP and Server 2003, because
    these Windows versions had localized updates. But all Windows versions
    since Vista have global/multilingual updates, and Office updates
    are all lumped together, with most updates in the ofc/glb directory.

    There are two known differences in the results between the Windows
    script CopyToTarget.cmd and the new Linux script copy-to-target.bash
    ( see http://forums.wsusoffline.net/viewtopic.php?f=5&t=8258 ):

    1. The original file wsusoffline/exclude/ExcludeListUSB-w60-x86.txt
       misses an entry for vcredist2017_x64.exe. This means, that this
       file is not excluded by the Windows script, if the update "w60"
       is selected.

    2. The file wsusoffline/client/bin/IfAdmin.cpp is only
       excluded by the Windows script CopyToTarget.cmd, if
       the option /includedotnet is NOT used. Then the file
       wsusoffline/exclude/ExcludeListISO-dotnet.txt is appended to the
       filter file. With xcopy.exe, the line "cpp\" matches both the
       directory "cpp" (as expected) and the source file "IfAdmin.cpp".

       But the file IfAdmin.cpp is neither needed for download nor for
       installation, and it should always be excluded. It is only included
       in WSUS Offline Update, because the GPL demands, that the source
       code of all utilities should be made available somewhere.

  Internal changes

  - The definition of the environment variables LINES and COLUMNS, and of
    the terminal colors was moved from the scripts update-generator.bash
    and download-updates.bash to the library messages.bash.

    These variables are only used in the library messages.bash, and then
    they should be defined there. The library messages.bash also provides
    standard values for the global variables logfile and debug, to make
    the library more self-contained and suitable for other scripts.

  - Hashdeep errors while checking the integrity of existing files are
    now reported as errors, not as warnings.

    Before each download run, the integrity of existing files is verified
    with hashdeep. As a forensic tool, hashdeep treats all changes to the
    examined directory as errors, including the manual removal of files.

    There are rarely real problems at this point, and the fix is
    to delete the corresponding hashdeep files in the directory
    wsusoffline/client/md. They will be rebuilt on the next download run.

    But this is actually the normal progress: The hashdeep files will
    be deleted and rebuilt after each download run anyway. Therefore,
    hashdeep errors for the verification of existing files were only
    reported as "warnings".

    Hashdeep errors for existing files still increment an internal counter
    for runtime errors, and for consistency they are now reported as
    "errors".

  - Corrected the copyright year of the files error-counter.bash and
    rebuild-integrity-database.bash to "2018".

    These files were added in 2018, but due to lazy copy-and-past the
    copyright was set to 2016-2018.


Release Notes for Version 1.7

  Release date: 2018-05-25
  Intended compatibility: WSUS Offline Update Version 11.3 and later

  Changes in this version

  - Bug fix: The script download-updates.bash may crash, if running in
    bash version 4.3, for example as of Debian 8 Jessie, and only Office
    Updates are selected.

    While parsing the command-line parameters, the script
    download-updates.bash creates four internal lists for the needed
    updates, architectures, languages and included downloads.

    The list of architectures will be empty, if only Office updates are
    selected. This list determines the architectures of the included
    downloads .NET Frameworks, Windows Defender and Microsoft Security
    Essentials. These downloads should match the specified Windows
    versions, not the Office versions.

    There is an old bug in bash up to version 4.3: Empty arrays are
    treated as "unset", even if the array variables are declared and
    initialized. This bug is solved in bash 4.4, as of Debian 9 Stretch.

    * bash: nounset treats empty array as unset, contrary to man. page
      https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=529627

    For the script this means, that the length of the array must be
    checked, before it is read. This has already been done in most places,
    but it had to be added to the function print_command_line_summary.

    This solves the bug reports
    http://forums.wsusoffline.net/viewtopic.php?f=9&t=8072 and
    http://forums.wsusoffline.net/viewtopic.php?f=9&t=8090 .


Release Notes for Version 1.6

  Release date: 2018-05-04
  Intended compatibility: WSUS Offline Update Version 11.3 and later

  Changes in this version

  - Added support for .NET Framework 4.7.2

  - Bug fix: The function create_integrity_database did not create a
    hashes file, if the hashed directory was a symbolic link.

    To prevent the creation of empty hashdeep files, the function
    create_integrity_database counts the number of files in the hashed
    directory. This was done by using "find" and "wc", but find by
    default does not follows symbolic links. The new implementation lets
    the bash itself calculate the file count.
