GitLab

* Inventory: Filter bulk records by update dates (MKH-537)

# Conflicts:
#	harvester/src/main/java/com/indexdata/masterkey/localindices/harvest/storage/folioinventory/InventoryRecordUpdater.java

  - not just for deletions, for all requests to `shared-inventory-upsert-matchkey`

  i.e. query=transformation.id=1234 or query=step.id=1234

  For the Harvester FOLIO wrapper project (mod-harvester-admin). To enable
  querying for values with spaces, parenthesis etc.

   Transformation-step association references non-existing step,
   making the "get tsas" REST request fail.

* Set bulk-file-from-date to most recent job's start-time MKH-534

  When only getting new files, don't use the most recent  job's finish time since new files may have arrived between the start and the finish.

* Fixes for issues with binary MARC - RIS-152

  - In case of a MarcException, attempt to continue reading
    stream if continue-on-errors is on.
  - Use MarcStreamReader instead of MarcPermissiveStreamReader
    for iterating MARC file.
  - Inject MarcPermissiveStreamReader in the process for reading
    AND converting single records to UTF-8.
  - Always keep the original MARC as a byte array (since that is
    created always anyway as input for the permissive reader.
  - Remove option for non-buffering input stream since the code as
    it's written will always apply buffering anyway.

* Downgrade CompletePendingCommand timeout error log to warn log

   MarcReader had problems finding record and/or field terminators
   for some records coming in in binary MARC files.

   Seems it was due to one transformation too many. With this commit
   it would no longer make a MARC string of the MARC byte array
   before serving it to MarcToJson.convertMarcRecordsToJson() as a
   ByteArrayInputStream.

  Storing the original MARC failed because it's not XML,
  now this condition is caught, certain special characters
  are stripped and the record is stored as a string.

   Remove deprecated APIs, add upsert APIs instead

  - when the transformation pipeline don't produce a `holdingsRecord`
    property, forward none -- rather than an empty `holdingsRecord`
    property

   - using MARC4J's MarcPermissiveStreamReader,
     with the `permissive` option set to true.

  - properly reconnect after ftp connection time outs between large files
  - XmlMarcClient sends deletes by record instead of by ID to storage
  - XmlMarcClient detects deletes by LEADER 05 (status) instead of 06 (type)
  - stop printing each binary MARC file to system.out
  - remove un/pw from logged ftp URLs

  Also fixes name of holdings records metrics (should be HOLDING_RECORD now)

* Crude failed-records web service
* Write records with errors to files in failed-records directory and serve them up with web service

Marcftp