Uploaded image for project: 'FTS'
  1. FTS
  2. FTS-1687

Add configuration to allow/disallow local streaming transfers per VO



    • Task
    • Status: Closed
    • Medium
    • Resolution: Fixed
    • fts-rest 3.9.4, fts 3.9.5
    • fts 3.10.0, fts-rest 3.10.0
    • MySQL, REST API, Server, URL Copy
    • Security Level: Public Data (This ticket is visible to anyone on the internet and will be indexed by search engines)
    • None


      When FTS/Gfal2 has to perform a protocol translation (e.g.: gsiftp --> https), the transfer is done via "streaming transfer". This means Gfal2 reads data from the source endpoint and writes to the destination endpoint.

      We've seen a couple of these transfers take up a lot of resources on the FTS3-Atlas instance. More so, Atlas stated these transfers shouldn't happen anymore and wanted FTS to fail any such transfer request (FAILED jobs immediately raise red flags).

      Gfal2 already provides a flag to enable/disable local streaming transfers:

      gfalt_params_t params->local_transfers = <true/false>
      // gfal_transfer_filecopy.cc
      static int perform_copy(..) {
          if (plugin == NULL) {
                  if (gfalt_get_local_transfer_perm(params, NULL)) {
                      res = perform_local_copy(context, params, src, dst, &tmp_err);
                  else {
                      gfal2_set_error(error, scope_copy_domain(), EPROTONOSUPPORT, __func__,
                              "No plugin supports a transfer from %s to %s, and local streaming is disabled",
                              src, dst);
                      res = -1;

      The proposal is to introduce a server-wide configuration field by VO.
      When set to true, allow local streaming transfers.
      When set to false, disable local streaming transfers.

      This would also meet Atlas requirement to permanently disable steaming transfers.
      In case they end up submitting again, FTS will fail those jobs.




            mipatras Mihai Patrascoiu
            mipatras Mihai Patrascoiu
            0 Vote for this issue
            3 Start watching this issue