Uploaded image for project: 'FTS'
  1. FTS
  2. FTS-1687

Add configuration to allow/disallow local streaming transfers per VO

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: fts-rest 3.9.4, fts 3.9.5
    • Fix Version/s: fts 3.10.0, fts-rest 3.10.0
    • Component/s: MySQL, REST API, Server, URL Copy
    • Security Level: Public Data (This ticket is visible to anyone on the internet and will be indexed by search engines)
    • Labels:
      None

      Description

      When FTS/Gfal2 has to perform a protocol translation (e.g.: gsiftp --> https), the transfer is done via "streaming transfer". This means Gfal2 reads data from the source endpoint and writes to the destination endpoint.

      We've seen a couple of these transfers take up a lot of resources on the FTS3-Atlas instance. More so, Atlas stated these transfers shouldn't happen anymore and wanted FTS to fail any such transfer request (FAILED jobs immediately raise red flags).

      Gfal2 already provides a flag to enable/disable local streaming transfers:

      gfalt_params_t params->local_transfers = <true/false>
      
      // gfal_transfer_filecopy.cc
      static int perform_copy(..) {
        ...
          if (plugin == NULL) {
                  if (gfalt_get_local_transfer_perm(params, NULL)) {
                      res = perform_local_copy(context, params, src, dst, &tmp_err);
                  }
                  else {
                      gfal2_set_error(error, scope_copy_domain(), EPROTONOSUPPORT, __func__,
                              "No plugin supports a transfer from %s to %s, and local streaming is disabled",
                              src, dst);
                      res = -1;
                  }
              }
        ...
      }
      

      The proposal is to introduce a server-wide configuration field by VO.
      When set to true, allow local streaming transfers.
      When set to false, disable local streaming transfers.

      This would also meet Atlas requirement to permanently disable steaming transfers.
      In case they end up submitting again, FTS will fail those jobs.

        Attachments

          Activity

            People

            Assignee:
            mipatras Mihai Patrascoiu
            Reporter:
            mipatras Mihai Patrascoiu
            Component Watchers:
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: