[DMC-880] If the option -t is passed, make sure that Davix knows it Created: 05/Sep/16  Updated: 06/Feb/17  Resolved: 12/Sep/16

Status: Closed
Project: DMC - Development
Component/s: gfal 2, gfal2 python , gfal2-util
Affects Version/s: gfal 2.12.1
Fix Version/s: gfal2-util 1.5.0, gfal 2.13
Security Level: Public Data (This ticket is visible to anyone on the internet and will be indexed by search engines)

Type: Improvement Priority: Minor
Reporter: Fabrizio Furano Assignee: Alejandro Alvarez Ayllon
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Component Watchers:

 Description   

When using gfal with davs: the connection/operation/redirection timeouts of Davix are left to the default, which is very low. They should be set to the value given with the -t option



 Comments   
Comment by Alejandro Alvarez Ayllon [ 06/Sep/16 ]

-t is used only by gfal2-util itself. It triggers a thread that will kill gfal2-util if gfal2 does not return in time.
It is very misleading indeed, because one would expect this to increase internal timeouts too...

There is only a function to set timeouts, and it is related to copies only, which is a separate, although related, API.
Checksums timeout in GridFTP are inside the GridFTP configuration.

Need to think about how to handle timeouts.

Comment by Alejandro Alvarez Ayllon [ 12/Sep/16 ]

So we need

1. An API for setting timeouts on gfal2
2. Using these timeouts by the plugins
3. Exposing this API in gfal2-python
4. Using the API in gfal2-util

Step 2 is the tricky one, because we already have timeouts as configuration values for some protocols/operations :/

Comment by Alejandro Alvarez Ayllon [ 12/Sep/16 ]

Existing timeouts

  • SRM
    • OPERATION_TIMEOUT
    • CONN_TIMEOUT
  • LFC
    • LFC_CONNTIMEOUT
  • GridFTP
    • CHECKSUM_CALC_TIMEOUT
    • OPERATION_TIMEOUT

PERF_MARKER_TIMEOUT we can ignore for this.

CHECKSUM_CALC_TIMEOUT is the most sensitive one, since it is relevant for FTS.

So let's add an operation timeout that defaults to the protocol specific one if not set, and document that they are deprecated.

Comment by Clockwork Droid [ 12/Sep/16 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of dmc/gfal2:
'DMC-880: Use global timeouts if not overriden'

Comment by Clockwork Droid [ 12/Sep/16 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of dmc/gfal2:
'DMC-880: Add timeouts to http plugin'

Comment by Alejandro Alvarez Ayllon [ 12/Sep/16 ]

No need to patch gfal2-python.

[CORE]
NAMESPACE_TIMEOUT
CHECKSUM_TIMEOUT

Protocol specific will override this, to keep backwards compatibility.

Comment by Clockwork Droid [ 12/Sep/16 ]

FAILURE: Integrated in Jenkins build (devel) gfal2 #161 (See https://jenkins.cern.ch/fts-dmc/job/(devel)%20gfal2/161/)
DMC-880: Use global timeouts if not overriden (alejandro.alvarez.ayllon: rev 88aae63aec79f376a3dfe1c9a48dc906d84b4726)
DMC-880: Add timeouts to http plugin (alejandro.alvarez.ayllon: rev 7de85de091a256381e5973c49c1c1be9d21bda2b)

Comment by GitLab service [ 12/Sep/16 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of dmc/gfal2-util:
'DMC-880: Pass timeouts to gfal2'

Comment by Clockwork Droid [ 12/Sep/16 ]

SUCCESS: Integrated in Jenkins build (devel) gfal2-util #48 (See https://jenkins.cern.ch/fts-dmc/job/(devel)%20gfal2-util/48/)
DMC-880: Pass timeouts to gfal2 (alejandro.alvarez.ayllon: rev 3a91f76d2795758bf281b4e205e93c68e1661325)

Comment by Burt Holzman [ 22/Sep/16 ]

We are running bleeding-edge code on our FTS server to do transfers to Google via http.

There's no protocol-specific override for the HTTP plugin AFAICT. I think this means that any http stream transfer that exceeds the NAMESPACE_TIMEOUT will fail (even if the actual transfer succeeds, since it doesn't check the timeout until after the return). The default NAMESPACE_TIMEOUT is set to 300s, which is pretty low. The FTS server itself also set -t 4000, but that doesn't seem to propagate.

Comment by GitLab service [ 27/Sep/16 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of ellert/gfal2:
'DMC-880: Use global timeouts if not overriden'

Comment by GitLab service [ 27/Sep/16 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of ellert/gfal2:
'DMC-880: Add timeouts to http plugin'

Comment by Clockwork Droid [ 06/Feb/17 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of dmc/gfal2:
'DMC-880: Use global timeouts if not overriden'

Comment by Clockwork Droid [ 06/Feb/17 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of dmc/gfal2:
'DMC-880: Add timeouts to http plugin'

Comment by GitLab service [ 06/Feb/17 ]

Alejandro Alvarez Ayllon mentioned this issue in a commit of dmc/gfal2-util:
'DMC-880: Pass timeouts to gfal2'

Generated at Wed Jun 28 21:19:41 CEST 2017 using JIRA 7.2.7#72009-sha1:68b7d86b7e8716f76c1f28a5095acccf67bb75b1.