Request minimum time delta between bringonline and expiration timeout at submission time

Given the recent bug discovered in FTS-1724, the FTS-REST server should request a minimum configurable time delta between the user submitted bringonline timeout and expiration timeout (max_time_in_queue field).

Considerations

• The expiration timeout only applies to files found in an queued state, namely STAGING or SUBMITTED
• Once a file is brought online, it moves from STARTED to SUBMITTED file state
• Having an expiration timeout lower than the bringonline timeout leaves the file vulnerable to a race condition once it is brought online:
  • It may be picked up first by the Scheduler thread, in which case it becomes ACTIVE
  • Or it may be picked up by the SanityCheck thread, in which case it becomes CANCELED

Proposal

To avoid this scenario, the FTS-REST server should ensure a configurable time delta is respected between the bringonline timeout and the expiration timeout. The formula would look like:

max_time_in_queue(seconds) - bringonline_timeout(seconds) >= delta(seconds)

The configurable delta will be a server config parameter, such as

# Time delta between bringonline timeout and expiration timeout
# Value must be set with a suffix: s (seconds), m (minutes), h (hours), d (days)
# No enforcement if config setting is missing
BringOnlineAndExpirationDelta=3d