Uploaded image for project: 'DMC - Development'
  1. DMC - Development
  2. DMC-1112

Setting BEARER_TOKEN in gfal-copy breaks the redir with a pre-signed URL

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: gfal 2.16.0
    • Fix Version/s: gfal 2.16.1
    • Component/s: gfal 2
    • Security Level: Public Data (This ticket is visible to anyone on the internet and will be indexed by search engines)
    • Labels:
      None

      Description

      ---------------------------------------------------------------------------------------

      ------ With the variable it does not work

      ------ Please note that the site giving the error is CephS3, with the presigned URL

      ------ I take the occasion to remark that this mechanism with the envvar BEARER_TOKEN

      ------ is inadequate, at best

       

      $ export BEARER_TOKEN="MDAyYWxvY2F0aW9uIC9kcG0vY2Vybi5jaC9ob21lL2R0ZWFtLzRHLTgKMDAxNmlkZW50aWZpZXIgY29uZmlnCjAwNWJjaWQgZG46L0RDPWNoL0RDPWNlcm4vT1U9T3JnYW5pYyBVbml0cy9PVT1Vc2Vycy9DTj1mdXJhbm8vQ049NjQ0NzQ2L0NOPUZhYnJpemlvIEZ1cmFubwowMDJlY2lkIGZxYW46L2R0ZWFtL1JvbGU9TlVMTC9DYXBhYmlsaXR5PU5VTEwKMDAyYWNpZCBwYXRoOi9kcG0vY2Vybi5jaC9ob21lL2R0ZWFtLzRHLTgKMDAyZnNpZ25hdHVyZSCQk3WXmm5KfzrEXVfkWEh4-_Evzu_K06EglmaiXId6hwo"

      $ gfal-copy -vv --just-copy --cert '/var/www/proxycache/%3A%2FDC%3Dch%2FDC%3Dcern%2FOU%3DOrganic%2BUnits%2FOU%3DUsers%2FCN%3Dfurano%2FCN%3D644746%2FCN%3DFabrizio%2BFurano/1ae0808cb0e59d29/userproxy.pem' --key '/var/www/proxycache/%3A%2FDC%3Dch%2FDC%3Dcern%2FOU%3DOrganic%2BUnits%2FOU%3DUsers%2FCN%3Dfurano%2FCN%3D644746%2FCN%3DFabrizio%2BFurano/1ae0808cb0e59d29/userproxy.pem' 'http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host' 'https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8'
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success
      Copying 0 bytes http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      event: [1540217073224] BOTH GFAL2:CORE:COPY LIST:ENTER
      INFO Event triggered: BOTH GFAL2:CORE:COPY LIST:ENTER
      event: [1540217073225] BOTH GFAL2:CORE:COPY LIST:ITEM http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Event triggered: BOTH GFAL2:CORE:COPY LIST:ITEM http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      event: [1540217073225] BOTH GFAL2:CORE:COPY LIST:EXIT
      INFO Event triggered: BOTH GFAL2:CORE:COPY LIST:EXIT
      event: [1540217073225] BOTH http_plugin PREPARE:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Event triggered: BOTH http_plugin PREPARE:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      event: [1540217073225] BOTH http_plugin PREPARE:EXIT http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Event triggered: BOTH http_plugin PREPARE:EXIT http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      event: [1540217073225] BOTH http_plugin TRANSFER:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Event triggered: BOTH http_plugin TRANSFER:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Trying copying with mode streamed
      event: [1540217073225] BOTH http_plugin TRANSFER:TYPE streamed
      INFO Event triggered: BOTH http_plugin TRANSFER:TYPE streamed
      INFO Performing a HTTP streamed copy
      INFO Davix: Operation failure: HTTP 400 : Server Error . After 1 retry
      INFO Davix: Failure: Impossible to execute operation on http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host, error Failure HTTP 400 : Server Error after 1 attempts
      INFO Davix: Try to Recover with Metalink...
      INFO Davix: Impossible to Recover with Metalink: Server does not support Metalink standard
      event: [1540217073241] BOTH http_plugin TRANSFER:EXIT ERROR: Copy failed with mode streamed, will delete destination and retry with the next available mode: Failure HTTP 400 : Server Error after 1 attempts
      INFO Event triggered: BOTH http_plugin TRANSFER:EXIT ERROR: Copy failed with mode streamed, will delete destination and retry with the next available mode: Failure HTTP 400 : Server Error after 1 attempts
      INFO Davix: Operation failure: HTTP 404 : File not found . After 1 retry
      event: [1540217073295] BOTH http_plugin TRANSFER:EXIT http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Event triggered: BOTH http_plugin TRANSFER:EXIT http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-8
      INFO Davix: Operation failure: HTTP 404 : File not found . After 1 retry
      gfal-copy error: 112 (Host is down) - TRANSFER Failure HTTP 400 : Server Error after 1 attempts

      ---------------------------------------------------------------------------------------

      ---------------------- With no variable it wors

      $ gfal-copy -vv --just-copy --cert '/var/www/proxycache/%3A%2FDC%3Dch%2FDC%3Dcern%2FOU%3DOrganic%2BUnits%2FOU%3DUsers%2FCN%3Dfurano%2FCN%3D644746%2FCN%3DFabrizio%2BFurano/1ae0808cb0e59d29/userproxy.pem' --key '/var/www/proxycache/%3A%2FDC%3Dch%2FDC%3Dcern%2FOU%3DOrganic%2BUnits%2FOU%3DUsers%2FCN%3Dfurano%2FCN%3D644746%2FCN%3DFabrizio%2BFurano/1ae0808cb0e59d29/userproxy.pem' 'http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host' 'https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7'
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success
      INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success
      Copying 0 bytes http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      event: [1540216837453] BOTH GFAL2:CORE:COPY LIST:ENTER
      INFO Event triggered: BOTH GFAL2:CORE:COPY LIST:ENTER
      event: [1540216837454] BOTH GFAL2:CORE:COPY LIST:ITEM http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      INFO Event triggered: BOTH GFAL2:CORE:COPY LIST:ITEM http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      event: [1540216837454] BOTH GFAL2:CORE:COPY LIST:EXIT
      INFO Event triggered: BOTH GFAL2:CORE:COPY LIST:EXIT
      event: [1540216837454] BOTH http_plugin PREPARE:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      INFO Event triggered: BOTH http_plugin PREPARE:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      event: [1540216837454] BOTH http_plugin PREPARE:EXIT http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      INFO Event triggered: BOTH http_plugin PREPARE:EXIT http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      event: [1540216837454] BOTH http_plugin TRANSFER:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      INFO Event triggered: BOTH http_plugin TRANSFER:ENTER http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host => https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7
      INFO Trying copying with mode streamed
      event: [1540216837454] BOTH http_plugin TRANSFER:TYPE streamed
      INFO Event triggered: BOTH http_plugin TRANSFER:TYPE streamed
      INFO Performing a HTTP streamed copy
      monitor: http://fab-bucket2.s3.cern.ch/4G?X-Amz-Signature=1f476caf808023ec93d6a38811fac643d7ad12371e87492c707525e7d6f561ab&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q6BLGPLOT5FONJACN2B1%2F20181022%2Fcern%2Fs3%2Faws4_request&X-Amz-Date=20181022T133530Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/4G-7 28154265 18446744072169334779 140771328 5

       

        Attachments

          Activity

            People

            • Assignee:
              amanzi Andrea Manzi
              Reporter:
              furano Fabrizio Furano
              Component Watchers:
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Actual End: