10.2. Accessing Object Storage

Internal networking

On internal netwoks it might be that requests are always routed outwards. You can bypass this with:

export no_proxy=$no_proxy,cloud.science-it.uzh.ch
export NO_PROXY=$NO_PROXY,cloud.science-it.uzh.ch

Manual Inspection

openstack catalog show swift
# Generate a temporary access token
openstack token issue

Automated Extraction

export RGW_URL=$(openstack catalog show swift -f json -c endpoints \
  | jq --raw-output '.endpoints[] | select(.interface=="public") | .url' \
  | head -n 1)
export RGW_TOKEN=$(openstack token issue -f value -c id)

Native Swift API (curl)

# List containers (buckets)
curl -i -X GET "${RGW_URL}" \
  -H "X-Auth-Token: ${RGW_TOKEN}"

# Upload an object
curl -i -X PUT "${RGW_URL}/bucket/file.txt" \
  -H "X-Auth-Token: ${RGW_TOKEN}" \
  --data-binary "@./file.txt"

Token Expiration

Keystone authentication tokens are temporary (typically expiring after 1 hour).

S3 Compatibility API (s3cmd)

Utilizes the S3-compatible interface, requiring EC2 credentials.

# Generate S3-compatible credentials
ACCESS_KEY=$(openstack ec2 credentials create \
  -f value -c access)
SECRET_KEY=$(openstack ec2 credentials show \
  "${ACCESS_KEY}" -f value -c secret)

Configuration (~/.s3cfg):

[default]
host_base = ${RGW_HOST}
host_bucket = ${RGW_HOST}
access_key = ${ACCESS_KEY}
secret_key = ${SECRET_KEY}
use_https = True

Configuration Overhead Mitigation

To circumvent manual credential generation and configuration file management, the entire authentication and execution sequence can be encapsulated within an Apptainer orchestration image: GitHub: pSciComp/s3cmdContainer