Skip to main content

Computation Checkpointing & Result Storage

Select S3 Endpoint
Choose preferred region.

Ray (checkpoints)​

Distributed compute framework for Python/AI workloads. Store training checkpoints and artifacts on S3‑compatible storage for resilience and portability.

import ray

ray.init()
uri = "s3://my-bucket/ray-checkpoints"
options = {
"aws_access_key_id": "<YOUR_ACCESS_KEY>",
"aws_secret_access_key": "<YOUR_SECRET_KEY>",
"endpoint_url": "https://",
}
# Use uri/options in your trainers or custom checkpoint logic

Snakemake (remote S3)​

Workflow engine for reproducible science. The S3 remote provider enables cloud storage of intermediate and final outputs.

Snakefile
from snakemake.remote.S3 import RemoteProvider as S3RemoteProvider
S3 = S3RemoteProvider(host="", secure=True)

rule all:
  input: S3.remote("s3://my-bucket/pipeline/output.txt")

Prefect 2.x​

Orchestration platform for data/ML with blocks for S3‑backed flow and result storage.

from prefect_aws.s3 import S3Bucket

block = S3Bucket(
  bucket_name="prefect-results",
  endpoint_url="https://",
  aws_access_key_id="<YOUR_ACCESS_KEY>",
  aws_secret_access_key="<YOUR_SECRET_KEY>",
)
block.save("aurora-s3")