Skip to main content

Machine Learning & Data Science

Aurora works with common ML tooling via S3-compatible URLs.

Select S3 Endpoint
Choose preferred region.

Hugging Face Datasets (fsspec/s3fs)​

High‑level dataset loader with out‑of‑the‑box support for S3 via fsspec/s3fs. Ideal for large, streaming datasets stored in Parquet/JSON on S3‑compatible storage.

from datasets import load_dataset

storage_options = {
"key": "<YOUR_ACCESS_KEY>",
"secret": "<YOUR_SECRET_KEY>",
"client_kwargs": {"endpoint_url": "https://"},
}

ds = load_dataset(
"json",
data_files="s3://my-bucket/datasets/*.json",
storage_options=storage_options,
)
print(ds)

DVC (Data Version Control)​

Data versioning and remote storage for ML projects. Track and push datasets and models to S3 remotes, enabling reproducible pipelines and CI.

dvc remote add -d aurora s3://my-bucket/my-project
dvc remote modify aurora endpointurl https://
dvc remote modify aurora access_key_id <YOUR_ACCESS_KEY>
dvc remote modify aurora secret_access_key <YOUR_SECRET_KEY>
dvc push

Feast (Feature Store)​

Open‑source feature store. Use S3 as an offline store for batch features, interoperating with your data lake and training pipelines.

feature_store.yaml
project: my_project
offline_store:
type: file
location: s3://my-bucket/feast/offline