# Machine Learning & Data Science

Aurora works with common ML tooling via S3-compatible URLs.

> Available S3 endpoint: `a-s3.aur.lu` (France, Albi).

## Hugging Face Datasets (fsspec/s3fs)

High‑level dataset loader with out‑of‑the‑box support for S3 via fsspec/s3fs. Ideal for large, streaming datasets stored in Parquet/JSON on S3‑compatible storage.

- Homepage: [Hugging Face Datasets](https://huggingface.co/docs/datasets)
- fsspec: [Docs](https://filesystem-spec.readthedocs.io/)
- s3fs: [Docs](https://s3fs.readthedocs.io/)
```python
from datasets import load_dataset

storage_options = {
  "key": "<YOUR_ACCESS_KEY>",
  "secret": "<YOUR_SECRET_KEY>",
  "client_kwargs": {"endpoint_url": "https://a-s3.aur.lu"},
}

ds = load_dataset(
  "json",
  data_files="s3://my-bucket/datasets/*.json",
  storage_options=storage_options,
)
print(ds)
```

## DVC (Data Version Control)

Data versioning and remote storage for ML projects. Track and push datasets and models to S3 remotes, enabling reproducible pipelines and CI.

- Homepage: [DVC](https://dvc.org/)
- Docs: [DVC Docs](https://dvc.org/doc)
```bash
dvc remote add -d aurora s3://my-bucket/my-project
dvc remote modify aurora endpointurl https://a-s3.aur.lu
dvc remote modify aurora access_key_id <YOUR_ACCESS_KEY>
dvc remote modify aurora secret_access_key <YOUR_SECRET_KEY>
dvc push
```

## Feast (Feature Store)

Open‑source feature store. Use S3 as an offline store for batch features, interoperating with your data lake and training pipelines.

- Homepage: [Feast](https://feast.dev/)
- Docs: [Feast Docs](https://docs.feast.dev/)

```yaml title="feature_store.yaml"
project: my_project
offline_store:
  type: file
  location: s3://my-bucket/feast/offline
```