# ETL / Data Pipelines

> Available S3 endpoint: `a-s3.aur.lu` (France, Albi).

## Airbyte

Open‑source data integration platform with connectors for S3 as both source and destination.

- Homepage: [Airbyte](https://airbyte.com/)
- Docs: [Airbyte Docs](https://docs.airbyte.com/)

## Apache NiFi

Low‑code data flow and ETL tool. Use S3 processors (Get/Put) with endpoint overrides for S3‑compatible storage.

- Homepage: [Apache NiFi](https://nifi.apache.org/)
- Docs: [NiFi User Guide](https://nifi.apache.org/docs.html)

```properties
nifi.properties overrides:
nifi.s3.endpoint=https://a-s3.aur.lu
nifi.s3.path.style.access=true
```

## Dagster (S3IOManager)

Modern data orchestrator with S3 IO managers for asset storage and caching.

- Homepage: [Dagster](https://dagster.io/)
- Docs: [Dagster AWS](https://docs.dagster.io/integrations/libraries/aws/s3)
```python
from dagster_aws.s3 import s3_pickle_io_manager

io_manager = s3_pickle_io_manager.configured({
    "s3_bucket": "my-bucket",
    "s3_prefix": "dagster/io",
    "endpoint_url": "https://a-s3.aur.lu",
    "aws_access_key_id": "<YOUR_ACCESS_KEY>",
    "aws_secret_access_key": "<YOUR_SECRET_KEY>",
    "s3_path_style": True,
})
```