ETL / Data Pipelines
Select S3 Endpoint
Choose preferred region.
Airbyte
Open‑source data integration platform with connectors for S3 as both source and destination.
- Homepage: Airbyte
- Docs: Airbyte Docs
Apache NiFi
Low‑code data flow and ETL tool. Use S3 processors (Get/Put) with endpoint overrides for S3‑compatible storage.
- Homepage: Apache NiFi
- Docs: NiFi User Guide
nifi.properties overrides:
nifi.s3.endpoint=https://
nifi.s3.path.style.access=true
Dagster (S3IOManager)
Modern data orchestrator with S3 IO managers for asset storage and caching.
- Homepage: Dagster
- Docs: Dagster AWS
from dagster_aws.s3 import s3_pickle_io_manager
io_manager = s3_pickle_io_manager.configured({
"s3_bucket": "my-bucket",
"s3_prefix": "dagster/io",
"endpoint_url": "https://",
"aws_access_key_id": "<YOUR_ACCESS_KEY>",
"aws_secret_access_key": "<YOUR_SECRET_KEY>",
"s3_path_style": True,
})