iRODS S3 API:
Presenting iRODS as S3
June 13-16, 2023
iRODS User Group Meeting 2023
Chapel Hill, NC
Terrell Russell, Ph.D.
@terrellrussell
Executive Director, iRODS Consortium
Overview
History - Desire / Justification
History - Desire / Justification
History - Desire / Justification
History - Desire / Justification
1. Update and maintain https://github.com/bioteam/minio-irods-gateway
This minio-based front end already exists and has been demonstrated, but has not been updated since its debut. I do not know the extent that it has been used in production work, perhaps John can talk about that. This uses the GoRODS client library, which will continue to need to be updated to wrap the latest iRODS C API as new releases of the server come out. Presumably, BioTeam / John would continue to own/maintain GoRODS and the minio-irods-gateway.
2. minio-irods-gateway converts to use https://github.com/cyverse/go-irodsclient
Illyoung has produced and is actively developing a pure Go iRODS client library. This new library could be 'swapped' for the GoRODS calls in the minio-irods-gateway. Presumably, then Arizona / Illyoung would own/maintain a fork of the minio-irods-gateway.
3. Add irods/gateway-irods.go to upstream https://github.com/minio/minio/tree/master/cmd/gateway
Someone / all of us would port / implement the same work from Option 2 (convert to pure Go), but work with the minio community to get iRODS to be an officially supported gateway for the MinIO server directly.
4. New C++ implementation - https://github.com/irods/irods_client_s3_cpp
The iRODS Consortium would work to implement the S3 specification directly with a new C++ client. This could be more performant (in the long run), but requires the most work and answers to open questions.
"I am leaning towards Option 3 as the best option both from a cost/benefit perspective, as well as exposure to a larger community and confidence that 'it just works'."
History - Research
History - Research - Phase 1
History - Research - Phase 2
History - Research - Phase 2 - Alternate Illyoung Universes
History - Research - Phase 3 (Feb-Mar 2023)
History - Research - Phase 4 (June 2023)
Status / Implementation - Architecture
Status / Implementation - Architecture
Status / Implementation - Configuration
Not Yet Real...
{
// Defines S3 options that affect how the
// client-facing component of the server behaves.
"s3_server": {
// ...
},
// Defines iRODS connection information.
"irods_client": {
// ...
}
}
Status / Implementation - Configuration - s3_server
"s3_server": {
"host": "0.0.0.0",
"port": 80,
"log_level": "warn",
"bucket_mapping": {
# local / static (JSON)
# external / dynamic - iRODS or third party API
},
"user_mapping": {
# local / static (JSON)
# external / dynamic - iRODS or third party API
},
"threads": 3
}
}
Status / Implementation - Configuration - irods_client
"irods_client": {
"host": "<string>",
"port": 1247,
"zone": "<zone>",
"proxy_rodsadmin": {
"username": "<string>",
"password": "<string>"
},
"connection_pool": {
"size": 6,
"refresh_timeout_in_seconds": 600
},
}
Next Steps
Questions?
Thank you.