Adding a new Datastore to FIRE

curl --request POST \
  --url http://$USERID:$PASSWORD@hx.fire.sdo.ebi.ac.uk/fire/admin/config \
  --data '{
	"section": "pag-test-int2",
	"key": "hosts",
	"value": "http://hxpwpota002.pag.ebi.ac.uk:4080",
	"valueType": "C"
}'

{
	"section": "pag-test-int2",
	"key": "vault",
	"value": "fire-test-03",
	"valueType": "S"
}

{
	"section": "pag-test-int2",
	"key": "io_block_size",
	"value": "1048576",
	"valueType": "I"
}

{
	"section": "pag-test-int2",
	"key": "ro_user",
	"value": "<RO Access Key ID>",
	"valueType": "S"
}

{
	"section": "pag-test-int2",
	"key": "ro_passwd",
	"value": "<RO Secret access key>",
	"valueType": "S"
}

{
	"section": "pag-test-int2",
	"key": "user",
	"value": "<RW Access Key ID>",
	"valueType": "S"
}

{
	"section": "pag-test-int2",
	"key": "passwd",
	"value": "<RO Secret access key>",
	"valueType": "S"
}

curl --request POST \
  --url http://$USERID:$PASSWORD@hx.fire.sdo.ebi.ac.uk/fire/admin/datastore \
  --data '{
	"name": "pag-test-int2",
	"configSection": "pag-test-int2",
	"dataStoreType": "S3_OTA",
	"dataStoreState": "NO_NEW_OBJECTS"
}'

# Adding a new Datastore to FIRE Adding a datastore to FIRE Involves two steps: * Adding datastore configs * Adding datastore to FIRE ## Adding Datastore Configs All AWS S3 compatible storage systems qualify as a "datastore" for FIRE. There are 7 elements to a datastore configuration. | No. | FIRE Configuration | Corresponding AWS name | | ------ | ------ | ----- | | 1. | hosts | endpoint | | 2. | vault | bucket | | 3. | io_block_size | - | | 4. | ro_user | Readonly Access Key ID for the bucket | | 5. | ro_passwd | Readonly Secret Access Key for the bucket | | 6. | user | Read-Write Access Key ID for the bucket | | 7. | passwd | Read-Write Secret Access Key for the bucket | Send all of these details as a POST call to `fire/admin/config` as in the example below. ### Example Datastore Configuration The 'section' in the configuration is the name of the data store in FIRE. Leave the other key values in the JSON payload as is. #### hosts ```sh curl --request POST \ --url http://$USERID:$PASSWORD@hx.fire.sdo.ebi.ac.uk/fire/admin/config \ --data '{ "section": "pag-test-int2", "key": "hosts", "value": "http://hxpwpota002.pag.ebi.ac.uk:4080", "valueType": "C" }' ``` Replace the JSON payload in the curl command to configure other elements. #### vault ```json { "section": "pag-test-int2", "key": "vault", "value": "fire-test-03", "valueType": "S" } ``` #### io_block_size ```json { "section": "pag-test-int2", "key": "io_block_size", "value": "1048576", "valueType": "I" } ``` #### ro_user ```json { "section": "pag-test-int2", "key": "ro_user", "value": "<RO Access Key ID>", "valueType": "S" } ``` #### ro_passwd ```json { "section": "pag-test-int2", "key": "ro_passwd", "value": "<RO Secret access key>", "valueType": "S" } ``` #### user ```json { "section": "pag-test-int2", "key": "user", "value": "<RW Access Key ID>", "valueType": "S" } ``` #### passwd ```json { "section": "pag-test-int2", "key": "passwd", "value": "<RO Secret access key>", "valueType": "S" } ``` Expect a "201 Created" response as a successful configuration for each element. ## Adding datastore to FIRE Once datastore configs are created, send a POST call to `fire/admin/datastore` create a datastore entry in FIRE. An example curl request is as below. ```sh curl --request POST \ --url http://$USERID:$PASSWORD@hx.fire.sdo.ebi.ac.uk/fire/admin/datastore \ --data '{ "name": "pag-test-int2", "configSection": "pag-test-int2", "dataStoreType": "S3_OTA", "dataStoreState": "NO_NEW_OBJECTS" }' ``` Here 'configSection' is the same as 'section' in the datastore configuration. 'dataStoreType' is 'S3' for storage backends that use spinning disks or SSDs, 'S3_OTA' for storage backends that use Tapes. It is recommended to keep 'dataStoreState' to 'NO_NEW_OBJECTS' as FIRE will not send objects to this backend. Once the datastore is ready to accept objects, change the state to 'READ_WRITE'.