• Adding a new Datastore to FIRE

    Adding a datastore to FIRE Involves two steps:

    • Adding datastore configs
    • Adding datastore to FIRE

    Adding Datastore Configs

    All AWS S3 compatible storage systems qualify as a "datastore" for FIRE. There are 7 elements to a datastore configuration.

    No. FIRE Configuration Corresponding AWS name
    1. hosts endpoint
    2. vault bucket
    3. io_block_size -
    4. ro_user Readonly Access Key ID for the bucket
    5. ro_passwd Readonly Secret Access Key for the bucket
    6. user Read-Write Access Key ID for the bucket
    7. passwd Read-Write Secret Access Key for the bucket

    Send all of these details as a POST call to fire/admin/config as in the example below.

    Example Datastore Configuration

    The 'section' in the configuration is the name of the data store in FIRE. Leave the other key values in the JSON payload as is.

    hosts

    curl --request POST \
      --url http://$USERID:$PASSWORD@hx.fire.sdo.ebi.ac.uk/fire/admin/config \
      --data '{
    	"section": "pag-test-int2",
    	"key": "hosts",
    	"value": "http://hxpwpota002.pag.ebi.ac.uk:4080",
    	"valueType": "C"
    }'

    Replace the JSON payload in the curl command to configure other elements.

    vault

    {
    	"section": "pag-test-int2",
    	"key": "vault",
    	"value": "fire-test-03",
    	"valueType": "S"
    }

    io_block_size

    {
    	"section": "pag-test-int2",
    	"key": "io_block_size",
    	"value": "1048576",
    	"valueType": "I"
    }

    ro_user

    {
    	"section": "pag-test-int2",
    	"key": "ro_user",
    	"value": "<RO Access Key ID>",
    	"valueType": "S"
    }

    ro_passwd

    {
    	"section": "pag-test-int2",
    	"key": "ro_passwd",
    	"value": "<RO Secret access key>",
    	"valueType": "S"
    }

    user

    {
    	"section": "pag-test-int2",
    	"key": "user",
    	"value": "<RW Access Key ID>",
    	"valueType": "S"
    }

    passwd

    {
    	"section": "pag-test-int2",
    	"key": "passwd",
    	"value": "<RO Secret access key>",
    	"valueType": "S"
    }

    Expect a "201 Created" response as a successful configuration for each element.

    Adding datastore to FIRE

    Once datastore configs are created, send a POST call to fire/admin/datastore create a datastore entry in FIRE. An example curl request is as below.

    curl --request POST \
      --url http://$USERID:$PASSWORD@hx.fire.sdo.ebi.ac.uk/fire/admin/datastore \
      --data '{
    	"name": "pag-test-int2",
    	"configSection": "pag-test-int2",
    	"dataStoreType": "S3_OTA",
    	"dataStoreState": "NO_NEW_OBJECTS"
    }'

    Here 'configSection' is the same as 'section' in the datastore configuration. 'dataStoreType' is 'S3' for storage backends that use spinning disks or SSDs, 'S3_OTA' for storage backends that use Tapes.
    It is recommended to keep 'dataStoreState' to 'NO_NEW_OBJECTS' as FIRE will not send objects to this backend. Once the datastore is ready to accept objects, change the state to 'READ_WRITE'.

Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment