Update source table for a next execution
Once a source has finished processing successfully, the system should update the Source table with the necessary changes to prepare it for the next execution. For example, suppose we are running the UNICHEM process for the sources ChEMBL and ChEBI, and the current state of the Source table is as follows:
| id | name | official_name | last_released_version | download_url |
|---|---|---|---|---|
| 1 | chembl | ChEMBL | 34 | /Users/carlosm/Documents/UNICHEM/unichem-loader/tests/mock_cluster_folder/chembl_34.sdf |
| 7 | chebi | ChEBI (Chemical Entities of Biological Interest). | 221 | /Users/carlosm/Documents/UNICHEM/unichem-loader/tests/mock_cluster_folder/chebi_lite.sdf.gz |
By checking the download_url or last_released_version column, we can determine that the current versions being processed are ChEMBL_34 and ChEBI 221. If the UNICHEM process completes successfully for both sources, then the next execution should target ChEMBL_35 and ChEBI 222. Therefore, the Source table must be updated accordingly:
| id | name | official_name | last_released_version | download_url |
|---|---|---|---|---|
| 1 | chembl | ChEMBL | 35 | /Users/carlosm/Documents/UNICHEM/unichem-loader/tests/mock_cluster_folder/chembl_35.sdf |
| 7 | chebi | ChEBI (Chemical Entities of Biological Interest). | 222 | /Users/carlosm/Documents/UNICHEM/unichem-loader/tests/mock_cluster_folder/chebi_lite.sdf.gz |
Depending on the source, will be update one or two columns with the necessary information for the next execution.