Script/workflow to migrate a list of project to from a space to another

changed milestone to %V3.2.6

After discussion:

this process should be a workflow.
The retention dates (runs and analyzes) have to be updated according to the new space_id.
We have to be able to launch this process from the project view.

mentioned in commit 2e82534c

mentioned in commit f9366424

closed

add logging to be able to easily find what was done if an error occurs during the process : which runs were modified, which files were transferred...

how does the new retention date set? please give examples

reopened

mentioned in commit a1c47bf3

Ok, so I found a bug while making those tests. Everything is fixed now, and here are the explanations:

To sumurize the algorithm:

- The workflow requires a project id and the new space_id
- We check if the given space_id exists in application.properties, and if the project doesn't already belong to the space_id
- From thoses informations, we can compute the path to the new directories
- Then, a very similar process will be applied to the project's analyzes, runs, and run's analyzes:
	Compute old and new directory
	Update directory in DB
	Move files
	Delete the old empty directory
	Compute the new retention
		if the given retention is a date : simple, just set a date
		if given retention is a duration : analysis or run's crdate + retention duration
	Save the new retention in DB

-----------------------------------------------------------------------------------------------------------------------

application.properties :

[space_default]
dir_name=data
retention_period=3


[space_seqoccin]
dir_name=data_seqoccin
retention_date=12/03/2021

-----------------------------------------------------------------------------------------------------------------------
Project 'purge_mail_test' (id 1292) space_id = 'seqoccin'
-----------------------------------------------------------------------------------------------------------------------

-> Analysis 57766
directory = /data_seqoccin/analyze/70205e13f
crdate = 1571044081 (Monday 14 October 2019 09:08:01)
retention_date = 1615503600 (Thursday 11 March 2021 23:00:00)

ls /work/ng6-test/jflow_rtherville/data_seqoccin/analyze/70205e13f
test.txt


-> Run 15059
directory = /data_seqoccin/run/fbc386b05
crdate = 1571044117 (Monday 14 October 2019 09:08:37)
retention_date = 1615503600 (Thursday 11 March 2021 23:00:00)

ls /work/ng6-test/jflow_rtherville/data_seqoccin/run/fbc386b05
md5sum.txt  MT_rep1_1_Ch6.fastq.gz  MT_rep1_2_Ch6.fastq.gz


	-> Analysis 57767
	directory = /data_seqoccin/analyze/b8abf8aa0
	crdate = 1571044237 (Monday 14 October 2019 09:10:37)
	retention_date = 1615503600 (Thursday 11 March 2021 23:00:00)
	
	ls /work/ng6-test/jflow_rtherville/data_seqoccin/analyze/b8abf8aa0
	test.txt

-----------------------------------------------------------------------------------------------------------------------
python3.4 /save/ng6-test/src/ng6-git-rtherville/bin/ng6_cli.py switchprojectspaceid --project-id 1292 --space-id default --admin-login CTD

Project 'purge_mail_test' (id 1292) new space_id = 'default'
-----------------------------------------------------------------------------------------------------------------------

-> Analysis 57766
directory = /data/analyze/70205e13f
crdate = 1571044081 (Monday 14 October 2019 09:08:01)
retention_date = 1578996481 (Tuesday 14 January 2020 10:08:01)

ls /work/ng6-test/jflow_rtherville/data/analyze/70205e13f
test.txt


-> Run 15059
directory = /data/run/fbc386b05
crdate = 1571044117 (Monday 14 October 2019 09:08:37)
retention_date = 1578996517 (Tuesday 14 January 2020 10:08:37)

ls /work/ng6-test/jflow_rtherville/data/run/fbc386b05
md5sum.txt  MT_rep1_1_Ch6.fastq.gz  MT_rep1_2_Ch6.fastq.gz


	-> Analysis 57767
	directory = /data/analyze/b8abf8aa0
	crdate = 1571044237 (Monday 14 October 2019 09:10:37)
	retention_date = 1578996637 (Tuesday 14 January 2020 10:10:37)
	
	ls /work/ng6-test/jflow_rtherville/data/analyze/b8abf8aa0
	test.txt
	
-----------------------------------------------------------------------------------------------------------------------
python3.4 /save/ng6-test/src/ng6-git-rtherville/bin/ng6_cli.py switchprojectspaceid --project-id 1292 --space-id seqoccin --admin-login CTD

Project 'purge_mail_test' (id 1292) new space_id = 'seqoccin'
-----------------------------------------------------------------------------------------------------------------------

-> Analysis 57766
directory = /data_seqoccin/analyze/70205e13f
crdate = 1571044081 (Monday 14 October 2019 09:08:01)
retention_date = 1615503600 (Thursday 11 March 2021 23:00:00)

ls /work/ng6-test/jflow_rtherville/data_seqoccin/analyze/70205e13f
test.txt


-> Run 15059
directory = /data_seqoccin/run/fbc386b05
crdate = 1571044117 (Monday 14 October 2019 09:08:37)
retention_date = 1615503600 (Tuesday 14 January 2020 10:08:37)

ls /work/ng6-test/jflow_rtherville/data/run/fbc386b05
md5sum.txt  MT_rep1_1_Ch6.fastq.gz  MT_rep1_2_Ch6.fastq.gz


	-> Analysis 57767
	directory = /data_seqoccin/analyze/b8abf8aa0
	crdate = 1571044237 (Monday 14 October 2019 09:10:37)
	retention_date = 1615503600 (Thursday 11 March 2021 23:00:00)
	
	ls /work/ng6-test/jflow_rtherville/data_seqoccin/analyze/b8abf8aa0
	test.txt

There is a 1 hour shift to the new retentions, but it's normal, they are converted in GMT for this example. The values are perfect when converted to GMT+1.

mentioned in commit ba7316e1

I have refactored the workflow, and Project/Run/Analysis classes.

The update can be seen here : 18a04beb

mentioned in commit 61a4c53d

mentioned in merge request !56 (merged)

An output file should be added to the script, to log messages and errors in case a problem occurs.

mentioned in commit 7c169bce

mentioned in commit db1b75b8

mentioned in merge request !69 (merged)

mentioned in commit 13279d3d

mentioned in merge request !72 (merged)

Script/workflow to migrate a list of project to from a space to another

Designs

Child items ...

Activity