Transferring Data Between Projects
===================================

This guide covers how to transfer DICOM data between XNAT projects, including copying (sharing) and moving (relocating) data.

Overview
--------

Transferring data between projects is useful when:

- **Sharing data** with collaborators in different projects
- **Moving data** to a new project or consolidating projects
- **Creating backups** or archival copies in separate projects
- **Reorganizing data** across project structures

**Primary Focus: DICOM Data Transfers**

This guide focuses on transferring DICOM data (raw imaging data from scanners), which is the most common transfer scenario. DICOM data can be downloaded and re-uploaded using standard XNAT tools.

**About Resources (Pipeline Outputs)**

Data stored in Resources folders (such as BIDS rawdata, fMRIPrep outputs, MRIQC reports) are typically **regenerated by running pipelines** in the destination project rather than transferred. However, advanced users can transfer Resources using API methods (Python xnatpy, MATLAB scripts, or REST API) if needed. See :doc:`../data_download/python_scripts` and :doc:`../data_download/matlab_scripts` for API-based transfer methods.

Prerequisites & Planning
------------------------

Before transferring data, ensure you have the necessary access and have planned the transfer carefully.

Access Requirements
~~~~~~~~~~~~~~~~~~~

You must have appropriate permissions in **both** source and destination projects:

- **Source Project**: Read access (Member or Owner role)
- **Destination Project**: Write access (Member or Owner role)
  - May also need subject creation permissions if creating new subjects

To check your access level, navigate to each project and verify you can view (source) or upload (destination) data. If you don't have the required access, contact the project administrators or see :doc:`access` for requesting project access.

Pre-Transfer Checklist
~~~~~~~~~~~~~~~~~~~~~~

Before starting a transfer, complete this checklist:

1. **Verify Access**: Confirm you have read access to source project and write access to destination project
2. **Check for Conflicts**: Check if subjects/sessions with the same labels already exist in the destination project
3. **Plan Label Mapping**: If projects use different naming conventions, plan how to map subject/session labels
4. **Review DICOM Metadata**: Consider Study Instance UIDs, Patient IDs, and routing strings
5. **Check Project Settings**: Verify upload destination (Prearchive vs Archive) and subject creation permissions
6. **Verify Anonymization**: Ensure data is properly anonymized before transfer (critical for compliance)

Important Considerations
-------------------------

Before transferring data, be aware of these critical considerations:

.. warning::
   **Data Overwriting and Merging**
   
   Uploading data to an existing project or session can overwrite existing files or merge new data into an existing session.
   
   - **Merging**: If you upload data with the same subject/session labels as existing data, XNAT will **merge** the new scans into the existing session (not overwrite)
   - **File Overwriting**: Files with identical names may be replaced
   
   Always check for existing subjects/sessions in the destination project before uploading to avoid unintended merges.

.. warning::
   **Data Anonymization**
   
   Data from MRI scanners is typically anonymized at acquisition. When transferring data manually, ensure that your data is properly anonymized before transfer. All personally identifiable information (PII) and protected health information (PHI) must be removed or de-identified according to your project's requirements and institutional policies.

DICOM Metadata Considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When transferring DICOM data, consider:

- **Study Instance UIDs**: Unique identifiers for each study
- **Patient IDs**: May need to be consistent or updated depending on project requirements
- **Routing Strings**: If using DICOM receiver for upload, Study Description routing strings (``xnat://project_id/subject_label/session_label``) need to be updated for the destination project

Project Settings Differences
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Different projects may have different upload settings:

- **Upload Destination**: Some projects send data directly to Archive, others use Prearchive first
- **Subject Creation**: Some projects allow users to create subjects during upload, others require pre-registration
- **Bulk Upload Restrictions**: Some projects restrict bulk uploads

Check with the destination project administrator if you encounter restrictions. For more information, see :doc:`uploading`.

Transfer Methods
-----------------

There are several methods for transferring DICOM data between projects. Choose the method that best fits your needs.

Method 1: Desktop Client (Recommended for Large Transfers)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The **XNAT Desktop Client** is recommended for large transfers because it provides:

- Resume capability if transfers are interrupted
- Progress monitoring and transfer management
- Reliable handling of large datasets
- Ability to select specific scans during upload

This method involves downloading from the source project using Desktop Client, then uploading to the destination project using the same tool.

For detailed instructions, see:
- :doc:`../data_download/desktop_client` for download procedures
- :doc:`uploading` for upload procedures (Desktop Client section)

Method 2: Browser Download + Compressed Uploader
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This method is best for:

- Small to medium transfers
- Single session transfers
- Quick one-off transfers

It involves downloading sessions via the web browser, then uploading compressed archives using the Compressed Uploader.

For detailed instructions, see:
- :doc:`../data_download/browser` for browser download procedures
- :doc:`uploading` for Compressed Uploader procedures

Method 3: Browser Download + Desktop Client Upload
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A hybrid approach where you download via browser but upload via Desktop Client. Useful when you prefer different methods for download and upload.

Step-by-Step Workflows
-----------------------

Workflow A: Copying DICOM Sessions (Desktop Client)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This workflow uses Desktop Client for both download and upload, recommended for large or multiple sessions.

**Step 1: Verify Access**

- Log into XNAT and verify you can access both source and destination projects
- Check that you have read access to source and write access to destination

**Step 2: Download from Source Project**

1. Navigate to the source project in XNAT web interface
2. Select the sessions you want to transfer (can select multiple)
3. Click **"Download Images"** from the Actions menu
4. Choose **"Catalog XML"** as the download format
5. Download the XML file to your computer
6. Open XNAT Desktop Client (see :doc:`install_desktop_client` if not installed)
7. Ensure you're logged into the correct XNAT server
8. Select **"Download Files"** from the main menu
9. Browse and select your downloaded XML file
10. Set download location and click **"Download"**
11. Wait for download to complete

For detailed download instructions, see :doc:`../data_download/desktop_client`.

**Step 3: Prepare Data**

Before uploading, verify your downloaded data:

- Check that all expected sessions were downloaded
- Verify DICOM file structure is intact
- Ensure data is properly anonymized
- Remove any non-DICOM files if present

**Step 4: Upload to Destination Project**

1. Navigate to the destination project in XNAT web interface
2. Click **"Upload Images"** from the Actions menu (this opens Desktop Client automatically)
   
   OR
   
   Open Desktop Client directly and click **"Upload Images"**
3. Select the destination project from the project dropdown
4. Select the downloaded DICOM folders (drag and drop or browse)
5. The client will scan files to identify sessions
6. **Configure Upload Settings**:
   - **Subject Selection**: Choose existing subject or create new (if allowed)
   - **Subject Label**: Ensure it matches destination project naming conventions
   - **Session Label**: Auto-generated or override if needed
   - **Check for Conflicts**: Verify no existing sessions with same labels
   - **Scan Selection**: Select which scans to include (uncheck any you don't want)
7. **Select Upload Destination**: Choose Prearchive (recommended) or Archive
8. Click **"Start Upload"**
9. Monitor progress in the transfer manager

For detailed upload instructions, see :doc:`uploading`.

**Step 5: Verify Transfer Success**

1. Navigate to the destination project
2. If uploaded to Prearchive, review the Prearchive first
3. Check that subjects and sessions were created correctly
4. Verify scan counts match between source and destination
5. Compare file sizes to ensure completeness
6. Test downloading a sample scan to verify integrity

**Step 6: Archive (if using Prearchive)**

1. Review data in Prearchive
2. Verify all metadata is correct
3. Archive the sessions to move them to permanent storage

Workflow B: Copying DICOM Sessions (Browser + Compressed Uploader)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This workflow is faster for single sessions or small transfers.

**Step 1: Verify Access**

- Verify access to both source and destination projects

**Step 2: Download from Source Project**

1. Navigate to the session in the source project
2. Use the session-level download option
3. Download the session as a ZIP archive via browser

For detailed instructions, see :doc:`../data_download/browser`.

**Step 3: Upload to Destination Project**

1. Log into XNAT web interface
2. Navigate to **Upload** > **Images** > **Compressed Uploader**
3. **Select Project**: Choose the destination project from dropdown
4. **Configure Upload Settings**:
   - **Importer handler**: Select "DICOM Zip"
   - **Custom Labeling**: Choose "Extract From DICOM (Default)" or "Customize" to specify labels
   - **Subject ID** and **Session Label**: Enter if using custom labeling
   - **Ignore unparsable files**: Check to skip problematic files
5. **Select Destination**: Choose **Prearchive** (recommended for safety)
6. **Choose File**: Select your downloaded ZIP file
7. Click **"Begin Upload"**
8. Wait for confirmation message

For detailed instructions, see :doc:`uploading`.

**Step 4: Verify and Archive**

1. Navigate to Prearchive in destination project
2. Review the uploaded session
3. Verify subject/session labels are correct
4. Check that all scans are present
5. Archive the session to move to permanent storage

Workflow C: Moving Data (Copy then Delete)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To **move** data (rather than copy), follow these additional steps after completing Workflow A or B:

.. warning::
   **Permanent Deletion**
   
   Deleting data from the source project is **permanent**. Only proceed after thorough verification that all data has been successfully transferred to the destination.
   
   If you accidentally delete data, contact XNAT administrators immediately. See :doc:`../support/data_loss_recovery` for information about data recovery options.

**Step 1: Complete Copy Workflow**

- Follow Workflow A or B to copy data to destination project
- Ensure upload is complete and archived

**Step 2: Verify in Destination**

- Thoroughly verify all data transferred correctly:
  - All subjects and sessions present
  - All scans present with correct counts
  - File sizes match
  - Sample downloads work correctly
  - No errors in upload logs

**Step 3: Delete from Source (if appropriate)**

- Only proceed if you have deletion permissions and have verified the transfer
- Navigate to source project
- Delete subjects or sessions as needed
- **Note**: Deletion may require Owner-level permissions

**Step 4: Final Verification**

- Confirm deletion from source project
- Verify destination data is complete and accessible
- Document the transfer for project records

About Resources (Advanced Users)
---------------------------------

What are Resources?
~~~~~~~~~~~~~~~~~~~

Resources are folders in XNAT that store pipeline outputs and processed data:

- **Session-level Resources**: BIDS rawdata, fMRIPrep outputs, MRIQC reports, processing logs
- **Subject-level Resources**: Subject-specific processed data
- **Project-level Resources**: Shared configuration files, project-wide data

Resources are typically stored in the ``Resources`` folder at each hierarchy level (Project, Subject, Session).

When to Transfer Resources
~~~~~~~~~~~~~~~~~~~~~~~~~~

**Generally, Resources should NOT be transferred** because:

- They are outputs from pipelines that can be **regenerated** by running the same pipelines in the destination project
- Transferring Resources can be complex and error-prone
- Pipeline outputs are often project-specific and may not work correctly in a different project context

**Transfer Resources only if**:

- You need to preserve specific pipeline outputs that cannot be easily regenerated
- You have custom processing results that are not reproducible
- You need to maintain exact version compatibility

How to Transfer Resources (Advanced)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you need to transfer Resources, use API-based methods:

- **Python**: Use xnatpy library or Python scripts (see :doc:`../data_download/python_scripts`)
- **MATLAB**: Use MATLAB scripts with XNAT API (see :doc:`../data_download/matlab_scripts`)
- **REST API**: Direct API calls to download and upload Resources

**Note**: Desktop Client is primarily designed for DICOM transfers, not Resources. For Resource transfers, API methods are required.

Verification Steps
------------------

After completing a transfer, verify the data integrity:

1. **Subject/Session Creation**: Check that all subjects and sessions were created in the destination project
2. **Scan Counts**: Verify the number of scans matches between source and destination
3. **File Sizes**: Compare file sizes to ensure no data loss during transfer
4. **DICOM Metadata**: If applicable, verify Study Instance UIDs and Patient IDs are correct
5. **Prearchive Review**: If using Prearchive, review all data before archiving
6. **Sample Download**: Test downloading a sample scan to verify file integrity
7. **Upload Logs**: Review upload logs for any warnings or errors

Troubleshooting
---------------

Common Issues and Solutions
~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Permission Errors**

- **Symptom**: Cannot access source project or cannot upload to destination
- **Solution**: Verify you have Member or Owner access to both projects. Contact project administrators if needed.

**Duplicate Label Conflicts**

- **Symptom**: Data merges with existing sessions instead of creating new ones
- **Solution**: Check for existing subjects/sessions with the same labels in destination. Use different labels or coordinate with project administrators. If you encounter UID errors (such as "Invalid modification of session UID") when merging different sessions, see :doc:`uid_errors_archiving` for detailed resolution steps.

**Upload Failures**

- **Symptom**: Uploads fail or timeout
- **Solution**: 
  - Check file permissions and quota limits
  - Verify network connectivity
  - Try smaller batches or use Desktop Client for better reliability
  - Check upload logs for specific error messages

**Missing Data After Transfer**

- **Symptom**: Not all scans appear in destination
- **Solution**: 
  - Verify all scans were selected during upload
  - Check upload logs for failed scans
  - Ensure all DICOM files were included in download
  - Retry failed uploads if needed

**DICOM Routing Issues**

- **Symptom**: Data doesn't route correctly when using DICOM receiver
- **Solution**: Verify Study Description routing strings (``xnat://project_id/subject_label/session_label``) are correct for destination project

**Prearchive Stuck**

- **Symptom**: Data remains in Prearchive and cannot be archived
- **Solution**: Contact XNAT administrators at admin.nyuad.xnat@nyu.edu for assistance

**Large Transfer Performance**

- **Symptom**: Very slow transfers or timeouts
- **Solution**: 
  - Use Desktop Client for large transfers (better resume capability)
  - Transfer during off-peak hours
  - Break large transfers into smaller batches
  - Ensure stable network connection

Getting Help
------------

If you encounter issues not covered here:

- Review :doc:`../support/troubleshooting` for additional troubleshooting steps
- Check :doc:`../support/faq` for common questions
- Contact XNAT administrators at admin.nyuad.xnat@nyu.edu

Best Practices
--------------

Follow these best practices for safe and efficient data transfers:

1. **Always Verify Before Deleting**: Never delete source data until you've thoroughly verified the transfer in the destination project. If data is accidentally deleted, see :doc:`../support/data_loss_recovery` for recovery options
2. **Use Prearchive**: Upload to Prearchive first to allow review before archiving
3. **Document Transfers**: Keep records of what was transferred, when, and why for project documentation
4. **Coordinate with Administrators**: For large transfers or complex scenarios, coordinate with project administrators
5. **Test First**: Test with a small dataset before attempting bulk transfers
6. **Check Naming Conventions**: Verify subject/session label conventions match destination project requirements
7. **Ensure Anonymization**: Always verify data anonymization compliance before transfer
8. **Monitor Transfers**: Keep an eye on transfer progress and logs to catch issues early

See Also
--------

**Related Documentation**

- :doc:`uploading` - Detailed upload procedures and methods
- :doc:`uid_errors_archiving` - Resolving UID errors during archiving
- :doc:`../data_download/browser` - Browser-based download methods
- :doc:`../data_download/desktop_client` - Desktop Client download and upload
- :doc:`install_desktop_client` - Desktop Client installation guide
- :doc:`access` - Project access and permissions information
- :doc:`../understanding_data/overview` - Understanding XNAT data structure

**Advanced Resource Transfers**

- :doc:`../data_download/python_scripts` - Python API methods for Resource transfers
- :doc:`../data_download/matlab_scripts` - MATLAB API methods for Resource transfers

**Support**

- :doc:`../support/data_loss_recovery` - Information about backups and data recovery
- :doc:`../support/troubleshooting` - Troubleshooting guide
- :doc:`../support/faq` - Frequently asked questions
- :doc:`../support/contact` - Contact information