Migration Guide

This guide provides detailed instructions for migrating your data from other storage systems to NexStorage. Whether you're moving from AWS S3, Azure Blob Storage, or any other object storage platform, NexStorage offers seamless migration paths to ensure a smooth transition.

Migration Planning

Pre-Migration Checklist

Before starting your migration to NexStorage, complete this checklist:

• Inventory your data: Catalog all buckets, objects, and access patterns
• Calculate total storage requirements: Ensure sufficient capacity in NexStorage
• Identify critical workloads: Prioritize migration order based on importance
• Document current access patterns: Understand how applications interact with storage
• Map security policies: Document current IAM policies, bucket policies, and ACLs
• Create a rollback plan: Ensure you can revert if needed
• Set up monitoring: Configure metrics to track migration progress
• Schedule maintenance windows: Plan for potential application downtime
• Test with sample data: Verify performance and compatibility before full migration

Choosing a Migration Strategy

NexStorage supports multiple migration strategies:

Strategy	Best For	Downtime Required
Direct Copy	Smaller datasets (less than 10TB)	Minimal
Batch Migration	Medium datasets (10-100TB)	Scheduled windows
Continuous Replication	Large datasets (greater than 100TB)	None
Application-Level Migration	Complex applications	Varies

Migrating from AWS S3

Using NexStorage Migration Tool

The NexStorage Migration Tool provides a seamless way to migrate from AWS S3:

1. Install the migration tool:

   pip install nexstorage-migration

2. Configure source and destination:

   nexstorage-migrate init \
     --source aws \
     --source-access-key YOUR_AWS_ACCESS_KEY \
     --source-secret-key YOUR_AWS_SECRET_KEY \
     --source-region us-east-1 \
     --dest nexstorage \
     --dest-access-key YOUR_NEXSTORAGE_ACCESS_KEY \
     --dest-secret-key YOUR_NEXSTORAGE_SECRET_KEY \
     --dest-endpoint https://s3.nexstorage.nexvecta.com

3. Create a migration plan:

   nexstorage-migrate plan \
     --buckets bucket1,bucket2,bucket3 \
     --output migration-plan.json

4. Review the plan:

   cat migration-plan.json

5. Execute the migration:

   nexstorage-migrate start \
     --plan migration-plan.json \
     --workers 10 \
     --bandwidth-limit 500MB

6. Monitor progress:

   nexstorage-migrate status

Using AWS CLI

For smaller migrations, you can use the AWS CLI:

# Configure AWS CLI profiles
aws configure --profile aws-source
aws configure --profile nexstorage

# Create target bucket in NexStorage
aws s3 mb s3://target-bucket \
  --profile nexstorage \
  --endpoint-url https://s3.nexstorage.nexvecta.com

# Sync all data
aws s3 sync \
  s3://source-bucket \
  s3://target-bucket \
  --profile aws-source \
  --profile nexstorage \
  --endpoint-url https://s3.nexstorage.nexvecta.com

For incremental migration with continuous synchronization:

# Create a script to run regularly (e.g., via cron)
#!/bin/bash
aws s3 sync \
  s3://source-bucket \
  s3://target-bucket \
  --profile aws-source \
  --profile nexstorage \
  --endpoint-url https://s3.nexstorage.nexvecta.com \
  --size-only \
  --no-sign-request

Migrating Bucket Policies and ACLs

1. Export existing S3 bucket policy:

   aws s3api get-bucket-policy \
     --bucket source-bucket \
     --profile aws-source \
     --output json > bucket-policy.json

2. Modify the policy for NexStorage compatibility:

   # Edit the policy file manually to adapt any AWS-specific elements
   nano bucket-policy.json

3. Apply the policy to NexStorage bucket:

   aws s3api put-bucket-policy \
     --bucket target-bucket \
     --policy file://bucket-policy.json \
     --profile nexstorage \
     --endpoint-url https://s3.nexstorage.nexvecta.com

Migrating from Azure Blob Storage

Using NexStorage Migration Tool

1. Configure the migration:

   nexstorage-migrate init \
     --source azure \
     --source-connection-string "YOUR_AZURE_CONNECTION_STRING" \
     --dest nexstorage \
     --dest-access-key YOUR_NEXSTORAGE_ACCESS_KEY \
     --dest-secret-key YOUR_NEXSTORAGE_SECRET_KEY \
     --dest-endpoint https://s3.nexstorage.nexvecta.com

2. Create and execute the migration plan:

   nexstorage-migrate plan \
     --containers container1,container2 \
     --output migration-plan.json
   
   nexstorage-migrate start \
     --plan migration-plan.json

Using AzCopy

For direct migration using Azure tools:

1. Install AzCopy:

   # Download and install AzCopy
   wget https://aka.ms/downloadazcopy-v10-linux
   tar -xvf downloadazcopy-v10-linux
   sudo cp ./azcopy_linux_amd64_*/azcopy /usr/bin/

2. Configure AWS credentials for NexStorage:

   export AWS_ACCESS_KEY_ID=YOUR_NEXSTORAGE_ACCESS_KEY
   export AWS_SECRET_ACCESS_KEY=YOUR_NEXSTORAGE_SECRET_KEY

3. Perform the migration:

   azcopy copy \
     "https://youraccount.blob.core.windows.net/container/*" \
     "https://s3.nexstorage.nexvecta.com/bucket/" \
     --recursive

Migrating from Google Cloud Storage

Using NexStorage Migration Tool

1. Configure the migration:

   nexstorage-migrate init \
     --source gcp \
     --source-credentials-file /path/to/gcp-credentials.json \
     --dest nexstorage \
     --dest-access-key YOUR_NEXSTORAGE_ACCESS_KEY \
     --dest-secret-key YOUR_NEXSTORAGE_SECRET_KEY \
     --dest-endpoint https://s3.nexstorage.nexvecta.com

2. Create and execute the migration plan:

   nexstorage-migrate plan \
     --buckets gcp-bucket1,gcp-bucket2 \
     --output migration-plan.json
   
   nexstorage-migrate start \
     --plan migration-plan.json

Using gsutil

For direct migration using Google Cloud tools:

1. Install gsutil (part of Google Cloud SDK):

   # Install Google Cloud SDK
   curl https://sdk.cloud.google.com | bash
   exec -l $SHELL
   gcloud init

2. Configure AWS credentials for NexStorage:

   aws configure --profile nexstorage

3. Perform the migration:

   gsutil -m rsync -r gs://gcp-bucket s3://nexstorage-bucket

Migrating Large Datasets

For datasets larger than 100TB, consider these approaches:

Batch Migration with Parallelization

Split the migration into manageable chunks:

# Create a list of prefixes to migrate separately
aws s3 ls s3://source-bucket/ --profile aws-source | awk '{print $2}' > prefixes.txt

# Process each prefix in parallel
cat prefixes.txt | parallel -j 5 'aws s3 sync s3://source-bucket/{} s3://target-bucket/{} --profile aws-source --profile nexstorage --endpoint-url https://s3.nexstorage.nexvecta.com'

Using Offline Data Transfer

For extremely large datasets, NexStorage offers an offline data transfer service:

1. Request a transfer appliance from NEXVECTA
2. Copy your data to the appliance
3. Ship the appliance to NEXVECTA
4. NEXVECTA loads your data into NexStorage
5. Perform incremental synchronization for changes since the appliance was sent

Contact NEXVECTA Support to arrange an offline data transfer.

Application Migration

Updating Application Configurations

Update your applications to use NexStorage:

Python (boto3):

# Before (AWS S3)
import boto3
s3 = boto3.client('s3')

# After (NexStorage)
import boto3
s3 = boto3.client(
    's3',
    endpoint_url='https://s3.nexstorage.nexvecta.com',
    aws_access_key_id='YOUR_NEXSTORAGE_ACCESS_KEY',
    aws_secret_access_key='YOUR_NEXSTORAGE_SECRET_KEY'
)

Node.js:

// Before (AWS S3)
const AWS = require('aws-sdk');
const s3 = new AWS.S3();

// After (NexStorage)
const AWS = require('aws-sdk');
const s3 = new AWS.S3({
  endpoint: 'https://s3.nexstorage.nexvecta.com',
  accessKeyId: 'YOUR_NEXSTORAGE_ACCESS_KEY',
  secretAccessKey: 'YOUR_NEXSTORAGE_SECRET_KEY',
  s3ForcePathStyle: true
});

Java:

// Before (AWS S3)
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
    .withRegion(Regions.US_EAST_1)
    .build();

// After (NexStorage)
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
    .withEndpointConfiguration(
        new EndpointConfiguration("https://s3.nexstorage.nexvecta.com", "us-east-1"))
    .withCredentials(new AWSStaticCredentialsProvider(
        new BasicAWSCredentials("YOUR_NEXSTORAGE_ACCESS_KEY", "YOUR_NEXSTORAGE_SECRET_KEY")))
    .withPathStyleAccessEnabled(true)
    .build();

Phased Migration Approach

For complex applications, consider a phased migration:

1. Dual-write phase:

   • Write new data to both old storage and NexStorage
   • Read from old storage
   • Validate consistency between systems

2. Backfill phase:

   • Migrate existing data to NexStorage
   • Continue dual writes for new data
   • Begin reading from NexStorage for test traffic

3. Cutover phase:

   • Switch all reads to NexStorage
   • Stop writing to old storage
   • Monitor for issues

4. Cleanup phase:

   • Decommission old storage
   • Remove dual-write logic
   • Optimize application for NexStorage

Post-Migration Verification

Data Integrity Verification

Verify your migration with these checks:

# Count objects in source and destination
SOURCE_COUNT=$(aws s3 ls s3://source-bucket --recursive --profile aws-source | wc -l)
DEST_COUNT=$(aws s3 ls s3://target-bucket --recursive --profile nexstorage --endpoint-url https://s3.nexstorage.nexvecta.com | wc -l)
echo "Source: $SOURCE_COUNT objects, Destination: $DEST_COUNT objects"

# Compare total size
SOURCE_SIZE=$(aws s3 ls s3://source-bucket --recursive --profile aws-source --summarize | grep "Total Size" | awk '{print $3}')
DEST_SIZE=$(aws s3 ls s3://target-bucket --recursive --profile nexstorage --endpoint-url https://s3.nexstorage.nexvecta.com --summarize | grep "Total Size" | awk '{print $3}')
echo "Source: $SOURCE_SIZE bytes, Destination: $DEST_SIZE bytes"

# Validate checksums for critical files
for key in $(aws s3api list-objects --bucket source-bucket --query "Contents[].Key" --output text --profile aws-source); do
  SOURCE_ETAG=$(aws s3api head-object --bucket source-bucket --key "$key" --query "ETag" --output text --profile aws-source)
  DEST_ETAG=$(aws s3api head-object --bucket target-bucket --key "$key" --query "ETag" --output text --profile nexstorage --endpoint-url https://s3.nexstorage.nexvecta.com)
  
  if [ "$SOURCE_ETAG" != "$DEST_ETAG" ]; then
    echo "Checksum mismatch for $key"
  fi
done

Performance Testing

After migration, conduct performance tests:

# Test upload performance
dd if=/dev/zero bs=1M count=100 | time aws s3 cp - s3://target-bucket/test-file \
  --profile nexstorage \
  --endpoint-url https://s3.nexstorage.nexvecta.com

# Test download performance
time aws s3 cp s3://target-bucket/test-file /dev/null \
  --profile nexstorage \
  --endpoint-url https://s3.nexstorage.nexvecta.com

Compare these results with your previous storage solution.

Monitoring Post-Migration

After migration, closely monitor:

1. Error rates: Look for increased application errors
2. Latency: Check for changes in response times
3. Throughput: Verify expected transfer rates
4. Cost: Compare actual costs to projections

Best Practices for Large Migrations

Optimizing Transfer Performance

Maximize migration speed with these techniques:

1. Use multiple threads:

   nexstorage-migrate start --plan migration-plan.json --workers 20

2. Adjust chunk size for large objects:

   nexstorage-migrate start --plan migration-plan.json --multipart-chunksize 100MB

3. Implement bandwidth schedules to limit impact during business hours:

   nexstorage-migrate start --plan migration-plan.json --bandwidth-schedule "workday:50MB,nights:500MB,weekends:1GB"

4. Use direct network paths between storage systems

5. Optimize for object size:

   • Use parallel transfers for small objects
   • Use multipart uploads for large objects

Handling Special Cases

Versioned Buckets:

# Migrate all versions
nexstorage-migrate start --plan migration-plan.json --include-versions

Encrypted Objects:

# Migrate and re-encrypt with NexStorage keys
nexstorage-migrate start --plan migration-plan.json --re-encrypt

Website-Configured Buckets:

# Export website configuration
aws s3api get-bucket-website --bucket source-bucket --profile aws-source > website-config.json

# Apply to NexStorage bucket
aws s3api put-bucket-website --bucket target-bucket --website-configuration file://website-config.json --profile nexstorage --endpoint-url https://s3.nexstorage.nexvecta.com

Troubleshooting Migration Issues

Issue	Possible Cause	Solution
Slow transfer rates	Network limitations	Use a direct connection, increase bandwidth, or consider offline transfer
Permission errors	Incorrect access keys	Verify credentials and bucket policies
Object count mismatch	Hidden objects or prefixes	Use --include-hidden flag or check for dot-prefixed objects
Failed transfers	Intermittent connectivity	Enable automatic retry with --retry-attempts 5 --retry-delay 30
Checksum mismatches	Corrupted transfers	Use --verify-checksums and --repair options

Migrating from On-Premises Storage

From Legacy NAS/SAN Systems

1. Mount your storage system:

   mount -t nfs storage-server:/share /mnt/source

2. Use the NexStorage migration tool:

   nexstorage-migrate init \
     --source filesystem \
     --source-path /mnt/source \
     --dest nexstorage \
     --dest-access-key YOUR_NEXSTORAGE_ACCESS_KEY \
     --dest-secret-key YOUR_NEXSTORAGE_SECRET_KEY \
     --dest-endpoint https://s3.nexstorage.nexvecta.com

3. Execute the migration:

   nexstorage-migrate plan --output migration-plan.json
   nexstorage-migrate start --plan migration-plan.json

From Existing Object Storage Systems

For other S3-compatible storage (Ceph, MinIO, etc.):

aws s3 sync \
  s3://source-bucket \
  s3://target-bucket \
  --source-endpoint-url https://source-storage-endpoint \
  --endpoint-url https://s3.nexstorage.nexvecta.com \
  --source-profile source-storage \
  --profile nexstorage

Next Steps

After completing your migration to NexStorage:

1. Optimize your environment:

   • Set up monitoring and analytics
   • Configure security policies
   • Explore integration options

2. Decommission old storage:

   • Archive access logs and audit information
   • Remove old credentials
   • Terminate instances or services no longer needed

3. Document your new architecture:

   • Update system documentation
   • Train team members on NexStorage features
   • Establish new monitoring and maintenance procedures