Help Center> > Getting Started> Creating an Object Storage Migration Task

Creating an Object Storage Migration Task

Updated at: Jul 25, 2019 14:42

Scenarios

This section describes how to create an object storage task to migrate your object data.

Background

The Object Storage Migration Service supports the following source cloud providers:

  • HUAWEI CLOUD
  • Amazon Web Services (AWS)
  • Alibaba Cloud
  • Tencent Cloud
  • Qiniu Cloud
  • QingCloud
  • Baidu Cloud
  • Kingsoft Cloud

The Object Storage Migration Service makes calls to APIs of the object storage service on source platforms. You need to pay the fees incurred to cloud service providers of the source platforms based on their billing rules.

Prerequisites

  • You have obtained the AK and SK for both source and destination platforms.
  • You have created at least one bucket in OBS.
  • You have created fewer than 50 migration tasks during the past 24 hours.
  • You have less than 20 migration tasks in the Waiting to migrate status.

Procedure

  1. Log in to the management console.
  2. Click on the upper left and select the desired region.
  3. Click Service List. Under Migration, click Object Storage Migration Service.

    The Object Storage Migration Service page is displayed.

  4. Click Create Migration Task.

    Read the privacy statement and click Agree to the above privacy agreements. The page for creating an object storage migration task is displayed.

  5. Select Source/Destination.

    1. Select the source platform. For details about the parameters, see Table 1.
      Table 1 Source platform parameters

      Parameter

      Description

      Cloud Service Provider

      Specifies the source cloud service provider.

      appID

      Specifies the identifier of the account for logging in to the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, the value of appID must be entered.

      Access Key

      Specifies the access key (AK) of the source cloud service providers. Enter a maximum of 100 characters.

      Secret Access Key

      Specifies the secret access key (SK) of the source cloud service providers. The SK and AK constitute a key pair. Enter a maximum of 100 characters.

      Specify Bucket Name

      If you do not have the permission to list buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects, enable this parameter to specify the path of the source bucket for the migration.

      For details, see Table 2.

      Table 2 Description of Specify Bucket Name

      If You Need To

      You Can

      Enable Specify Bucket Name.

      1. Select the region where the source bucket resides from the Select Region drop-down list.
      2. In the Bucket Name or Path text box, enter the source bucket name or the source bucket path and name.
      3. Click Connect to Source Bucket.

        The Object Storage Migration Service will check whether the source platform information is correct. If the entered information is correct, a message is displayed indicating that the source bucket is successfully connected.

      Disable Specify Bucket Name (the Specify Bucket Name option is disabled by default).

      1. Click Connect to Source Bucket.

        The Object Storage Migration Service will check whether the source platform information is correct. If the entered information is correct and your account has the permission to list bucket names, all buckets are listed in the Bucket drop-down list.

      2. In the Bucket drop-down list, select the bucket where the data to be migrated resides.
    2. Select the destination platform. For details about the parameters, see Table 3.
      Table 3 Destination platform parameters

      Parameter

      Description

      Access Key

      Specifies the AK for accessing HUAWEI CLOUD. Enter a maximum of 100 characters.

      Secret Access Key

      Specifies the SK for accessing HUAWEI CLOUD. The SK and AK constitute a key pair. Enter a maximum of 100 characters.

      After entering the AK and SK, click Connect to Destination Bucket.

      Bucket

      Specifies a HUAWEI CLOUD OBS bucket for storing the migrated data.

  6. Click Next to set task parameters.
  7. Select a migration method in the Select Migration Method area.

    • File/Folder
      Click Select. In the displayed dialog box, select the file or folder to be migrated, and click OK. You can click to delete a selected file or folder from the object list.
      NOTE:
      • If you select Qiniu Cloud for Cloud Service Provider, you cannot select files or folders.
      • If no file or folder is specified, all objects in the source bucket will be migrated.
      • For each migration task, you can select a maximum of 500 files or folders under the same directory.
      • The name of each object cannot exceed 1024 characters in OBS. Ensure that the name of the object migrated to the destination platform cannot exceed 1024 characters. Otherwise, the migration task will fail.
    • Object List
      Generally, this parameter is used to migrate incremental data by selecting an object list file. You need to write the name of the source object to be migrated into an object list file and store the object list file in a HUAWEI CLOUD OBS bucket. The Object Storage Migration Service migrates all objects in the specified object list file.
      1. Select List File Bucket (HUAWEI CLOUD).
        NOTE:

        The buckets listed in this parameter are the buckets obtained by entering the destination AK and SK in 5.b.

        Ensure that the bucket for storing object list files locates in the same region as the destination bucket. Otherwise, the object list files cannot be selected.

      2. Click Select next to the Object List File area. Select the specified object list files and click OK.
      NOTE:
      • The size of the object list file cannot exceed 100 MB.
      • The object list file must be a .txt file.
      • Object list files must be in UTF-8 format.
      • Only one object name can be included in each row in the object list file.
      • Invalid spaces are not allowed in each line in the object list file. Invalid spaces may cause migration failures because invalid spaces may be used as object names.
      • Archive data can be accessed and obtained by the Object Storage Migration Service only after it is resorted. If the object list file is archived data, manually restore it before migration.
    • Object Name Prefix

      Enter the name or prefix of objects to be migrated. The system will automatically list the matched data. You can click to delete the data that does not need to be migrated from the list.

      NOTE:

      If no object name or name prefix is specified, all objects in the source bucket will be migrated.

  8. (Optional) Set Source Parameters.

    Parameters described in Table 4 are available only when the source cloud service provider offers the related functions.
    Table 4 Source parameters

    Parameter

    Description

    Use Source Custom Domain Name

    Specifies whether to use a custom domain name of the source cloud service provider to download data to reduce downloading fees. Select this option if the default domain name cannot meet your migration requirements and the source cloud service provider supports custom domain names.

    NOTE:

    If the source cloud service provider is Qiniu Cloud, this option is selected by default to ensure a smooth migration because Qiniu Cloud has restrictions on using of default domain name.

    If you select this option, complete the following settings:

    1. Select a transmission protocol for Transfer Protocol.
      NOTE:

      HTTP is less secure than HTTPS.

    2. Enter a custom domain name of the bucket for Custom Bucket Domain Name.
    3. If you use the CDN authentication provided by the source cloud service provider, select Enable CDN Authentication and configure the authentication type and keys.
      NOTE:

      You can use the CDN function only after configuring CDN by referring to documents of the source cloud service provider.

      The fees generated by using CDN on the source platform are billed by the source cloud service provider.

    4. Click Check to check whether the configuration is correct.
      • If the configuration check fails, click Details to view the failure causes.
      • If the configuration passes the check, complete other migration configurations.

    Archive Data Restoration

    Specifies that only restored archive data can be migrated. You can select this option if your source cloud service provider supports automatic restoration of archive data.

    • If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system records it in the list of objects that fail to be migrated and continues to migrate other objects in the migration task.
    • If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.
    NOTE:
    • Currently, the following cloud service providers can automatically restore archive data: HUAWEI CLOUD, Alibaba Cloud, and Kingsoft Cloud.
    • It takes more than one minute to restore archive objects. The larger the archive object data is, the longer it takes to restore it. The restoration fails if archive data cannot be restored within 3 minutes. The required restoration duration varies depending on cloud service providers. For details, see related documents of the specific cloud service provider.
    • Two types of fees are generated during archive data restoration: API invoking fee and archive data retrieval fee. These fees are charged by the source cloud service provider.
    • After restoration, archive objects will be in restored state for a period. During this period, objects can be downloaded or accessed. After this period expires, archive objects need to be restored again for migration. For details about the period of the restored state, see related documents of the specific cloud service provider.
    • After the object storage data is migrated from the source platform to HUAWEI CLOUD, the storage type of the migrated data must be the same as that of the destination bucket.

  9. Click Show next to Advanced Settings to expand the advanced parameters.
  10. (Optional) Set Advanced Settings.

    Table 5 Advanced settings

    Parameter

    Description

    Data Encryption

    If you select KMS, migrated data is encrypted before being stored to the destination OBS buckets.
    NOTE:

    Using KMS to encrypt migrated data may decrease the migration speed by about 10%.

    This option is available only when KMS encryption is supported in the current region.

    Failed Object Record

    Specifies whether to record the objects that fail to be migrated.

    • If this parameter is set to No, the system does not record the objects that fail to be migrated. If the migration fails, all objects in the source bucket are transmitted again.
    • If this parameter is set to Yes, the system will record the objects that fail to be migrated. If the migration fails, only the objects that fail to be migrated are retransmitted.
    NOTE:

    The objects that fail to be migrated are recorded in an object list file named by the task name and uploaded to the oms/failed_object_lists directory of the destination bucket. This file records the summary information about the failed migration task, including failure cause, occurrence time (local UTC), and objects fail to be migrated.

    This file is stored in the destination bucket after the objects in the file are successfully retransmitted. If you do not need the file, delete it manually.

    Incremental Data

    Specifies whether to filter source objects to be migrated by time.

    • If this option is set to No, source objects to be migrated are not filtered by time.
    • If this option is set to Yes, only the source objects that are modified after the specified time are migrated. You need to click to specify the time.

    SMN Notification

    Specifies whether to use SMN to notify you of migration task results.
    • If this parameter is set to No, no SMN message is sent after the migration.
    • If this parameter is set to Yes, you will receive an SMN message about the result of the migration task.

      Perform the following steps to set SMN parameters:

      1. Click Add next to the Select SMN Topic area.

        The Select SMN Topic dialog box is displayed.

      2. Select a topic for receiving SMN messages and click OK.

        If the Object Storage Migration Service is not authorized to publish messages to the selected SMN topic, a Warning dialog box will display after you click OK. If you need the authorization, click OK to authorize the Object Storage Migration Service. Otherwise, click Cancel to return to the Select SMN Topic page and select another SMN topic.

      1. Configure Trigger Condition. You can select one of the following options or both:
      • Migration succeeded indicates that an SMN message will be sent after the migration task succeeds.
      • Migration failed indicates that an SMN message will be sent after the migration task fails.
      NOTE:

      If you experience trouble in using SMN, see What Should I Do If the SMN Function Works Improperly?

    Description

    In the Description area, enter a brief description about this migration task.

    Enter a maximum of 255 characters.

  11. Click Next to enter the confirmation page.
  12. Confirm the settings of the migration task and start the migration.

    1. Check whether the settings of migration parameters are correct.

      If the information is incorrect, click Previous to return to the previous page for modification.

    2. Click Migrate Now.

      A message is displayed, indicating that the migration task is successfully created. The migration task is displayed in the migration task list.

      For details about the migration task status, see Viewing Migration Tasks.
      NOTE:

      After the task is created, a task ID is generated for you for quick identification. It is displayed in the upper left corner of the task list. It is composed of the source bucket name, destination bucket name, and time sequence ID.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel