Help Center > > User Guide> Getting Started> Submitting a Spark Job

Submitting a Spark Job

Updated at: Jun 21, 2019 GMT+08:00

This section describes how to submit a Spark job using DLI. The general procedure is as follows:

Step 1: Logging In to The Public Cloud

Step 2: Uploading Data to OBS

Step 3: Entering the Spark Job Editor Page

Step 4: Creating a Cluster

Step 5: Creating a Package

Step 6: Submitting a Spark Job

Step 1: Logging In to The Public Cloud

To use DLI, you need to log in to the public cloud.

  1. Open the cloud homepage.
  2. On the login page, enter the Username and Password, and click Login.

Step 2: Uploading Data to OBS

Before submitting Spark jobs, upload data files to OBS.

  1. From the menu on top of the public cloud homepage, move your cursor on Products.
  2. In the services displayed, click Object Storage Service in Storage.
  3. On the OBS product page, click Console. The OBS console page is displayed.
  4. Create a bucket. The bucket name is globally unique. In this example, assume that the bucket name is obs1.
    1. Click Create Bucket.
    2. On the Create Bucket page that is displayed, specify Region and Bucket Name.
      NOTE:

      When creating an OBS bucket, you must select the same region as the DLI management console.

    3. Click Create Now.
  5. Click obs1 to switch to the Summary page.
  6. From the left navigation tree, click Object. Click Upload File. In the displayed dialog box, click the button next to the text box to the right of File and select the required file, for example, spark-examples_09.jar. Then, click OK.

    After the file is uploaded successfully, the file path to be analyzed is s3a://obs1/spark-examples_09.jar.

Step 3: Entering the Spark Job Editor Page

To submit Spark jobs, you need to enter the Spark job editor first.

  1. From the menu on top of the public cloud homepage, move your cursor on Products.
  2. In the service list displayed, click Data Lake Insight in Big Data.
  3. On the DLI product page, click Access Console to go to the DLI management console page.
  4. Click Spark Job on the Overview page or click Create Job on the right to go to the Job Editor page of the Spark job.

Step 4: Creating a Cluster

If it is your first time submitting a Spark job, create a cluster first. For example, create a cluster named testnew. For details about how to create a cluster, see Creating a cluster.

Step 5: Creating a Package

Before submitting a Spark job, you need to create a package, for example, spark-examples_09.jar. For details about how to create a package, see Creating a Package.

Step 6: Submitting a Spark Job

  1. On the Job Editor page, click .
  2. Select the target queue in the upper right corner of the current Spark job editing window.
  3. In the Spark job editing window, set related parameters. For details, see the description of the Spark job editing window in Job Editor Page Description.
  4. Click Execute in the upper right corner of the Spark job editing window to submit the job. The message Batch submission succeeded. is displayed.
  5. (Optional) Switch to the Job Management page to view the status and logs of the submitted Spark job.
    NOTE:

    After the job is executed successfully, the job record is saved for only 6 hours.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel