How Do I Create a Training Job When a Dependency Package Is Referenced by the Model to Be Trained?
Store the pip-requirements.txt file in the training code directory.
Any one of the following file names can be used. This section uses pip-requirements.txt as an example.
- pip-requirement.txt
- pip-requirements.txt
- requirement.txt
- requirements.txt
Before the training boot file is executed, the system automatically runs the following command to install the specified Python packages:
pip install -r pip-requirements.txt
- For details about the code directory, see Storing the Installation File in the Code Directory.
- For details about the specifications of pip-requirements.txt, see Installation File Specifications.
Storing the Installation File in the Code Directory
ModelArts allows you to install third-party dependency packages during model training in either of the following ways:
- Method 1 (recommended): Before creating an algorithm, store the required files or installation packages in the code directory.
Figure 1 Creating an algorithm
- Method 2: Before using a common framework to create a training job, store the required files or installation packages in the code directory. (This function will be unavailable soon.)
Figure 2 Using a common framework to create an algorithm
Installation File Specifications
The installation file varies depending on the dependency package type.
- Open-source installation packages
Installation using the source code from GitHub is not supported.
Create a file named pip-requirements.txt in the code directory, and specify the name and version number of the dependency package in the file. The format is [Package name]==[Version].
Take for example, an OBS path specified by Code Dir that contains model files and the pip-requirements.txt file. The code directory structure would be as follows:
|---OBS path to the model boot file |---model.py #Model boot file |---pip-requirements.txt #Defined configuration file, which specifies the name and version of the dependency package
The following shows the content of the pip-requirements.txt file:
alembic==0.8.6 bleach==1.4.3 click==6.6
- WHL packages
If the training background does not support the download of open source installation packages or use of user-compiled WHL packages, the system cannot automatically download and install the package. In this case, place the WHL package in the code directory, create a file named pip-requirements.txt, and specify the name of the WHL package in the file. The dependency package must be a .whl file.
Take for example, an OBS path specified by Code Dir that contains model files, the .whl file, and the pip-requirements.txt file. The code directory structure would be as follows:
|---OBS path to the model boot file |---model.py #Model boot file |---XXX.whl #Dependency package. If multiple dependencies are required, place multiple dependency packages here. |---pip-requirements.txt #Defined configuration file, which specifies the name of the dependency package
The following shows the content of the pip-requirements.txt file:
numpy-1.15.4-cp36-cp36m-manylinux1_x86_64.whl tensorflow-1.8.0-cp36-cp36m-manylinux1_x86_64.whl
Compiling the Training Code FAQs
- How Do I Create a Training Job When a Dependency Package Is Referenced by the Model to Be Trained?
- What Is the Common File Path for Training Jobs?
- How Do I Install a Library That C++ Depends on?
- How Do I Check Whether a Folder Copy Is Complete During Job Training?
- How Do I Load Some Well Trained Parameters During Job Training?
- How Do I Obtain Training Job Parameters from the Boot File of the Training Job?
- Why Can't I Use os.system ('cd xxx') to Access the Corresponding Folder During Job Training?
- How Do I Invoke a Shell Script in a Training Job to Execute the .sh File?
- How Do I Obtain the Dependency File Path to be Used in Training Code?
- What Is the File Path If a File in the model Directory Is Referenced in a Custom Python Package?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore