You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Aug 11, 2020. It is now read-only.
The above example demonstrate running a python script locally and having that script transmit itself to the paperspace jobs cluster for further execution. To do this a copy of the local script is modified before transmission to the jobs cluster, in order to strip out the `import paperspace` statements and other `paperspace` library references. There are also some limitations on the types of import statements that are supported, and the dependencies that are supported in each environment (local vs. remote):
66
+
67
+
1. You need to use a bare import statement, `import paperspace`, and not use the `import paperspace as ...` form.
68
+
2. The import form `from paperspace import ...` is currently not supported.
69
+
3. Everything after the `paperspace.run()` function call is ignored when running locally (when no script name is provided). The local script execution stops after the `paperspace.run()` call.
70
+
4. Dependencies that are included before `paperspace.run()` must be available locally.
71
+
5. If you need to reference dependencies that are not available locally but are available remotely, those should be imported after the `paperspace.run()` call.
72
+
6. Dependencies that are needed remotely need to either be already installed in the container used for the job, or need to be installed using one of the techniques below in the section [Setting up python script dependencies remotely](#setting-up-python-script-dependencies-remotely)
73
+
74
+
Because of these limitations it may not always be appropriate to run python scripts automatically from within the same script file. As an alternative you can run your python scripts unmodified using the techniques in the next section.
75
+
76
+
77
+
Running a python script by name
78
+
===============================
79
+
You can run an python script on paperspace from the command line as follows:
80
+
81
+
paperspace-python run myscript.py
82
+
83
+
You can also provide additional jobs options on the command line:
84
+
85
+
paperspace-python run myscript.py --project myproject --machineType P5000 --container Test-Container
86
+
87
+
Alternatively you can use the `paperspace.run()` fuction in code with, a script file name as the first argument:
88
+
89
+
import paperspace
90
+
91
+
paperspace.run('myscript.py') # runs myscript on paperspace
92
+
93
+
In code you can provide additional paperspace jobs create options in a dict in the second argument to run():
See the Paperspace API [jobs create](https://paperspace.github.io/paperspace-node/jobs.html#.create) documentation for the full list of jobs create options that can be specified.
98
+
99
+
100
+
Setting up python script dependencies remotely
101
+
==============================================
102
+
When running python scripts on paperspace you may need to provide additional dependencies to your scripts.
103
+
The `paperspace-python run` command and `paperspace.run()` function provide several options to support this:
The `-m` option runs the specified library module as a script. This is equivalent to the `-m` option of the `python` executable.
108
+
109
+
The `--python 2|3` option allows you specify whether to use `python2` or `python3` when running the script on paperspace.
110
+
If ommitted, the script will be run with the same major version as is being used to run `paperspace-python` locally.
111
+
112
+
The `--init [<init.sh>]` option is used to specify a script to be run on the remote machine, inside the container, before the python script is run.
113
+
If the script name is ommitted, it is assumed to be the script named `init.sh` in the current directory. The script is run using
114
+
`source init.sh` in the container bash shell. You can use this option to provide a list of commands to run to set up the dependencies for the script, such as running a list of `pip install` commands. However, if you are using `pipenv` or a `requirements.txt` file we recommend you use one of the options below. Multiple dependency setup options can be combinded however.
115
+
116
+
The `--pipenv` option is used to upload and run the `Pipfile` and `Pipfile.lock` files in the current directory, if found. These files
117
+
are used on the paperspace machine to initialize the python environment using the `pipenv` tool, by running `pipenv install` within the container. Note: `pipenv` must already be installed in the container for this option to work. The default container used by paperspace-python already has the `pipenv` package installed.
118
+
119
+
The `--req [<requirements.txt>]` option is used specify that a `requirements.txt` file should be used to install the required python dependencies using `pip`.
120
+
By default this option looks for a file named `requirements.txt` in the current directory, but you can override this by specifying a different file name. Note: `pip` must already be installed in the container for this option to work. The default container used by paperspace-python already has the `pip` package installed.
121
+
122
+
The jobs create `--workspace` option is also available with any or all of the above options.
123
+
With the `--workspace` option you can specify a workspace directory to upload. For example, to upload the current directory along with the script file run:
124
+
125
+
paperspace-python run myscript.py --workspace .
126
+
127
+
See the Paperspae API [jobs create](https://paperspace.github.io/paperspace-node/jobs.html#.create) documentation for more details on the `--workspace` option and related options.
128
+
129
+
The `--ignoreFiles "<file-or-dir>,<file-or-dir>,..."` option can be used specify a simple comma separated list of files and directories to ignore for the workspace upload:
130
+
131
+
paperspace-python run myscript.py --workspace . --ignoreFiles "hello.py,paperspace"
132
+
133
+
The following files and directories are ignored by default: `.git`, `.gitignore`, `__pycache__`.
134
+
135
+
The `--dryrun` option allows you to see the resultant script that will be run on the paperspace job runner without actually running it.
136
+
137
+
Finally, you can provide a trailing list of arguments to pass to the script.
138
+
139
+
All of the above options can be combined in any combination, however, the order of operations is fixed to the following:
140
+
141
+
1.`source <init.sh>` is run if `--init <init.sh>` is specified
142
+
2.`pipenv [--two|--three] install` is run if `--pipenv` is specified
143
+
3.`pip[2|3] install -r requirements.txt` is run if `--req <requirements.txt>` is specified
144
+
4.`python[2|3] myscript.py` is run
145
+
146
+
147
+
Default Container
148
+
=================
149
+
If no `container` option is specified when using `paperspace run <script.py>` or the `paperspace.run()` function the default container image used is `paperspace/tensorflow-python` on Docker Hub. This container has the tensorflow-gpu libraries installed for both python2 and python3, as well as several other popular packages, including scipy, scikit-learn, pandas, Pillow and pysym.
150
+
It is based off the Google docker image gcr.io/tensorflow/tensorflow:latest-gpu` with the addition of support for python3, pip3, and pipenv.
151
+
152
+
57
153
Other examples
58
154
==============
59
-
See the scripts in the test folder for other examples.
155
+
See the scripts in the `tests` folder for other examples.
60
156
61
157
62
158
Other Authentication options
@@ -78,6 +174,10 @@ Other Authentication options
78
174
Note: the above methods take precedence over use of the cached api key in
79
175
`~/.paperspace/config.json`
80
176
177
+
4. Set the key in the `~/.paperspace/config.json` file from the command line by running:
0 commit comments