We have five GPU enabled machines, and they are not the same machines. What
we want to do is to make a make a virtual cluster with these machines and these
serve Jupiter notebooks on top of it, so that when somebody in the lab needs to
run a machine learning experiment, they don’t necessarily have to either connect
directly through SSH to these computers or sit in front of them, but they can
just go to a single URL. Open a new Jupiter notebook which
automatically gets assigned one of the available GPU’s and they can run their
experiments accordingly.
So, Jupiter hub is the Jupiter projects whose
work is to. Share computational environments and resources it’s still.
However, we do need to establish a Kubernetes cluster
first. So, we need to have a Kubernetes cluster with several number of machines
learning related add on configured and then configured Jupiter hub on the
cluster.
We also need to add authentication and
prioritization to this solution, so we want to have a user management system
username and passwords are needed.
We also want certain users to have higher
priority than other users in accessing resources, so if let’s say I hire an
undergraduate student who needs to train. A deep learning model and at the same time, I
have Nancy rushing against a paper submission deadline, I want Nancy as a PhD
student to have higher priority in terms of access to gpus than my undergraduate
student.
So, user management, as well as prioritization
resource access prioritization these two will probably require a little bit of
programming and implementation.
If there may be open-source solutions out
there, on top of Jupiter hub or Kubernetes that already handled as far as I’m
not aware of any, but the rest is relatively straightforward, have you ever set
up a Kubernetes Cluster on Prem computers, not on the cloud. Computer so
on Prem is a different story it’s a little more different Okay.
it’s a cluster of all computing resources in all
our five GPU enabled computing systems, so there will be, I think.
In terms of scheduling, we are not going to put
a time limit on how long your task can run.
So, we have different types of gpus in
different computers.
Last Completed Projects
| topic title | academic level | Writer | delivered |
|---|