Commit a384a17c authored by David Yu Yuan's avatar David Yu Yuan
Browse files

Merge branch 'latest' into 'master'

Latest to master

See merge request !124
parents 56e0f914 d1c7dc3b
Pipeline #41523 passed with stages
in 1 minute and 40 seconds
......@@ -12,14 +12,19 @@ Kubeflow can run on any environment with Kubernetes. If it is used for ML, model
1. Create a GKE cluster. Choice of a zone is important if GPUs are needed. See `GPU availability at europe-west`_ for GPU accelerators in europe-west.
2. Follow instructions at https://www.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/ to deploy Kubeflow. It requires a Kubernetes cluster with LoadBalancer support. Create MetalLB if needed.
3. Get the IP address and open Kubeflow dashboard.
3. Get the credentials for the newly created cluster `gcloud container clusters get-credentials ${CLUSTER} --zone ${ZONE} --project ${PROJECT}`.
4. Get the IP address and open Kubeflow dashboard.
Accessing Kubeflow with the following commands::
gcloud container clusters get-credentials ${CLUSTER} --zone ${ZONE} --project ${PROJECT}
IP_KUBEFLOW=$( kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}' )
open https://${IP_KUBEFLOW}
Alternatively, use port forward on local host::
kubectl port-forward -n istio-system svc/istio-ingressgateway 8443:443
open https://localhost:8443
Setting up Jupyter notebook
---------------------------
......@@ -104,6 +109,17 @@ As documented in https://www.kubeflow.org/docs/pipelines/sdk/component-developme
This API fits for relatively small files in GB range if from cloud storage. There is no good solution for large files in TB range and large volume in PB range.
Accessing data via OneData
--------------------------
OneClient can be installed. However, it can not be run without root privilege. It is a serious security risk to run a container with root privilege. Kubeflow will never agree to that. The integration of OneData with Kubeflow is out of the question::
conda install -c onedata oneclient=18.02.2
oneclient --help
oneclient: error while loading shared libraries: libprotobuf.so.19: cannot open shared object file: No such file or directory
The only sensible option is for OneData to provide a provisioner as other storage vendors https://github.com/kubernetes-incubator/external-storage, where a common library by Kubernetes SIG is available at https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner.
GPU quotas
----------
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment