Set up Local DSP Dev Env

Prerequisites

• Have a Dev OCP 4.11+ cluster with cluster admin
• Be logged in to cluster as cluster admin via OC
• kubectl

sudo yum install -y kubectl

• yq

wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
  tar xz && mv ${BINARY} /usr/bin/yq

Setup

Setup env

DEV_SETUP_REPO=dsp-dev-setup
DSP_REPO=data-science-pipelines
DSPO_REPO=data-science-pipelines-operator
DSPA_NS=dspa
OPERATOR_NS=odh-applications
git clone [email protected]:HumairAK/dsp-dev-setup.git ${DEV_SETUP_REPO}
git clone [email protected]:opendatahub-io/data-science-pipelines.git ${DSP_REPO}
git clone [email protected]:opendatahub-io/data-science-pipelines-operator.git ${DSPO_REPO}

Deploy a DSPO

pushd ${DSPO_REPO}
oc new-project ${OPERATOR_NS}
make deploy OPERATOR_NS=${OPERATOR_NS}
popd

Deploy DSPA

pushd ${DSPO_REPO}
oc new-project ${DSPA_NS}
oc -n ${DSPA_NS} apply -f config/samples/dspa_simple.yaml
popd

Locally run each component

Based on the components you want to run locally, the steps are different. For each component, assume a fresh dspo/dsp install from Setup instructions above.

API Server

Scale down api server:

oc -n ${DSPA_NS} scale --replicas=0 deployment/ds-pipeline-sample

cd ${DEV_SETUP_REPO}
mkdir output
./main.sh ${DSPA_NS} sample /home/${USER}/.kube/config output

This will generate all the files required to configure API Server deployment.

API Server creates a k8sclient connection using in cluster config, so to emulate this, the above script pulls cluster certs, and your OC user token in the output folder. Please don't paste the contents of this folder anywhere public for your own security.

Copy the creds to the following locations on your file system:

sudo mkdir -p /var/run/secrets/kubernetes.io/serviceaccount
sudo cp output/ca.crt /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
sudo cp output/token /var/run/secrets/kubernetes.io/serviceaccount/token

sudo mkdir -p /var/run/secrets/kubeflow/tokens
sudo cp output/token /var/run/secrets/kubeflow/tokens/persistenceagent-sa-token

This simulates an in cluster config similar to what API Server sees, when running inside a Pod.

API Server needs to hook into the db and mlmd, we can port forward both services to allow this:

cd ${DEV_SETUP_REPO}
./output/forward-db.sh
./output/forward-mlmd-grpc.sh

Now run API Server

export $(cat output/vars.env | xargs)

cd ${DSP_REPO}
go build -o bin/apiserver backend/src/apiserver/*.go
./bin/apiserver --config=../${DEV_SETUP_REPO}/output --sampleconfig=../${DEV_SETUP_REPO}/output/sample_config.json -logtostderr=true

Persistence Agent

To run PA run the following:

Scale down:

oc -n ${DSPA_NS} scale --replicas=0 deployment/ds-pipeline-persistenceagent-sample

Export env vars:

export $(cat output/vars.env | xargs)

Build and run:

go build -o bin/pa backend/src/agent/persistence/*.go

./bin/pa --kubeconfig=/home/hukhan/.kube/config \
    --master=https://api.hukhan-3.dev.datahub.redhat.com:6443 \
    --mlPipelineAPIServerName=localhost \
    --mlPipelineServiceHttpPort=8888 \
    --mlPipelineServiceGRPCPort=8887 \
    --namespace=${DSPA_NS}

Current issues / workarounds

kfp-driver

Currently the kfp-driver creates a cache client that has ml-pipeline service hardcoded, which means it requires an end point to the k8s api server service named ml-pipeline. Work around is to tunnel the local api server using an app like ngrok, and sub that endpoint into the cache client's cacheDefaultEndpoint() in cache.go. Then rebuild kfp-driver and sub this image in the api server's env var V2_DRIVER_IMAGE=your/image.

persistence agent

Persistent agent seems to keep encountering transient error due to PROTOCOL_ERROR when trying to report workflows to the api server grpc endpoint. No clue what the issue is here.

Troubleshooting

Go Env issues

Configure go env

go env -w GO111MODULE=on
go env -w GOPROXY="https://proxy.golang.org,direct"