Leveraging a docker-compose file in a devfile definition

[elsony]

Use Case

As a developer working on an existing project, I have already setup my environment and built a docker-compose file which defines all the different components that I need to test my application.
I want to leverage that docker-compose in my devfile - so I can make my developer environment codified/versioned/repeatable and keep using my docker-compose as well.

Goals

It's really common for a project to get a docker-compose file. In general this is a convenient way for the developers to more easily setup their application and all its components on their local environment. It can also be used along with different containers to completely build the source code of each application components and deploy those. In this case, there might be different dockerfiles associated with each application components which will trigger a specific "task".

The goal of the effort, is to tie the docker-compose file with the devfile. So if a developer knows how to build a docker-compose or already have one, it'll get easier to onboard on devfile and leverage work and effort that has already been produced. More importantly, it would let somebody continuing to evolve the devfile and the dockercompose so they complement each others.

Examples

There are many different samples we could be looking at:

• https://www.digitalocean.com/community/tutorials/containerizing-a-node-js-application-for-development-with-docker-compose

One that might be even more interesting to look at would be this one https://github.com/RediSearch/redisearch-getting-started:

• Inside of sample-app, there is a dockercompose file
• Each application components are getting built from a defined dockerfile
• Environments variables are used to connect the services

Steps / Challenges

1- Docker-compose file with existing containers

Let's take the case that the docker-compose file is only relying on containers that are already built and available in the container catalog (docker-compose not relying on dockerfiles). We should be able to use the docker-compose file and use its services as components for the devfile.

Targets:

• We can start by having a tool, that reads the docker-compose file and convert that as a devfile.
• We would need to formalize an improvement to the specification, so that it is possible to link devfile components from the docker-compose file - so it'll not be a "convertion" of the docker-compose file to devfile, but a way to use the docker-compose and the devfile together.
• We code the API that allows to do that.

2- Docker-compose file with to-be-built containers

In this case, we need to ensure that the devfile/api will have the ability to build the containers and provide them to the developers.

Targets:

• We should look at how we can do the build of the containers on the cluster (podman, sandboxed containers... )

[elsony]

Implementation plan:

1. Docker-compose file with existing containers
   1. Create a library as part of the existing devfile library for converting a given docker-compose to existing segment(s) of the devfile model that maps the docker compose elements
      • convert something like this in a compose file to a container component:

       db:
         image: mongo:4.1.8-xenial
         container_name: db
         restart: unless-stopped
         env_file: .env
         environment:
           - MONGO_INITDB_ROOT_USERNAME=$MONGO_USERNAME
           - MONGO_INITDB_ROOT_PASSWORD=$MONGO_PASSWORD
         volumes:
           - dbdata:/data/db
         networks:
           - app-network
      

      • We will document and limit the number of elements that we support. We are not planning to support elements that do not map well to our existing devfile model, e.g. networks and restart.
      • We can refer to https://github.com/kubernetes/kompose that may help the implementation
   2. Update the merged model in the devfile library: for devfiles with the docker-compose component, convert the docker-compose component to other existing components using the library above, e.g. container components, and replace the docker-compose component with those generated components in the merged model
   3. Add a way to refer to a docker compose file from the devfile, e.g. a docker-compose component element
2. Docker-compose file with to-be-built containers
   1. Prereq: Add a way to refer to a docker compose file from the devfile, e.g. a docker-compose component element (from above)
   2. Add conversion of Dockerfile build to a dockerfile image component to the conversion function in the library (see above), e.g.

   services:
     nodejs:
       build:
         context: .
         dockerfile: Dockerfile
        image: nodejs
        container_name: nodejs
        restart: unless-stopped
        env_file: .env
        environment:
          - MONGO_USERNAME=$MONGO_USERNAME
          - MONGO_PASSWORD=$MONGO_PASSWORD
          - MONGO_HOSTNAME=db
          - MONGO_PORT=$MONGO_PORT
          - MONGO_DB=$MONGO_DB
        ports:
          - "80:8080"
        volumes:
          - .:/home/node/app
          - node_modules:/home/node/app/node_modules
        networks:
          - app-network
        command: ./wait-for.sh db:27017 -- /home/node/app/node_modules/.bin/nodemon app.js
   

   • In the library above, covert the docker-compose element As part of the outer loop support, a new image dockerfile component will be added (Introduce outerloop 'dockerfile' build & deploy #51). We can convert to that image dockerfile component
   • As part of Inner loop dockerfile support #346, it provides an inner loop container definition that allows the image to be built from a dockerfile, we'll use the same mechanism for this conversion
   • We will document and limit the number of elements that we support. We are not planning to support elements that do not map well to our existing devfile model, e.g. networks and restart.

[Prateeknandle]

Hey @l0rd, I've applied for LFX summer'22 in this project, although I understood the project and its requirements but I'm new to devfile so if I got selected(I hope i get) then initailly I'll start with some issues to understand the core project and meanwhile discussing the implementation ideas also....:)
....thnx

[kernelpanic77]

Hello! I will work on this issue.

[l0rd]

@kernelpanic77 is going to work on this issue as part of the LFX Summer Mentorship program

[kadel]

Can we please pause and first think and define how this is going to work?

Can we please first create some examples and show how this going to work with existing compose files?

How is the innerloop going to work with docker-compose files? Existing compose files are not designed to work as odo or Che works in innerloop. Every existing compose file assumes that you rebuild the image every time you want to see changes in your application!!

There are more questions that we need to answer. It looks like we want to make docker-compose file as kind of extension to the devfile. In this case, we must define what we will do with the overlapping containers.

# docker-compose.yaml
version: "3.9"
services:
  web:
    image: "foo"

# devfile.yaml
schemaVersion: 2.1.0
metadata:
  name: foo
components:
  - name: compose
    docker-compose:
      uri: docker-compose.yaml
  - name: web
    container:
       image: bar
commands:
  - id: run
    exec:
      component: web
      commandLine: run.sh

what should we do in this case where there is a container called web defined in both the devfile and compose file? Should this error be out, or is this a valid scenario? If it is valid, then where the run command should be executed?

[l0rd]

@kadel that sounds like a rant but I can assure you that @kernelpanic77 has been in a "pause and think" mode since the beginning 😄. I don't see any risk of adding anything to the spec without you or others review and approval.

First we are not going to merge anything in the spec until we have a working PoC (most probably DevWorkspace based). As an example we have been looking at the microservices-demo.

Second we don't want to start with a comprehensive support of the composefile spec. We want to implement a basic support first and, then, incrementally improve it (for instance the Docker-compose file with to-be-built containers mentioned by @elsony above was out of scope).

To answer your questions:

How is the innerloop going to work with docker-compose files? Existing compose files are not designed to work as odo or Che works in innerloop. Every existing compose file assumes that you rebuild the image every time you want to see changes in your application!!

That's not different from the already supported kubernetes/openshift components isn't it? And, from a Che point of view, a Devfile defines a remote development environment. Developers should decide if building an image is part of their inner loop or not at their convenience. As they do locally. I think that should apply to odo as well.

We must define what we will do with the overlapping containers.

I think that we should be consistent with kubernetes/openshift components. But I am afraid that the spec doesn't clarify the behavior of overlapping containers with kubernetes/openshfit components :-(. If that's the case I would vote for the simplest: don't support it.

[Ankita13-code]

Hi! I'd like to contribute to this issue as a part of LFX 2022 Fall term. Any pointers on how to get started with this?