openldap exit status 1 on Kubernetes with persistent volume

[cetsupport]

Trying to run openldap container over Kubernetes 1.8 with persistent volumes by NFS.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: scnea
  name: ldap
  labels:
    app: ldap
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ldap
    spec:
      containers:
        - name: ldap
          image: localhost:30001/openldap:1.1.10
          args: ["--copy-service"]
          volumeMounts:
            - name: ldap-persistent-storage-data
              mountPath: /var/lib/ldap
            - name: ldap-persistent-storage-config
              mountPath: /etc/ldap/slapd.d
            - name: ldap-persistent-storage-certs
              mountPath: /container/service/slapd/assets/certs
          ports:
            - containerPort: 389
              name: openldap
          env:
            - name: LDAP_LOG_LEVEL
              value: "256"
            - name: LDAP_ORGANISATION
              value: "Example"
            - name: LDAP_DOMAIN
              value: "example.com"
            - name: LDAP_ADMIN_PASSWORD
              valueFrom:
              secretKeyRef:
                name: ldap-auth
                key: adminPsw
            - name: LDAP_CONFIG_PASSWORD
              valueFrom:
              secretKeyRef:
                name: ldap-auth
                key: configPsw
            - name: LDAP_READONLY_USER
              value: "false"
            - name: LDAP_READONLY_USER_USERNAME
              value: "readonly"
            - name: LDAP_READONLY_USER_PASSWORD
              value: "readonly"
            - name: LDAP_RFC2307BIS_SCHEMA
              value: "false"
            - name: LDAP_BACKEND
              value: "hdb"
            - name: LDAP_TLS
              value: "false"
            - name: LDAP_TLS_CRT_FILENAME
              value: "ldap.crt"
            - name: LDAP_TLS_KEY_FILENAME
              value: "ldap.key"
            - name: LDAP_TLS_CA_CRT_FILENAME
              value: "ca.crt"
            - name: LDAP_TLS_ENFORCE
              value: "false"
            - name: LDAP_TLS_CIPHER_SUITE
              value: "SECURE256:+SECURE128:-VERS-TLS-ALL:+VERS-TLS1.2:-RSA:-DHE-DSS:-CAMELLIA-128-CBC:-CAMELLIA-256-CBC"
            - name: LDAP_TLS_VERIFY_CLIENT
              value: "demand"
            - name: LDAP_REPLICATION
              value: "false"
            - name: LDAP_REPLICATION_CONFIG_SYNCPROV
              value: "binddn=\"cn=admin,cn=config\" bindmethod=simple credentials=$LDAP_CONFIG_PASSWORD searchbase=\"cn=config\" type=refreshAndPersist retry=\"60 +\" timeout=1 starttls=critical"
            - name: LDAP_REPLICATION_DB_SYNCPROV
              value: "binddn=\"cn=admin,$LDAP_BASE_DN\" bindmethod=simple credentials=$LDAP_ADMIN_PASSWORD searchbase=\"$LDAP_BASE_DN\" type=refreshAndPersist interval=00:00:00:10 retry=\"60 +\" timeout=1 starttls=critical"
            - name: LDAP_REPLICATION_HOSTS
              value: "#PYTHON2BASH:['ldap://ldap-one-service', 'ldap://ldap-two-service']"
            - name: KEEP_EXISTING_CONFIG
              value: "false"
            - name: LDAP_REMOVE_CONFIG_AFTER_SETUP
              value: "true"
            - name: LDAP_SSL_HELPER_PREFIX
              value: "ldap"
      volumes:
      - name: ldap-persistent-storage-config
        persistentVolumeClaim:
          claimName: nfs-pvc-ldap-config
      - name: ldap-persistent-storage-data
        persistentVolumeClaim:
          claimName: nfs-pvc-ldap-data
      - name: ldap-persistent-storage-certs
        persistentVolumeClaim:
          claimName: nfs-pvc-ldap-certs

The error I got:

*** CONTAINER_LOG_LEVEL = 3 (info)
*** Copy /container/service to /container/run/service
*** Search service in CONTAINER_SERVICE_DIR = /container/run/service :
*** link /container/run/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools
*** link /container/run/service/slapd/startup.sh to /container/run/startup/slapd
*** link /container/run/service/slapd/process.sh to /container/run/process/slapd/run
*** Set environment for startup files
*** Environment files will be proccessed in this order : 
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.startup.yaml
/container/environment/99-default/default.yaml
 To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
*** Running /container/run/startup/:ssl-tools...
*** Running /container/run/startup/slapd...
Start OpenLDAP...
*** /container/run/startup/slapd failed with status 1
 *** Killing all processes...

I modified the startup.sh, trying to create a test folder under "/var/lib/ldap" and "/etc/ldap/slapd.d". result is that those folders been create. This prove that NFS should work fine.

If add debug will get below information:

*** CONTAINER_LOG_LEVEL = 3 (info)
*** Copy /container/service to /container/run/service ignored
*** /container/run/service already exists
*** Search service in CONTAINER_SERVICE_DIR = /container/run/service :
*** link /container/run/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools
*** failed to link /container/run/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools: [Errno 17] File exists
*** link /container/run/service/slapd/startup.sh to /container/run/startup/slapd
*** failed to link /container/run/service/slapd/startup.sh to /container/run/startup/slapd: [Errno 17] File exists
*** link /container/run/service/slapd/process.sh to /container/run/process/slapd/run
*** directory /container/run/process/slapd already exists
*** failed to link /container/run/service/slapd/process.sh to /container/run/process/slapd/run : [Errno 17] File exists
*** Set environment for startup files
*** Environment files will be proccessed in this order : 
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.startup.yaml
/container/environment/99-default/default.yaml

To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
*** Running /container/run/startup/:ssl-tools...
*** Running /container/run/startup/slapd...
Create folders
Start OpenLDAP...
Waiting for OpenLDAP to start...
5a21e04c @(#) $OpenLDAP: slapd  (Aug 10 2017 19:12:46) $
        Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
5a21e04c could not stat config file "/etc/ldap/slapd.conf": No such file or directory (2)
5a21e04c slapd stopped.
5a21e04c connections_destroy: nothing to destroy.

Any suggestion how to fix this issue? As from document, the /etc/ldap/slapd.conf should be depreciated.

Not sure if possible this problem caused by NFS export folder have .rmtab to record NFSclient information. Should we add "-I .rmtab" to ignore this file, then startup.sh can create those initial files.

Confirmed that the problem was caused by above issue. PR submitted.

[BertrandGouny]

This will be fixed by #169 ?

[msdotnetclr]

I am having a similar issue but not with NFS, I am using Azure File volume. Not sure if this will be fixed bu #169

--loglevel debug output:
Start OpenLDAP...
Waiting for OpenLDAP to start...
5a309df7 @(#) $OpenLDAP: slapd  (Aug 10 2017 19:12:46) $
        Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
5a309df7 ldif_read_file: Permission denied for "/etc/ldap/slapd.d/cn=config.ldif"
5a309df7 slapd stopped.
5a309df7 connections_destroy: nothing to destroy.

[BertrandGouny]

@msdotnetclr can you try version 1.1.11 and let us know ?
thanks

[msdotnetclr]

I tried version 1.1.11 last night and got the same result. On Dec 13, 2017 5:15 AM, "Bertrand Gouny" <notifications@github.com> wrote: @msdotnetclr <https://github.com/msdotnetclr> can you try version 1.1.11 and let us know ? thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2-dfL6ITi22f4GG3xbEDVzvxGGVfFGks5s_6O2gaJpZM4QyfEB> .

[cetsupport]

For my case, this issue was fixed by the PR I submitted. @BertrandGouny , can you please share more detail?

[nelsg]

Got the same issue with 1.1.11, my logs:

Waiting for OpenLDAP to start...
+ '[' '!' -e /run/slapd/slapd.pid ']'
+ sleep 0.1
5b194639 @(#) $OpenLDAP: slapd  (Aug 10 2017 19:12:46) $
	Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
5b194639 daemon: bind(9) failed errno=99 (Cannot assign requested address)
+ '[' '!' -e /run/slapd/slapd.pid ']'
+ sleep 0.1
5b194639 ldif_read_file: Permission denied for "/etc/ldap/slapd.d/cn=config.ldif"
5b194639 slapd stopped.
5b194639 connections_destroy: nothing to destroy.

I my case I have a NFS mount. If I copy config files locally it works.

NFS on my host:

• NFS type is nfs4
• NFS Options: (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.2.3.4,local_lock=none,addr=1.2.3.5,_netdev)

First, I suspected selinux, but it's in permissive mode, not in enforcing and no warning in /var/log/messages file.
Furthermore security-context of configuration files is system_u:object_r:nfs_t:s0 (s0=SystemLow), nothing special.

Any idea ?

[nelsg]

I found a solution for me:

My container executes main process with root user and Nfs4 block root access on directories. I changed mount type from Nfs4 to Nfs3 to solve my problem.

[abinet]

Now I run into the same issue but using GlusterFS Volumes. They contain .trashcan folder.
My suggestion would be to ignore all the folders starting with .