ci-release: no space on pi device

[BethGriggs]

When running the builds for v8.16.1 I am hitting:

21:04:22 Building remotely on release-requireio_rvagg-debian9-armv6l_pi1p-1 (pi1-docker pi1-raspbian-stretch) in workspace /home/iojs/build/ws
...
21:04:23 java.nio.file.FileSystemException: /home/iojs/.jenkins/cache/jars/5C: No space left on device
21:04:23 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
21:04:23 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

https://ci-release.nodejs.org/job/iojs+release/nodes=pi1-docker/4633/console

[sam-github]

I can't ssh into that machine:

build/hosts (build-10-with-devtoolset-s390x $%) % ssh release-requireio_rvagg-debian9-armv6l_pi1p-1                                                                          
zsh:1: parse error near `}'
ssh_exchange_identification: Connection closed by remote host
build/hosts (build-10-with-devtoolset-s390x $%) % grep -A 5 release-requireio_rvagg-debian9-armv6l_pi1p-1 ~/.ssh/config                                                      
Host release-requireio_rvagg-debian9-armv6l_pi1p-1
  HostName 192.168.2.40
  IdentityFile ~/.ssh/nodejs_build_release
  User pi
  ProxyCommand ssh -i {{ ansible_ssh_private_key_file }} -W %h:%p -p 2222 jump@vagg-arm.nodejs.org                                                                           

It looks like the .ssh/config has some ansible garbage in it, so I hacked it up, worked a bit better, but still wants a password:

 build/hosts (build-10-with-devtoolset-s390x $%) % ssh release-requireio_rvagg-debian9-armv6l_pi1p-1                    
The authenticity of host '[vagg-arm.nodejs.org]:2222 ([203.217.18.114]:2222)' can't be established.
ECDSA key fingerprint is SHA256:cQOh+H/IRjq/rFvpoqcgfFfneK7H2FRqo69rfzeS+Bs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[vagg-arm.nodejs.org]:2222,[203.217.18.114]:2222' (ECDSA) to the list of known hosts.
jump@vagg-arm.nodejs.org's password: 

build/hosts (build-10-with-devtoolset-s390x $%) % grep -A 5 release-requireio_rvagg-debian9-armv6l_pi1p-1 ~/.ssh/config
Host release-requireio_rvagg-debian9-armv6l_pi1p-1 
  HostName 192.168.2.40
  IdentityFile ~/.ssh/nodejs_build_release
  User pi
  ProxyCommand ssh -i ~/.ssh/nodejs_build_release -W %h:%p -p 2222 jump@vagg-arm.nodejs.org

[sam-github]

I tried directly sshing into jump, too, but either I have the wrong ssh command, or don't have access.

[mhdawson]

I can get into the jump box, but trying to use as per the config files I just get:

user1@user1-lbuntu:~/.ssh$ ssh release-requireio_rvagg-debian9-armv6l_pi1p-1
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

[BethGriggs]

The pi is now offline in ci-release Jenkins with:
Disk space is too low. Only 0.000GB left on /home/iojs/build

[mhdawson]

I've kind of run out of ideas. I can get on the jump box but it won't let me run ping to see if the machines are reachable, and ssh does not seem to be installed so I can't try to ssh from there to one of the machines either.

At this point I think we need to release for the rest of the platforms without ARMv6 and then wait for @rvagg to help fix the release machines.

[rvagg]

Awake and on it, full disk and something weird with jump host, will address them both.
FYI half of the Pi's operate on one dedicated disk from the nfs host and the other half on another disk, so a full disk shouldn't take them all out of action—but will likely make the cluster a mess.

[rvagg]

OK, that was a bit traumatic. Raspbian seem to have released a firmware earlier this month that prevented most of them from booting and a fix was out just recently but they didn't get that fix (because I wasn't aware of this problem) before I shut them all down .. bringing them back was a bit tedious and individual. Most of them are back now and I'm running some CI jobs to make sure they are operating properly.

WRT ssh access, I've updated the ProxyCommand in #1890 to something that should work (I updated the jump host recently and did some security modifications but didn't notice it stopped -W from working anymore since I don't use that, mostly). But I still don't know why {{ ansible_ssh_private_key_file }} ends up in there, it's been doing that for some time now and nobody's spent time trying to work out what broke.

[rvagg]

@BethGriggs after a couple of reruns we got it warmed up and producing binaries again, you'll find a v8.16.1: linux-armv6l sitting in staging for you ready to go when you're around. Sorry for the dramas.