EM: Add restart on-failure for metadata service

[surajssd]

This PR adds Restart=on-failure and RestartSec=5s to the metadata service.

Fixes #1298

---

The other services which have Type=oneshot are following: wait-for-dns, bootkube, delete-node, create-etcd-config, persist-data-raid.

Most of them are the type of services which we want them to fail early and the user know about it, instead of them endlessly trying and someone else doing a time out on them.

[knrt10]

Seeing this PR, I think we should slowly start changing Packet -> Equinx Metal

[invidian]

Seeing this PR, I think we should slowly start changing Packet -> Equinx Metal

We have #1060 :)

[invidian]

The other services which have Type=oneshot are following: wait-for-dns, bootkube, delete-node, create-etcd-config, persist-data-raid.

Most of them are the type of services which we want them to fail early and the user know about it, instead of them endlessly trying and someone else doing a time out on them.

If there is a DNS outage, I think we should retry wait-for-dns.service, as it will block starting kubelet.

delete-node.service we should also retry IMO, so the node does not go away unregistered. As the pods will stay assigned for long on this node and won't be re-scheduled.

2 points above also applies to other platforms, not only Packet.

[surajssd]

If there is a DNS outage, I think we should retry wait-for-dns.service, as it will block starting kubelet.
delete-node.service we should also retry IMO, so the node does not go away unregistered. As the pods will stay assigned for long on this node and won't be re-scheduled.

I don't know what the unforeseen consequences will be for adding retries for wait-for-dns and delete-node, but lets try. I can only see them doing indefinite attempt to reconcile leaving user stranded either at installation or while removing the node.

[surajssd]

I can wait until #1368 is merged.

[pothos]

One more thing to consider: The oneshot services will be triggered again when the service that depends on it is restarted. I would add RemainAfterExit=yes everywhere, too (but for the resolv.conf waiter it seems harmless if it's triggered on a kubelet restart).

[invidian]

Good point @pothos.

I can wait until #1368 is merged.

This is now merged.

[invidian]

@surajssd did you see @pothos comment before merging? Did you verify what he suggests?

[surajssd]

@surajssd did you see @pothos comment before merging? Did you verify what he suggests?

Yep, I thought it is to be done in a separate PR. This is now merged. kinda hinted me towards it is good to merge this one.

[surajssd]

Creating a new issue for what Kai has suggested.