Perform CR4 and CR0 pinning from hypervisors

[kees]

Similar to the recent CR4 and CR0 pinning done in the native_write_cr*() functions, it would be even better to have KVM (and other hypervisors) pin those bits for their guest kernels too.

[johnandersen777]

Hi. I've been working on this here: torvalds/linux@master...pdxjohnny:hc_harden almost ready to send out for review.

[kees]

Ah! Very cool. I wonder, though, if it would be better to make this entire a host based defense? i.e. do not involve the guest at all and just detect when special bits are set and keep them set? Or maybe, have a default set of bits and let the guest add more? (I'd like this defense to work "out of the box" for all guests without having the guest need to know to make a hypercall to gain the protection.)

[johnandersen777]

Moving discussion here (CC: @redgecombe @kaccardi)

With regards to the Container as VM usecase.
I noticed that ChromeOS does Linux support in a VM with a hardened
config. Is this something that would be enabled there if implemented?
We're taking this through review internally and was told an "ack from
those [chromeos] folks would go a long way"

Ah! Very cool. I wonder, though, if it would be better to make this entire a host based defense? i.e. do not involve the guest at all and just detect when special bits are set and keep them set? Or maybe, have a default set of bits and let the guest add more? (I'd like this defence to work "out of the box" for all guests without having the guest need to know to make a hypercall to gain the protection.)

I like this idea. I'm not sure how we could convince upstream to take that though, so far when we'd thought about that we'd come to the conclusion that we don't know what other kernels people might be running under KVM that would do whacky things like flip those bits intentionally. In which case the KVM people might not be so keen on us breaking functionality for those.

But minimally I'd guess that patch would pretty much be:

int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
{
     unsigned long old_cr4 = kvm_read_cr4(vcpu);
     unsigned long pdptr_bits = X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE |
                                X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_PKE;

     if (kvm_valid_cr4(vcpu, cr4))
             return 1;

     if (~cr4 & old_cr4 & (X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP))
             return 1;

    /* ... */

We could make some sort of IOCTL to have QEMU, etc. request that we run in a once-turned-on-never-turn-off mode. The downside to this is that now we have to go touch QMEU, firecracker, etc. Whereas the hypercall approach is pretty self contained within the kernel (aside from the one line QEMU cpuid bit patch). I'm not sure what the most upstreamably friendly approach his here though.

As it stands right now the patch is a hypercall that allows for pinning SMEP, SMAP, UMIP, and WP. The same bits you had already pinned. The initial approach had been to allow pinning of arbitrary bits. That was met with skepticism over the TS and MP bits in CR0. So I think it might be easiest to upstream if we first allow for only the pinning of the bits we already pin in software and then we could add the ability to pin others if we care to do that later in a different patch.

I'm also meeting with skepticism around KEXEC. I think I'll try to debug what's up with the early boot code clearing those bits to see if we can make this an always on option. If it works with KEXEC, then we probably don't need a CONFIG_ for it, since we can argue that it's functionally equivalent to existing pinning and therefore should be on by default.

[johnandersen777]

Kexec is working!

[johnandersen777]

Well shit, I spoke too soon. The patch I have to the early boot code works in KVM, but not on my physical host... If I can't get it working by mid next week I'll just keep the Kconfig option and make it !KEXEC for the public RFC.

CR0.WP and CR4.SMEP work, still trying to narrow down what's causing it not to boot when on the physical host (never makes it to first console prints to screen, haven't hooked up serial).

Setting CR4.UMIP or CR4.SMAP causes it not to boot. I think this is because of EFLAGS.AC. Either way it's wednesday and so I'm giving up on Kexec for now :/

Also, I finally ran across the code that's really supposed to be enabling SMAP / UMIP and I see that surprise surprise there is a well thought out existing process for enabling them. It was also pointed out that if we kexec to a new kernel the new kernel might be an older kernel that doesn't know it shouldn't turn off those bits. In which case it would blow up. So the boot time disablable kexec is probably the safest and most usable option long term.

[johnandersen777]

Just an update. Almost done addressing all RFCv1 comments to send out RFCv2. One of the comments was that we need to ensure SMIs can't modify pinned bits. Which lead to the discovery that AMD and Intel SDMs specify that the CR* fields within SMRAM are readonly. So we shouldn't be able to modify them in the first place. Should be done with the patch to make sure they are readonly soon. It'll be the new first patch in the pv cr pinning patchset.

[johnandersen777]

Update, PATCH sent: https://lkml.org/lkml/2020/6/17/921

[johnandersen777]

Just an update on this. Intel has a review process that I'm gated by right now. I have the next version ready but haven't been able to get a review. https://github.com/pdxjohnny/linux/tree/patch_v2_internal_v6

[johnandersen777]

If someone wants to take this and put a co-authored by on it that works for me. Otherwise I probably need another probably 6 months to get this rebased and out the door, provided there are no major issues with the rebase. I've been working on getting Intel proper CI/CD so that the review process can scale.

[bwendling]

I can give a shot at this.

[johnandersen777]

@gwelymernans Want to have a meeting sometime I can go over the patchset as it was while ago and the test cases?

[bwendling]

@pdxjohnny That's a good idea. :-) Are you available Wednesday? My schedule's fairly open then or later in the week.