pthread_setaffinity_np fails

[spatodia]

Hi,

I am running the pcm tool on Xeon Gold and I see the following error after ~2mins of runtime.

ERROR: pthread_setaffinity_np for core 2 failed with code 22
ERROR: pthread_setaffinity_np for core 3 failed with code 22
ERROR: pthread_setaffinity_np for core 4 failed with code 22
ERROR: pthread_setaffinity_np for core 5 failed with code 22
ERROR: pthread_setaffinity_np for core 6 failed with code 22
ERROR: pthread_setaffinity_np for core 7 failed with code 22
ERROR: pthread_setaffinity_np for core 8 failed with code 22
ERROR: pthread_setaffinity_np for core 9 failed with code 22
...

Has anyone run into this issue?

Regards,
Saurabh

[spatodia]

Seems to be the CPUs_allowed field was changed by the cpucounters.cpp file.

It looks like something is changing the cpu mask in the below code: lines 229-230

CPU_SET(core_id, &new_affinity);
pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &new_affinity);

[opcm]

@spatodia PCM pins threads to cores to speedup certain things. It seems the cores become unavailable after 2 minutes? Anybody offlined them?

When this happens, could you run

• "lscpu" to check if the cores are online
• "taskset -c <CORE_NR_FROM_THE_PCM_ERROR_MESSAGE> sleep 0" to check if the taskset has the same issue?

[spatodia]

Thanks opcm for your response.

>>> "lscpu" to check if the cores are online
With lscpu I get the below:
[\output]
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
Stepping: 7
CPU MHz: 829.171
CPU max MHz: 3900.0000
CPU min MHz: 800.0000
BogoMIPS: 4191.73
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku flush_l1d arch_capabilities
[\output]
Seems all 80 logical cores are online.

>>> "taskset -c <CORE_NR_FROM_THE_PCM_ERROR_MESSAGE> sleep 0" to check if the taskset has the same issue?
[output]
taskset -c 2 sleep 0
taskset: failed to set pid 0's affinity: Invalid argument
[\output]
Taskset seems to see the same issue.

The interesting part is somehow cores 0-1,40-41 stay online while all rest are offlined by pcm
Cpus_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000300,00000003
Cpus_allowed_list: 0-1,40-41

Any ideas why from 80 online cores pcm keeps only 4 online? Is there a way to turn off pinning in PCM for profiling? I dont care much abt speed.

[opcm]

pcm does not change the cpus_allowed_list. It sets affinity temporarily for some threads. Something else on your system restricts the cpus_allowed list after two minutes. As you see the taskset is also impacted. Maybe it is a Linux bug... I realize for some code in PCM is setting affinity for correct functioning (for example when retrieving topology ids from a core using CPUID instruction it needs to make sure it runs on this core). You can disable the error check in the code but it might not work correctly anymore.

[spatodia]

Hmm.. When I reboot the system below is the cpu_allowed_list.

Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-447

Also if I do not run pcm - this never changes during and after my test. However, when I do run PCM parallely with the test cpus_allowed_list does change after 2 mins (runs fine till then) to below.
Cpus_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000300,00000003
Cpus_allowed_list: 0-1,40-41

So, its definitely something to do with running PCM along with the workload (as I only see the issue then).

[opcm]

definitely pcm process is not touching cpu_allowed_list for other programs (like taskset). Is there any administrator/cgroup scipt running?

cat /sys/fs/cgroup/cpuset/cpuset.cpus

[opcm]

did you find the root cause?

[opcm]

I am going to close the issue because it is not caused by PCM