CRC/Blake3 using multiple cores?

[sirksel]

I have a lot of files that I check often for integrity, or generate new Blake3 hashes. Is there an ini setting or something to make TC use all the available cores for this? Seems like the bottleneck I'm experiencing isn't really disk speed. Also, there are at least 10 cores (20 threads) sitting idle. I'm using an AMD Ryzen 9 (16 cores, 32 threads). I realize there are limits to paralleling tasks, but from what I read about Blake3, it seems like it's even more able to be paralleled than its predecessor algorithms. True?

Anyhow my questions are twofold:
(1) Is there a way to get TC to use more cores for CRC and/or other commands like file compare?
(2) If not, are there external utilities you all recommend for Blake3 that more efficiently utilize a multi-core architecture?

Many thanks!

[Fla$her]

I don't know about Blake3, but you can take FCHash with fast xxh and xxh3. I don't know how multithreading is implemented there, but it can be implemented using nircmd setprocessaffinity.

[Dalai]

[...] but it can be implemented using nircmd setprocessaffinity.

That's not how it works. By default, a process runs on all cores/threads - unless a process is a child of a process with changed affinity. Running on all cores/threads doesn't mean that a process or an algorithm is multi-threaded. It can be but it doesn't have to be. Windows (and also Linux for that matter) move processes around on the available threads how the process scheduler sees fit. And that's the only thing affinity can change: limit the cores/threads available to move a process to. Limiting processes to certain cores/threads can be beneficial, but often it's not.

Regards
Dalai

[Fla$her]

By default, a process runs on all cores/threads - unless a process is a child of a process with changed affinity.

Are you saying that Christian deliberately imposed a restriction?

Running on all cores/threads doesn't mean that a process or an algorithm is multi-threaded.

That's understandable. But in the request it was about the processor threads, not the process.

[nsp]

Currently only rust version of Blake3 support multi_threading. Christian use C implementation and do not have // computation.

[Dalai]

By default, a process runs on all cores/threads - unless a process is a child of a process with changed affinity.

Are you saying that Christian deliberately imposed a restriction?

Uh, no. I don't see how my post could be read as such an implication. I just described how affinity works in general and what it can and can't do, nothing specific to TC.

Either a software or an algorithm in a software is written to make use of multiple threads, which the process scheduler then spreads across the available cores/threads, or it's not. Affinity can only limit the cores/threads available to a process, but it can't add anything.

I hope this post clears things up.

Regards
Dalai

[Fla$her]

Uh, no. I don't see how my post could be read as such an implication.

Not a post, but a quote given, taking into account the initial complaint.

Affinity can only limit the cores/threads available to a process, but it can't add anything.

OK. How can this be checked except for a heavy load?
I see threading distribution by processes only in System Idle Process.

[Dalai]

OK. How can this be checked except for a heavy load?

The easiest thing is to look in Windows Task Manager. There you can see the CPU usage for each process, and also inspect and change process affinities of running processes. Task Manager can also show how many threads a process currently has (after enabling the corresponding column). Note that having more than one thread doesn't necessarily mean that its load is spread across multiple available cores/threads, nor does it imply that the process can make use of more than one core. It's just an indication that it might be able to do so. Having just one thread does imply that it can't be multi-threaded, though.

Here's a simple example:
Pack some large file in TC using its built-in ZIP packer. You'll notice that the load doesn't stay on one core/thread. That's the scheduler working, moving threads around, preferably onto cores that have nothing to do. You'll also see that the overall CPU usage is roughly 100% divided by the number of available threads, e.g. around 13% for a quad-core system with SMT (100 divided by eight). Note: the CPU utility shown in Windows Task Manager on Windows 10 and later is something different than CPU usage.

Now change the affinity of totalcmd.exe/totalcmd64.exe by right-clicking on it in Task Manager. Assign the first two cores/threads for example and pack some file again. When doing so, you'll see that Windows won't move the process to other cores/threads anymore, and also that the load on the first two cores/threads will be much higher than previously. The overall CPU usage will stay roughly the same (which doesn't apply to all workloads, but that's a completely different story). The NirCmd command you gave does the same to the process affinity as Task Manager, only via command line.

Play around with different affinities and you'll get a feeling for how it works.

Note to moderators: It might be a good idea to split this deeper disussion into a new thread. And sorry for going off on a tangent.

Regards
Dalai

[Fla$her]

The easiest thing is to look in Windows Task Manager. There you can see the CPU usage for each process, and also inspect and change process affinities of running processes.

It would be strange to ask about this without checking Task manager. Only here's the catch - I don't have Win 10/11.

Task Manager can also show how many threads a process currently has (after enabling the corresponding column).

That's not what I asked for. I can observe the threads in many managers. I only asked about load distribution across processor threads.

You'll notice that the load doesn't stay on one core/thread.

I asked without a special load. Just check any current process with setprocessaffinity influence.

The NirCmd command you gave does the same to the process affinity as Task Manager, only via command line.

I believe, but Win 7 doesn't have that functionality.

[ghisler(Author)]

Indeed I'm using the C implementation of Blake3 because I have no experience with Rust and don't even know whether it can be used to create DLLs usable from C or Delphi. If they ever make a C version with multi-core support, I will use it.

[Dalai]

Only here's the catch - I don't have Win 10/11.
[...]
but Win 7 doesn't have that functionality.

Assigning affinities to processes in Task Manager works at least since WinXP, maybe even Win2k. So, yes, of course you can set affinities in Win7 Task Manager.

I asked without a special load.

It was an example of a workload that I know doesn't utilize all available cores/threads (in other words it is single-threaded), so you can see the Windows scheduler at work, moving threads around the available cores.

If a process doesn't generate enough load and/or doesn't create enough threads, it won't load the CPU to 100% or close to it. For single-threaded workloads the load won't be higher than 100% divided by the number of available cores/threads. It's that simple. If a process can do so - like 7-zip, WinRAR or the Total7zip plugin can - the load will be much higher than 100% divided by the number of available threads.

Regards
Dalai

[Fla$her]

So, yes, of course you can set affinities in Win7 Task Manager.

I agree, we just call it otherwise (accordance, compliance, conformance). But I wasn't interested in selection, but in viewing information.
However, nothing optionally changes if you mark fewer cores first, and then more. There is no block to the increase in threads.

If a process doesn't generate enough load and/or doesn't create enough threads, it won't load the CPU to 100% or close to it. For single-threaded workloads the load won't be higher than 100% divided by the number of available cores/threads. It's that simple. If a process can do so - like 7-zip, WinRAR or the Total7zip plugin can - the load will be much higher than 100% divided by the number of available threads.

This does not give an understanding of how exactly the load of a particular process is distributed across the cores/threads. And the number of processor threads for PercentProcessorTime from Win32_PerfFormattedData_PerfProc, paradoxically, has to be divided before exceeding 100%.

[Dalai]

However, nothing optionally changes if you mark fewer cores first, and then more. There is no block to the increase in threads.

What do you mean by "no block"?

This does not give an understanding of how exactly the load of a particular process is distributed across the cores/threads.

Well, if you want to to dive this deep, I guess you have to read about how the Windows scheduler works. I know very little about its internals and specifics. What I've been observing on the hardware I've worked with is what I've written in my previous posts here. Generally the scheduler tries to spread the load across all available cores/threads, more or less evenly. This has pros and cons.

But I didn't (and still don't) want to make it more complicated than necessary. So I'll leave it at that.

Regards
Dalai

[Fla$her]

What do you mean by "no block"?

You wrote in the second answer that it is impossible to increase the number of threads. I gave an example of such a possibility with a decrease and a reverse increase. That is, according to your logic, unmarked checkboxes should become locked.

Well, if you want to to dive this deep

No, no, I only asked you about the tool. You pointed out the one that I can't access because of the version of Windows. The rest is particular.

[Dalai]

You wrote in the second answer that it is impossible to increase the number of threads.

Yes, and this is still true. As I said, processes run on all available cores/threads by default. There can't be more than "all". Sure, after assigning a different affinity, i.e. by limiting a process to fewer than all cores/threads, the number of cores/threads a process runs on can be increased again.

But that won't change anything for single-threaded software/algorithms, and that's exactly what sirksel asked about, and Ghisler already confirmed. A single-threaded load can't be magically made faster or multi-threaded by assigning more cores to it; software must be specifically designed for this.

I gave an example of such a possibility with a decrease and a reverse increase. That is, according to your logic, unmarked checkboxes should become locked.

I don't understand your conclusion. Checkboxes for process affinity are only disabled for the cores a system doesn't have. For example: It doesn't make sense to enable a checkbox for core/thread 5 for a dual-core system (with or without SMT). Disabling unchecked checkboxes (which is what I think you mean by "become locked") after changing process affinity doesn't make sense because a user can change his/her mind and set a different affinity, moving a process to different core(s).

No, no, I only asked you about the tool.

It's just a different way to assign process affinities. The tool can't do more than the OS allows.

2sirksel
Sorry for moving the thread slightly off the rails. I hope you find it at least informative and/or entertaining.

Regards
Dalai