We said that the maximum amount should be 20 OSDs per instance - that would correspond to ~40 CPU Units and ~100GB RAM (for a OSD-only node) - this is the reason why 128GB RAM is the highest choice. If the node is larger, there would be no change in the solver calculation.

That is intentional, because we want to allocate enough resources to allow for failover. Thus we allocate additional resources for everything that is not fault-tolerant by itself (like the RGW or MONs) even though that will not be consumed immediately. We do that to ensure the cluster is properly sized.

What you see is the difference between the node's available resources versus the resources that are consumed by OCS services. If these two differ, you know that there is room to put additional non-OCS workloads on these nodes if you decide not to run infra nodes.

This was a deliberate decision. Especially for bare metal deployments, customers would want to fill up nodes, before they add new ones. If not all nodes are full, they would want to have homogenous deployments and one "spillover" node, which they could easily fill up later on (instead of going to every single server and adding a disk)

This OCS solver does not add OCP base services by default, because we do not know that overall OCP cluster size. Under normal circumstances you will need an additional 0.5 CPU Units and 1GB RAM for OCP base services.