No! Not a dead-end. I realise now, that the negative "s" coefficient on gpf(n/d) (prime-limit) only occurs when you take the log of the primes as weights (thereby making it like Tenney height).
cmloegcmluin wrote: ↑Mon Jun 29, 2020 3:24 pm
The subscript a in the sopa
is to say that in this case I found that a logarithmic shape for the primes fit better. But a sublinear exponent fits better for the terms.
It's confusing, calling that log-base "a" too. I'm going to think of it as alpha "α". sopα
It's a pity alpha hardly looks different from "a", in the font I see it in here. But it probably won't matter because ....
If we really think both should be sublinear exponents, the best fit I could get was 0.0057104024991967, with k=0.66, a=0.62, y=0.858, s=0.192. In other words, when you switch to using a logarithmic weight on the primes, the prime limit has to do more work.
Yes I really think both p and r should have sublinear power functions applied, because when you apply a log to p you need that psychologically-implausible negative coefficient on the prime limit to counteract its apparent excess.
Part of me feels like there could be a way to fold prime limit in too.
I wondered about that too. But I've since convinced myself it's not possible, although I can't easily explain it.
By doing an actual sort (as a manually issued command) I have confirmed your sum of squares figure of 0.005710402 for (k=0.66, a=0.62, y=0.858, s=0.192), although I haven't confirmed that it's the minimum, or even a
Some more good new is that I am now finding values of those coefficients close to yours, by using the Excel Solver. This is the result of two major innovations.
First, I gave another degree of freedom to the monotonic function I'm using in lieu of sorting, to obtain the rank. Instead of simply est_rank = bmetric
, I'm now using est_rank = m×bmetric
. Typical values are m ≈ 0.5, b ≈ 1.46.
Second, I'm omitting the error for the ratio 1/1 from the sum of squared errors that the Solver is told to minimise.
When I do both of those things, Excel finds a minimum at (k=0.644, a=0.671, y=0.866, s=0.201). When I do an actual sort with those values, I get sum of squares = 0.006612407, showing that your values are better (k=0.66, a=0.62, y=0.858, s=0.192).
I now trust your optimisations.
So how much worse is the minimum sum-of-squares if we force s=0, or y=1, or both? And what are the optimum values of the other coefficients in those 3 cases?
You earlier claimed:
cmloegcmluin wrote: ↑Mon Jun 29, 2020 6:16 am
Without using sopf at all, the best fit I can find is 0.006905116251199162 (k=0.397, a=0.872, s=0, u=0.168, z=-1, cut-off-point=80).
But this was with usages of "s" and "u" swapped, so this should be what we would now call (k=0.397, a=0.872, y=1, s=0.168). But when I put in those values, and do the actual sort, I get sum-of-squares 0.02022529. Nearly three times what you claimed.
When I use the Solver (with my rankifying function instead of a sort), with y fixed at 1, I get a minimum at a very different place, namely (k=0.711, a=0.686, y=1, s=0.324). And when I apply a sort to this, I get sum-of-squares 0.007801175.
I similarly get (k=0.641, a=0.878, y=0.825, s=0) with sum-of-squares 0.009200898,
and (k=0.744, a=1.041, y=1, s=0) with sum-of-squares 0.008898435.
I note that good-old-fashioned sopfr(n*d) alone (i.e. k=1, a=1, y=1, s=0) has sum-of-squares 0.011375524. That's almost double what we get for (k=0.66, a=0.62, y=0.858, s=0.192), sum-of-squares 0.005710402.
The case of maximally-negative correlation (ranks exactly reversed) gives sum-of-squares 3.019814879. Randomly shuffled ranks (near zero correlation) seem to give sum-of-squares around 2.5.