## developing a notational comma popularity metric

cmloegcmluin
Posts: 721
Joined: Tue Feb 11, 2020 3:10 pm
Location: San Francisco, California, USA
Real Name: Douglas Blumeyer
Contact:

### Re: developing a notational comma popularity metric

Sadly the overnight run was not a success. It got only a fraction of a percent of the way through before it snagged on some sector where somehow every sample point got identified as a local minimum and then it just infinitely recursed on omniminima until it crashed. I'll have to iron out that bug today when I get time.
Dave Keenan wrote:
Mon Jul 06, 2020 4:21 pm
...I remind you that, with this scenario, the log base is not constrained by the data. You can lock it to whatever you want and you will find the same minimum. You only have to scale w and c by lognew_base(old_base). ...

So this is a 4-parameter metric.
Thank you for reminding me. I get the basic idea, but perhaps not fully through and through yet. It might help if I had three formulas, one defining a in terms of c and w, one defining w in terms of a and c, and one defining c in terms of a and w. I should probably do it myself as an exercise.

Would it not have to be a condition that all of the parameters to the coapfar for which c is a coefficient be the same as the parameters to the soapfar with the related a and w? Well, but a and w are among those parameters, so should those be excluded? Yeah, I am finding ways to get confused about this relationship. Nothing I shouldn't be able to figure out through controlled experimentation with the dials and knobs I've built though.

I think this means that I waste resources searching for metrics which use all three. I should always leave one out and then pick reasonable ranges for the remaining two. Leaving w out means setting w = 0. Leaving c out means c = 0. Leaving a out means... wait, this one's tougher, because there is no identity base (as there is an identity exponent, which is 1). So if we're pretty well set that a logarithmic a is good, perhaps I should default to leaving out c, since that eliminates two "chunks" of complexity: the c, and the coapfar it coefficiates.

I believe the notion that this is a 4-parameter metric is what you were getting at earlier, and I just didn't grasp it yet. I took the conversation of defining a parameter in terms of complexity when we share this metric out. But this is a more urgent definition of parameter: in terms of finding the metric in the first place.
Dave Keenan wrote:
Mon Jul 06, 2020 6:39 pm
That suggests another question. What scale factor do we have to apply to your winning metric above (in its log2 version), to make it give almost the same values as sopfr, for the first few ratios? (Looks like we have to multiply it by about 5.7) And can we then describe it as a correction-function applied to sopfr?
I agree that finding the correct scale factor on our metric to make it as similar as possible to SoPF>3 where it counts most is a good idea. I'm not positive that I'll want to include it in the metric itself; I may prefer to have a SoPF>3-scaled version of it for when it is needed.

I don't understand your last sentence here, though, at all. Sorry. Please say more.

cmloegcmluin
Posts: 721
Joined: Tue Feb 11, 2020 3:10 pm
Location: San Francisco, California, USA
Real Name: Douglas Blumeyer
Contact:

### Re: developing a notational comma popularity metric

Golly, spent about 8 hours today diagnosing and resolving the issues preventing the recursive run from completing successfully. I'll try running it overnight now and see what I get.

Dave Keenan
Posts: 1024
Joined: Tue Sep 01, 2015 2:59 pm
Location: Brisbane, Queensland, Australia
Contact:

### Re: developing a notational comma popularity metric

cmloegcmluin wrote:
Tue Jul 07, 2020 1:11 am
Sadly the overnight run was not a success. It got only a fraction of a percent of the way through before it snagged on some sector where somehow every sample point got identified as a local minimum and then it just infinitely recursed on omniminima until it crashed. I'll have to iron out that bug today when I get time.
I really appreciate the work you're putting into this.
Dave Keenan wrote:
Mon Jul 06, 2020 4:21 pm
...I remind you that, with this scenario, the log base is not constrained by the data. You can lock it to whatever you want and you will find the same minimum. You only have to scale w and c by lognew_base(old_base). ...

So this is a 4-parameter metric.
Thank you for reminding me. I get the basic idea, but perhaps not fully through and through yet. It might help if I had three formulas, one defining a in terms of c and w, one defining w in terms of a and c, and one defining c in terms of a and w. I should probably do it myself as an exercise.
This is missing the point. You can't calculate a from c and w. The point is:

If you multiply your entire metric by a positive number (non-zero and finite), you don't change the ranking and therefore you don't change the SoS. This is just a special case of the more general fact that applying any monotonically increasing function to your entire metric does not change the ranking.

With the particular metric under discussion, if you multiply each of c, w and loga(p) by the same positive number (call it "g" for gain), that is equivalent to multiplying the whole metric by g. This is merely an application of the distributive law of multiplication over addition. gx + gy + gz = g(x+y+z). Then we have the change-of-base law, that tells us (after a bit of algebra) that a change of scale is equivalent to a change of log base, as follows:
g × loga(p) = logb(p) where b = a1/g
Would it not have to be a condition that all of the parameters to the coapfar for which c is a coefficient be the same as the parameters to the soapfar with the related a and w?
No. You only need to be able to scale the soapfar and the coapfar by the same factor, so that you are scaling the whole metric without changing the proportions of each.
I think this means that I waste resources searching for metrics which use all three.
Yes. And it means that the minimum, instead of being a bowl, is a canyon of infinite length, the same depth all the way along. Your search could waste a lot of time zig-zagging along the bottom of that canyon, in the hope of finding a drop-off, or a rising end, that will never come. All that would happen is that the metric would grow and grow, or shrink and shrink, in overall scale, without changing the ranking or the SoS, until some number overflows or underflows.
I should always leave one out and then pick reasonable ranges for the remaining two. Leaving w out means setting w = 0. Leaving c out means c = 0. Leaving a out means... wait, this one's tougher, because there is no identity base (as there is an identity exponent, which is 1). So if we're pretty well set that a logarithmic a is good, perhaps I should default to leaving out c, since that eliminates two "chunks" of complexity: the c, and the coapfar it coefficiates.
No. Setting w or c to zero doesn't work because that's not consistent with applying a monotonic function to the metric. But you could fix c at any positive number (e.g. 1) and allow a and w to vary. Or you could fix w at any negative number and allow a and c to vary. But it seems more natural to me, to fix a at 2 (because this is about music, where octaves reign), and allow c and w to vary. Don't even call it log2(). Call it lb() (for log-binary, by analogy with ln() for log-natural). This is ISO standard notation.

You might think you will sometimes want to set the log base to something other than 2, e.g. to have different logs applied to the numerator and denominator. But you can always just multiply lb(p) by a new parameter, g.

g × lb(p) = loga(p) where a = 21/g and g = 1/lb(a)

So there's never any need to change log bases at all. Just pick one and stick to it.

And having a multiplying factor on the log makes it obvious that you can set that multiplying factor to anything you like, including 1, and thereby get rid of it, by dividing the whole metric by that factor. In this case, it is sufficient to divide w and c to achieve this.
I believe the notion that this is a 4-parameter metric is what you were getting at earlier, and I just didn't grasp it yet. I took the conversation of defining a parameter in terms of complexity when we share this metric out. But this is a more urgent definition of parameter: in terms of finding the metric in the first place.
Right. Although I'd describe it slightly differently. I'd say you correctly pointed out that metric complexity includes more than just the parameter count. But there remains a definite answer to the question "How many parameters does it have?". And this is relevant to finding the metric in the first place.
Dave Keenan wrote:
Mon Jul 06, 2020 6:39 pm
That suggests another question. What scale factor do we have to apply to your winning metric above (in its log2 version), to make it give almost the same values as sopfr, for the first few ratios? (Looks like we have to multiply it by about 5.7) And can we then describe it as a correction-function applied to sopfr?
I agree that finding the correct scale factor on our metric to make it as similar as possible to SoPF>3 where it counts most is a good idea. I'm not positive that I'll want to include it in the metric itself; I may prefer to have a SoPF>3-scaled version of it for when it is needed.

I don't understand your last sentence here, though, at all. Sorry. Please say more.
I've already decided that this is a dead-end, not worthy of further consideration, but what I imagined was, solving for the function f() in the equation scaled_metric(ratio) = sopfr(ratio) + f(ratio) or alternatively scaled_metric(ratio) = sopfr(ratio) × f(ratio). Either way, f could be called a correction function. This would only be worthwhile if such an f() was simpler to write than the metric itself.

Dave Keenan
Posts: 1024
Joined: Tue Sep 01, 2015 2:59 pm
Location: Brisbane, Queensland, Australia
Contact:

### Re: developing a notational comma popularity metric

So our best 4-parameter metric so far is of the form:

Σp(
1  × (lb(p)+w) × npy + c × np +
k × (lb(p)+w) × dpy + c × dp
)
where
k ≈ 0.180
y ≈ 0.493
w ≈ -2.02
c ≈ 0.571

Is that correct?

I think the next question is, what's the best one we can find that is less complex than that?

How well can we do with:

Σp(
(     lb(p)+w) × npy +
(g×lb(p)+ z  ) × dpy
)

Still 4 parameters, but arguably simpler. It could be tried with a simple n>d numinosity as well as with sopfr(n)>sofr(d).

cmloegcmluin
Posts: 721
Joined: Tue Feb 11, 2020 3:10 pm
Location: San Francisco, California, USA
Real Name: Douglas Blumeyer
Contact:

### Re: developing a notational comma popularity metric

Dave Keenan wrote:
Tue Jul 07, 2020 3:23 pm
This is missing the point. You can't calculate a from c and w. The point is:
I'm pretty confused by this section. I've gone over it several times (enough to notice your edits to it). I understand all the points you make about not monotonic functions not changing the ranking, and those arithmetic laws, but I don't understand how these add up or function in relation to the introduction of this section. In other words, I don't understand how these two things explain that one can't calculate a from c and w, how I've missed the point, how these two things are the point, or how the difference in what is the point is related to not being able to calculate a from c and w.
Would it not have to be a condition that all of the parameters to the coapfar for which c is a coefficient be the same as the parameters to the soapfar with the related a and w?
No. You only need to be able to scale the soapfar and the coapfar by the same factor, so that you are scaling the whole metric without changing the proportions of each.
Well... but I think we slice & dice this stuff differently. See below.
I think this means that I waste resources searching for metrics which use all three.
Yes. And it means that the minimum, instead of being a bowl, is a canyon of infinite length, the same depth all the way along. Your search could waste a lot of time zig-zagging along the bottom of that canyon, in the hope of finding a drop-off, or a rising end, that will never come. All that would happen is that the metric would grow and grow, or shrink and shrink, in overall scale, without changing the ranking or the SoS, until some number overflows or underflows.
I can't imagine this with the same clarity that you can, but it might explain why my script ran overnight and when I woke up it was still running but hadn't found anything better than it had found in the first fraction of a second.
Setting w or c to zero doesn't work because that's not consistent with applying a monotonic function to the metric.
I thought that we had shown that c can be redundant with w because c is a coefficient on the count and w is a constant added to each prime and those are equivalent. So assuming you get the other parameters (k and j in particular, I think) in the right place there shouldn't be anything you couldn't express with w that you couldn't express with c, and vice versa, and that's why I thought you could set one or the other to 0.
But it seems more natural to me, to fix a at 2 (because this is about music, where octaves reign), and allow c and w to vary. Don't even call it log2(). Call it lb() (for log-binary, by analogy with ln() for log-natural). This is ISO standard notation.
Agreed. Will do.

I still don't really get most of this. Sorry. It's probably best for me not to try to wrangle with natural language to increase my understanding. I do understand my code and so I'll write some tests to prove facts about how the parameters relate to each other.
Dave Keenan wrote:
Wed Jul 08, 2020 12:13 am
So our best 4-parameter metric so far is of the form:

Σp(
1  × (lb(p)+w) × npy + c × np +
k × (lb(p)+w) × dpy + c × dp
)
where
k ≈ 0.180
y ≈ 0.493
w ≈ -2.02
c ≈ 0.571

Is that correct?
I believe your articulation of it is equivalent, where you slice it by prime first, submetric second, I slice it by submetric first, prime second. I could have designed the code the other way. I don't see any particular reason why it would work better or worse that way. But in any case, what I've got works more like this:

Σp(
1  × (lb(p)+w) × npy +
k × (lb(p)+w) × dpy
) +
c × Σp(
np +
dp
)
I think the next question is, what's the best one we can find that is less complex than that?

How well can we do with:

Σp(
(     lb(p)+w) × npy +
(g×lb(p)+ z  ) × dpy
)

Still 4 parameters, but arguably simpler. It could be tried with a simple n>d numinosity as well as with sopfr(n)>sofr(d).
Why is it called g now instead of k?

We found something as low as 0.00622 with an equivalent way of writing that. Back when you asked me to a c=0 variant on the lowest one I'd found yet at that time.

By the way, I would call that a 6-parameter one myself, in terms of chunks of complexity presented to the user, as you've got
• the lb
• y
• w
• g
• z (because it's different than w)
• a point for the numinator/diminuator concept which people have to understand, unless this is truly n and d, but I assume they are numinator and diminuator because we've found better results that way and I'm pretty sure we've been assuming this for a while

Dave Keenan
Posts: 1024
Joined: Tue Sep 01, 2015 2:59 pm
Location: Brisbane, Queensland, Australia
Contact:

### Re: developing a notational comma popularity metric

cmloegcmluin wrote:
Wed Jul 08, 2020 1:37 am
Dave Keenan wrote:
Tue Jul 07, 2020 3:23 pm
This is missing the point. You can't calculate a from c and w. The point is:
I'm pretty confused by this section. I've gone over it several times (enough to notice your edits to it). I understand all the points you make about not monotonic functions not changing the ranking, and those arithmetic laws, but I don't understand how these add up or function in relation to the introduction of this section. In other words, I don't understand how these two things explain that one can't calculate a from c and w, how I've missed the point, how these two things are the point, or how the difference in what is the point is related to not being able to calculate a from c and w.
Perhaps I overstated the case. There is a sense in which you can calculate a from c and w, in fact you can calculate the other two from any one of them. But that's only after you have found one set of them, call them a1, c1 and w1. Then, given a different c, you can calculate the corresponding w and a as:
w = w1 × c/c1
a = a1c1/c

But in regard to finding the first set, you must fix a, not calculate it anew every time you try a new c or w during the search. Or you could choose to fix c, and vary a and w. Or fix w and vary a and c. But I find that fixing a (as 2) and using the lb(p) notation*, makes things easier to think about. * And when necessary g × lb(p). e.g. when two submetrics would otherwise have needed different log bases.
Would it not have to be a condition that all of the parameters to the coapfar for which c is a coefficient be the same as the parameters to the soapfar with the related a and w?
No. You only need to be able to scale the soapfar and the coapfar by the same factor, so that you are scaling the whole metric without changing the proportions of each.
Well... but I think we slice & dice this stuff differently. See below.
But you agree the two ways of "slicing and dicing" are mathematically equivalent.
I think this means that I waste resources searching for metrics which use all three.
Yes. And it means that the minimum, instead of being a bowl, is a canyon of infinite length, the same depth all the way along. Your search could waste a lot of time zig-zagging along the bottom of that canyon, in the hope of finding a drop-off, or a rising end, that will never come. All that would happen is that the metric would grow and grow, or shrink and shrink, in overall scale, without changing the ranking or the SoS, until some number overflows or underflows.
I can't imagine this with the same clarity that you can, but it might explain why my script ran overnight and when I woke up it was still running but hadn't found anything better than it had found in the first fraction of a second.
I'm guilty of mixing metaphors. Sorry. I earlier talked more realistically about hyperpolyhedral regions where SoS didn't change. In that metaphor, you had to imagine the SoS as a scalar field (physics sense, not mathematical sense) over the multidimensional parameter space. In that case you could at least visualise a 3D space (for 3 parameters) with the SoS being the colour (e.g. like a 3D heat map) at every point of that space. And we are trying to find the coldest/bluest point.

But in the metaphor used above, I am taking the SoS to be a spatial dimension (the vertical one), and so you can only visualise 2 parameters. The SoS forms a surface, a landscape, with hills and valleys, and we are trying to find the lowest point.

If I describe the above problem using the heatmap metaphor, it is that when you don't fix a, there is no single point of minimum temperature. Instead there is an infinite blue thread.
Setting w or c to zero doesn't work because that's not consistent with applying a monotonic function to the metric.
I thought that we had shown that c can be redundant with w because c is a coefficient on the count and w is a constant added to each prime and those are equivalent. So assuming you get the other parameters (k and j in particular, I think) in the right place there shouldn't be anything you couldn't express with w that you couldn't express with c, and vice versa, and that's why I thought you could set one or the other to 0.
I think I said "almost" equivalent, and I think that was when the y values under discussion were closer to 1 than they are now. The only time c and w are equivalent is when y = 1 [or when the same y is applied to both soapfar and copfar]. That's one reason I wrote it with the Σp outside of everything, so you could see more clearly, the relationship between w and c.

(lb(p)+w) × npy + c × np

Unless y=1, it's polynomial in np, so you can't combine the coefficients (lb(p)+w) and c.
I still don't really get most of this. Sorry. It's probably best for me not to try to wrangle with natural language to increase my understanding. I do understand my code and so I'll write some tests to prove facts about how the parameters relate to each other.
Good idea. You should be able to easily confirm that you get the same SoS with the different (a, w, c) sets (really tuples) I have given above (with your original a, a = 2 and a = e). And confirm that you will never get the same SoS with c = 0, although you may still get a usefully low minimum.
Dave Keenan wrote:
Wed Jul 08, 2020 12:13 am
So our best 4-parameter metric so far is of the form:

Σp(
1  × (lb(p)+w) × npy + c × np +
k × (lb(p)+w) × dpy + c × dp
)
where
k ≈ 0.180
y ≈ 0.493
w ≈ -2.02
c ≈ 0.571

Is that correct?
I believe your articulation of it is equivalent, where you slice it by prime first, submetric second, I slice it by submetric first, prime second. I could have designed the code the other way. I don't see any particular reason why it would work better or worse that way. But in any case, what I've got works more like this:

Σp(
1  × (lb(p)+w) × npy +
k × (lb(p)+w) × dpy
) +
c × Σp(
np +
dp
)
My spreadsheet actually does it in exactly the same way as your code. Of course it makes no difference how the code or spreadsheet does it. My rewriting was purely intended to aid human comprehension, in figuring out how parameters can or can't be combined or eliminated by rescaling the whole metric.
I think the next question is, what's the best one we can find that is less complex than that?

How well can we do with:

Σp(
(     lb(p)+w) × npy +
(g×lb(p)+ z  ) × dpy
)

Still 4 parameters, but arguably simpler. It could be tried with a simple n>d numinosity as well as with sopfr(n)>sofr(d).
Why is it called g now instead of k?
It's not very important. I thought it would be confusing if I called it k because k is traditionally outside the brackets of (lb(p)+w). But yes, I could have written say k × (lb(p) + b). But I wanted a multiplier directly on the lb(p), so that it corresponded directly to a change of log base. And yes, I could have written (k×lb(p) + k×b), but it seemed strange to have k×b when it could just be a single parameter z. But if I write (k×lb(p) + z) then the k can no longer be taken outside the brackets and so it is no longer the same kind of k that we have been using, and so I thought it best to use g instead, as (g×lb(p) + z), because I had already been using g in my attempted explanation of change-of-base being equivalent to change-of-scale.
We found something as low as 0.00622 with an equivalent way of writing that. Back when you asked me to a c=0 variant on the lowest one I'd found yet at that time.
OK. But I thought you might set your wonderful new machine churning away to try to find a lower SoS for that form of metric.
By the way, I would call that a 6-parameter one myself, in terms of chunks of complexity presented to the user, as you've got
• the lb
• y
• w
• g
• z (because it's different than w)
• a point for the numinator/diminuator concept which people have to understand, unless this is truly n and d, but I assume they are numinator and diminuator because we've found better results that way and I'm pretty sure we've been assuming this for a while
Why don't we call it a 6-chunk metric, and keep the term "parameter" for the numbers you change during a search for a SoS minimum? You may know that "chunk" is actually a respectable term in cognitive psychology.
https://en.wikipedia.org/wiki/Chunking_(psychology)

Dave Keenan
Posts: 1024
Joined: Tue Sep 01, 2015 2:59 pm
Location: Brisbane, Queensland, Australia
Contact:

### Re: developing a notational comma popularity metric

cmloegcmluin wrote:
Wed Jul 08, 2020 1:37 am
Dave Keenan wrote:
Wed Jul 08, 2020 12:13 am
How well can we do with:

Σp(
(     lb(p)+w) × npy +
(g×lb(p)+ z  ) × dpy
)

Still 4 parameters, but arguably simpler. It could be tried with a simple n>d numinosity as well as with sopfr(n)>sofr(d).

We found something as low as 0.00622 with an equivalent way of writing that. Back when you asked me to a c=0 variant on the lowest one I'd found yet at that time.
The only place I can find any number beginning with 0.0062 in this thread is here:
cmloegcmluin wrote:
Tue Jun 30, 2020 3:44 pm
with c = 0, I can only get 0.006251296.
that's with k = 0.635, a = 1.430, y = 0.850, w = -2.770
When I scale that to use lb(p) instead of loga(p), then w → w/lb(a) and we get:
k = 0.635, y = 0.850, w = -1.4294

I see that those values do give SoS = 0.00622, so the earlier 0.00625 was only due to not yet having implemented fractional ranks for ties.

However, this is not of the form I request above, since it uses w for both the numerator and denominator. The form above could also be written as:

Σp(
1  × (lb(p)+w) × npy +
k × (lb(p)+b ) × dpy
)

where z has been replaced by k×b (and g replaced by k).

Regardless of how it's written, this form was proposed, based on the insight gained from the following graph (which you can click to go to the original post), but with the simplification of forcing v = y.

I note that this graph was based on n being simply the largest side of the ratio. If we do the same for the proposed metric, that would eliminate a chunk — getting us down to 5 chunks (4 parameters and the lb()).

However, I did write "It could be tried with a simple n>d numinosity as well as with sopfr(n)>sofr(d). ", by which I meant that we could see whether we got a lower SoS for this form when n was chosen to be the greatest side of the ratio, or when n was chosen as the side with the greatest sopfr.

cmloegcmluin
Posts: 721
Joined: Tue Feb 11, 2020 3:10 pm
Location: San Francisco, California, USA
Real Name: Douglas Blumeyer
Contact:

### Re: developing a notational comma popularity metric

Dave Keenan wrote:
Wed Jul 08, 2020 10:05 am
There is a sense in which you can calculate a from c and w, in fact you can calculate the other two from any one of them. But that's only after you have found one set of them, call them a1, c1 and w1. Then, given a different c, you can calculate the corresponding w and a as:
w = w1 × c/c1
a = a1c1/c

But in regard to finding the first set, you must fix a, not calculate it anew every time you try a new c or w during the search. Or you could choose to fix c, and vary a and w. Or fix w and vary a and c. But I find that fixing a (as 2) and using the lb(p) notation*, makes things easier to think about. * And when necessary g × lb(p). e.g. when two submetrics would otherwise have needed different log bases.
This is very helpful to me. Thank you for the explanation.
Well... but I think we slice & dice this stuff differently. See below.
But you agree the two ways of "slicing and dicing" are mathematically equivalent.
Yes, I agree. And I appreciate now how the way you wrote it is good for humans and clarifies some things.
I'm guilty of mixing metaphors.
No worries. I confused my partner in a similar fashion when I was asking her some sanity check questions about local minima just the other day; I tried to describe a minimal "2D" example, but I failed to recognize that what I was describing was actually 3D, with the third dimension either a color or a z/elevation.
If I describe the above problem using the heatmap metaphor, it is that when you don't fix a, there is no single point of minimum temperature. Instead there is an infinite blue thread.
I can see it now. Well, maybe I see something in between that and a blue river running through the bottom of a red canyon. Thank you.

I'm going to put off responding to the a, c, w stuff in detail for now, since I think it will take me some time to have something meaningful to say further. But I'd like to respond to the other stuff sooner than that.
We found something as low as 0.00622 with an equivalent way of writing that. Back when you asked me to a c=0 variant on the lowest one I'd found yet at that time.
OK. But I thought you might set your wonderful new machine churning away to try to find a lower SoS for that form of metric.
Dave Keenan wrote:
Wed Jul 08, 2020 1:35 pm
The only place I can find any number beginning with 0.0062 in this thread is here:
cmloegcmluin wrote:
Tue Jun 30, 2020 3:44 pm
with c = 0, I can only get 0.006251296.
that's with k = 0.635, a = 1.430, y = 0.850, w = -2.770
However, this is not of the form I request above, since it uses w for both the numerator and denominator.
Yes, that was the bit I was referring to. But again, unfortunately, I'm finding that I'm not quite ready to respond in any helpful or intelligent way on this front. I'll need to really sit down with it to get the difference between what we've tried already and what you asked for. And how it relates to insights from your chart. Or maybe we should video call for this one. It's clearly not rocket science; I'm confident I can get it, but I'm struggling a bit. I think my head is already overflowing this week with all the surprisingly complex gymnastics I have to do to get this code to do the things I want.
You may know that "chunk" is actually a respectable term in cognitive psychology.
https://en.wikipedia.org/wiki/Chunking_(psychology)
I did not know that! Nice!
Why don't we call it a 6-chunk metric, and keep the term "parameter" for the numbers you change during a search for a SoS minimum?
Yes, I agree that differentiating these two concepts is a good idea and important to do, and I also think parameter and chunk are good names for them.

cmloegcmluin
Posts: 721
Joined: Tue Feb 11, 2020 3:10 pm
Location: San Francisco, California, USA
Real Name: Douglas Blumeyer
Contact:

### Re: developing a notational comma popularity metric

I have added another layer on top of the code. My previous update was that it could now take an initial configuration (a set of submetrics, parameters, and ranges of settings for the parameters) and recursively bisect along every dimension of the search space and follow all local minima branches until it found a global minimum. My latest update is the ability for it to start from zero chunks and work its way up, for each chunk count calculating every possible such initial configuration (for the parameter ranges, it starts with the widest reasonable range for each), performing the recursive search for each one, and returning the best possible metric for that chunk count.

I'm not ready to share out any grand conclusions, though:
• It can't get past 3 chunks yet, because it keeps drowning in infinite blue threads. This was happening before but I wasn't able to figure it out because I was straight away attacking 8- and 9- chunk metrics which would run so slowly with so much noise I couldn't understand where things were going wrong. It has become clear now that I will eventually need to get my head around every parameter relationship in order to proceed. But that's fine. I've already identified a few more combinations that don't play nice.*
• I am butting up against the limits of the JavaScript engine in various ways that are going to force some performance-related changes.
• I am butting up against the limits of my own faculties to reason about this system in ways that are going to force me to statically type the code now, which will take a bit of time to retroactively apply.
That said, I do have at least a couple fun tidbits that I'm confident enough about to share.

Among the 1-chunk metrics, my code found that the best choice was...
...
\text{sopfr}
aka SoPF>3. Right, no surprises there. It gives the SoS as 0.014206087, which is a bit higher than the 0.011375524 value found back here, but I attribute that to it occurring B.F.R. (Before Fractional Ranks). In this case though I'm happy to get the SoS nicked, as it just means whatever better SoS's we find with our new metrics will be relatively better than that which we're improving upon!

And among the 2-chunk metrics, my code found that the best choice was...
...
\text{sopfr}(n) + k × \text{sopfr}(d), k = 0.7901234567901236
(where n is the numerator and d is the denominator, not the numinator and diminuator)
that decimal for k has a suspicious pattern to its digits that a continued fraction calculator assisted me in determining is almost exactly \frac{64}{81}. Fascinating! That metric gets you an SoS of 0.009491243, about \frac{2}{3} that of SoPF>3.

As for 3-chunk metrics, the result was pretty amusing. It spit out
\text{sopfr} + \text{copfr}(n) + k × \text{copfr}(d), k = -1.\overline{1}
which gives an SoS of 0.008291286. So it would seem that adding the counts of the primes in the numerator while subtracting the counts of the primes in the denominator gets you a pretty good fit. Of course this doesn't feel psychologically motivated, so I think we should pass over this one.

Speaking of \text{copfr} (and \text{copf}), I discovered yesterday while chatting about this project with a friend that there's an established name for this function: the prime omega function. This might explain why Dave was using that fancy lookin' w earlier, because that's actually what a lowercase Greek letter omega looks like; little omega \omega(p) is our \text{copf} and big omega \Omega(p) is our \text{copfr}. I don't suggest we stop using \text{copf} and \text{copfr} (or \text{coapf} and \text{coapfar}, as it were) since those make the relationship with \text{soapfar} and \text{soapf} clear. But just thought I'd put it out here.

*An interesting example: t and w get stuck in an infinite blue thread, though of a different nature I believe. When t is very tiny, you can nudge it by a tiny amount, and then nudge w by just the right corresponding tiny amount, even though this won't result in the exact same antivotes for each individual ratio, it will result in the aggregate across all 80 ratios an SoS which is very close. So even though w affects each prime and t affects each repetition, and those two counts don't match on each ratio, it can still continuously work out into a subtly different and thus "new" local minimum according to my code even as the space it is searching contracts into an ever tinier and tinier space. None of the obvious fixes I've thought of yet have solved the problem (max recurse depth? lose out on valuable results elsewhere. only pursue local minima which are lower than the one you're coming from? also lose out on results you actually do care about, i.e. sometimes as you stab around you do have to step up and out a little bit to find the correct pit that goes the deepest). My current solution: scratch t from the picture altogether, since we had determined it was a bit problematic anyway.

Dave Keenan