developing a notational comma popularity metric

[Dave Keenan]

Lauren and I do, actually, share a strange fascination with Wat Tambor, the lead engineer for the Techno Union (which manufactures the NR-N99), so I'm amused to share this odd affiliation with him now.

Cool.

Thanks for crunching the numbers and preparing that table of min and max prime exponents for me. That's exactly everything I need to get the answer to your question without really having to think about anything... just hammer out a little code.

Please note my recent correction to that table. I realised the denominator limit was 23, not 17, due to 25/19 and 25/23 having N2D3P9 ≤ 136.

Out of curiosity I asked what N2D3P9 had to say about the 121k vs 1225k issue. Interestingly, while 243/242 has lower SoPF>3 than 19683/19600 (22 vs. 24), it has higher N2D3P9 (≈16 vs. ≈12). We'll wait on the comma-no-pop-rank metric though to make a call on that front.

I get N2D3P9(121) ≈ 37, N2D3P9(1225) ≈ 60. I can't figure out what you're doing wrong.

So as you know I'm currently reading my way back over this topic from page 1 and taking notes about the different threads we pursued (including the blue threads of death... ). So far we've each only almost given up once apiece (I seem to recall almost giving up three times myself, you maybe only twice). It's occasionally agonizing how close you or I come to wbl₁ before my code gave us some confidence that we couldn't do better.

Interesting. But I can't see that sort of history being of much interest to others. I'm thinking of the writeup as a Xen Wiki entry for N2D3P9. I'm imagining a reader who has seen the term somewhere but knows nothing about it. They want to know: What is it good for. Why should I believe it is fit for those purposes? How do I calculate it? And maybe: Who came up with it and when? How did they come up with it.?

I still have no idea how we're going to weight abs3exp and apotome slope. What objective data do we have to model that after?

Bugger all. Although we did throw away some data on 3-exponents from the Scala archive data, because initially it was the full pitch ratios with their individual counts, before I crunched them down to 5-rough. I think the full data is in a spreadsheet somewhere in this forum. That's data on what 3-exponents people use in pitch ratios. But I assume the average (signed) 3 exponent will be close to zero.

We're more interested in: When choosing what comma to give to some symbol, how much are we willing to allow the 5-rough-no-pop-rank to increase in order to have a comma that doesn't require so many sharps or flats to go along with it.

Besides of course Sagittal commas, which if — and this is a big "if" — we no longer cared about our reputations, we could just fit it to those and call it day.

I must admit to similar wicked thoughts myself.

I could imagine it might be something to the effect of: for ratios of a given N2D3P9 rank, what does the distribution of apotome slopes look like? Does this comma under consideration fall on the better or worse side of this distribution?

I see no harm in looking at that.

The other thing we should do is look again at George's comma complexity metric.

I'm thinking we'll have to multiply N2D3P9 by some number raised to the power of the apotome-slope, rather than add something.

I'm biased towards using apotome-slope alone, where George took a kind of mean between that and abs3exp. Of course the two are indistinguishable for commas as small as tinas.

I'll soon be off hiking for a few hours.

[Dave Keenan]

Awesome work! Thanks! And you independently found and fixed my errors in the exponent ranges. Well done. But I'm concerned we might still have missed some. 128 doesn't seem close enough to 136.

Maybe increase the exponent range of every prime by one up and one down and rerun it, to see if we missed any.

That 14641 is bizarre. We have to get rid of that, ASAP. I presume it's only there because it was in the Scala list (it had a count of 2 and so a rank of 329). What is the N2D3P9 that appears (maybe around 140 on your graph) just before that little step up. Maybe that should be the limit of our list. And I suggest using slashes instead of colons, and rounding the N2D3P9's to 2 decimal places.

[Dave Keenan]

Found another error in my exponent ranges. We can have 49 as the denominator. N2D3P9(125/49) = 66.17.

[cmloegcmluin]

Please note my recent correction to that table. I realised the denominator limit was 23, not 17, due to 25/19 and 25/23 having N2D3P9 ≤ 136.

Awesome work! Thanks! And you independently found and fixed my errors in the exponent ranges. Well done.

You're welcome, though you did the real thinking on it.

You must have snuck your changes in before I used your table, because I didn't find or fix anything myself

Found another error in my exponent ranges. We can have 49 as the denominator. N2D3P9(125/49) = 66.17.

Okay, well that one doesn't seem to be reflected in the table yet. That alone might account for the missing 8 or so. But I'm running the thing now with an extra +1 in each direction for each prime just in case, as you suggested.

Out of curiosity I asked what N2D3P9 had to say about the 121k vs 1225k issue. Interestingly, while 243/242 has lower SoPF>3 than 19683/19600 (22 vs. 24), it has higher N2D3P9 (≈16 vs. ≈12). We'll wait on the comma-no-pop-rank metric though to make a call on that front.

I get N2D3P9(121) ≈ 37, N2D3P9(1225) ≈ 60. I can't figure out what you're doing wrong.

I figured it out: I did not include the step to flip ratios so that n > d after 5-roughening.

I'm thinking of the writeup as a Xen Wiki entry for N2D3P9. I'm imagining a reader who has seen the term somewhere but knows nothing about it. They want to know: What is it good for. Why should I believe it is fit for those purposes? How do I calculate it? And maybe: Who came up with it and when? How did they come up with it.?

I'll divert my efforts to first preparing something that would suit a Xen Wiki entry like this, keeping it terse. I think your talking points here are good ones.

I still have no idea how we're going to weight abs3exp and apotome slope. What objective data do we have to model that after?

Bugger all. Although we did throw away some data on 3-exponents from the Scala archive data, because initially it was the full pitch ratios with their individual counts, before I crunched them down to 5-rough. I think the full data is in a spreadsheet somewhere in this forum. That's data on what 3-exponents people use in pitch ratios. But I assume the average (signed) 3 exponent will be close to zero.

I know you shared it because I have a copy of it in my Google Sheets.

Yes, I've found it here: viewtopic.php?p=1331#p1331

We're more interested in: When choosing what comma to give to some symbol, how much are we willing to allow the 5-rough-no-pop-rank to increase in order to have a comma that doesn't require so many sharps or flats to go along with it.

Or also perhaps how much are we willing to allow the 5-rough-no-pop-rank to increase in order to have a comma which allows notation of ratios which are not yet exactly notatable?

I could imagine it might be something to the effect of: for ratios of a given N2D3P9 rank, what does the distribution of apotome slopes look like? Does this comma under consideration fall on the better or worse side of this distribution?

I see no harm in looking at that.

The other thing we should do is look again at George's comma complexity metric.

Ah yes, good call.

At that time perhaps we should ping some of the other folks who were hacking on this with us over the Magrathean thread.

I'm thinking we'll have to multiply N2D3P9 by some number raised to the power of the apotome-slope, rather than add something.

I haven't the foggiest myself. Sounds good though

I'm biased towards using apotome-slope alone, where George took a kind of mean between that and abs3exp. Of course the two are indistinguishable for commas as small as tinas.

I'm biased toward simplicity. Fewer trunks/chunks the better.

But I'd like to understand George's reasons for abs3exp before dismissing it.

But I'm concerned we might still have missed some. 128 doesn't seem close enough to 136.

Maybe increase the exponent range of every prime by one up and one down and rerun it, to see if we missed any.

Oof, that put my code over the edge and I had to refactor it a bit to get any results.

Well, it got the number up to 131. The three ratios it had missed before were indeed all 49-denominated ones: 125/49, 55/49, and 65/49.

I've just gone ahead and added them into the previous table.

Good instinct that we still hadn't found them all. What do you think... is 131 close enough now?

That 14641 is bizarre. We have to get rid of that, ASAP. I presume it's only there because it was in the Scala list (it had a count of 2 and so a rank of 329).

That'd do it.

Turns up in young-g.scl (not La Monte; "Gayle Young's Harmonium, see PNM 26(2): 204-212 (1988)" and ligon7.scl "Jacky Ligon, superparticular 7 tone 11-limit MOS, 27/22=generator, MMM 22-01-2002" in the form of a 29282. Both legit looking scales, i.e. not lists, and they're not dupes of each other. Yet perhaps 14641 turning up twice is just a natural anomaly and not predictive of future popularity.

What is the N2D3P9 that appears (maybe around 140 on your graph) just before that little step up. Maybe that should be the limit of our list.

Just so we have it, here's the full list of the eleven N2D3P9 values which were over 136, and the symbols and primary commas which bore them:

[Update: this table is wrong! See later post: viewtopic.php?p=2252#p2252]
140.49
147.18
195.76
200.82
202.78
215.16
306.13
361.08
1118.41

And I suggest using slashes instead of colons, and rounding the N2D3P9's to 2 decimal places.

Also updated in the previous post.

[Dave Keenan]

Or also perhaps how much are we willing to allow the 5-rough-no-pop-rank to increase in order to have a comma which allows notation of ratios which are not yet exactly notatable?

Hmm. I don't think George and I ever did that. But it doesn't seem unreasonable.

Maybe increase the exponent range of every prime by one up and one down and rerun it, to see if we missed any.

Oof, that put my code over the edge and I had to refactor it a bit to get any results.

Well, it got the number up to 131. The three ratios it had missed before were indeed all 49-denominated ones: 125/49, 55/49, and 65/49.

I've just gone ahead and added them into the previous table.

Thanks for that. But I don't see them. I'm guessing you were short on time again.

Good instinct that we still hadn't found them all. What do you think... is 131 close enough now?

Well I think the "guard band" of ±1 in all exponents guarantees we haven't missed any. Of course we didn't have a guard on the negative powers of 7, but there's no way we could have 7^-3, under 136.

What is the N2D3P9 that appears (maybe around 140 on your graph) just before that little step up. Maybe that should be the limit of our list.

Just so we have it, here's the full list of the eleven N2D3P9 values which were over 136, and the symbols and primary commas which bore them:

Thanks. But I saw no commas, or 5-rough ratios with these.

I see the value before the step was 147.18. So maybe push the list to 148. And later we can try to reassign the 8 symbols above 195.

[cmloegcmluin]

Thanks for that. But I don't see them.

Sunnuva... I guess I must have only pushed preview one last time instead of actually pushing submit, and then was doing three other (related) things at the same time per usual (e.g. test results or script results came back) and by the time I came back to the tab it looked posted because it was unusually long so the chrome of the POST A REPLY screen wasn't visible so I just closed the tab.

Guhhhh that makes me angry! Well, at least you received the critical bit which was that the only new entries were those three with 49's in their denominator, before I had to turn in.

Sorry about that! It should be updated now.

Might be interesting to see the introducing levels of each of symbols too, no, both here and in the list < 136, or whatever cut-off we choose? Let me know if you're interested in that and I can get them in there.

Why did you want slashes instead of colons, by the way? For me colons seemed appropriate because we're dealing with undirected ratios, e.g. I had to make sure my code grabbed for r = 5/1. Just trying to improve my intuition for this. Perhaps I'm a lost cause with respect to the : vs / issue. I always seem to get it wrong.

I'm guessing you were short on time again.

I was short on time, but only because it was almost bedtime. This type of short on time won't be solved by not having a full-time job. I chalk this one up to the time difference and my rush to get things over to you before I'm gone for 8 hours. Sometimes, as I've said before, the time difference is helpful. Other times it's a bummer.

Or also perhaps how much are we willing to allow the 5-rough-no-pop-rank to increase in order to have a comma which allows notation of ratios which are not yet exactly notatable?

Hmm. I don't thing George and I ever did that. But it doesn't seem unreasonable.

If one goal is to enable as many ratios as possible to be exactly notatable, it seems reasonable. But I'm thinking this could turn into quite the rabbit hole of a thing to optimize for.

Thanks. But I saw no commas, or 5-rough ratios with these.

Sorry! At some point I planned to supplement the symbols with their commas in that little chunk of info, but then I didn't follow through. I've finished the job now.

Or... I would have updated the previous table... if it were indeed nothing other than a fleshing out of what was already there. It looks like actually that table was just straight up wrong!

The problem was that it used data from before I updated N2D3P9 not to assume you were giving it ratios whose 5-roughened versions had n ≥ d.

So the bad news is: there are actually more Sagittal symbols with primary commas in the JI notation which have big N2D3P9 (by our current threshold of big being > 136). Seventeen of them, actually, not just eleven. And the worst offender is actually much bigger: 1827.98, not 1118.41.

ratio	N2D3P9	symbol
1121931/1120000	1827.98
131769/131072	1118.41
375/368	306.13
252/247	304.18
18711/18200	215.16
2925/2816	215.16
2097152/2083725	208.42
2080/2079	200.82
22599/22528	195.76
8019/7936	195.76
1024/1001	180.74
256/253	161.64
2720/2673	147.18
88/85	147.18
185895/180224	147.18
595/576	140.49
19683/19040	140.49

To illustrate the problem, consider the Sagittal comma which we now know to have the worst N2D3P9:  , 1121931/1120000, or the 19/4375s. It has monzo [ -8, 10, -4, -1, 0, 0, 0, 1 ⟩. So after 5-roughening it you get 19/4375. So that's a huge difference when you flip it!

(19/2) * (5/3) * (5/3) * (5/3) * (5/3) * (7/3) * 19 * (1/9) = 361.08
(5/2) * (5/2) * (5/2) * (5/2) * (7/2) * (19/3) * 19 * (1/9) = 1827.98

I just checked and there's no problem with the main table in the earlier post where I list ratios with the lowest N2D3P9. That's because even though my N2D3P9 function was implemented wrong at the time I prepared that table, I never called in the way that illuminated its wrongness, because outside of it I was filtering ratios with n < d (because those get generated by the hardcoded ranges of prime exponents you prepared, which is fine, but there was no need to waste time redundantly processing them).

I see the value before the step was 147.18. So maybe push the list to 148. And later we can try to reassign the 8 symbols above 195.

I assume you mean get all ratios with N2D3P9 < 148, yeah?

You may also want to change the target now that I have accurate results. Though I actually think 148 is still the appropriate cutoff. So that would be 12 of the above 17 symbols would need different commas.

Here's an updated chart:



I didn't happen to implement my code to generalize to any max N2D3P9... I just hardcoded your ranges. I followed your explanations to some extent and it looks like I have a formula for the numerators but the denominator I understand is more complicated because it can't exist without a greater value already in the numerator. Anyway if I get time today I'll figure it out but I may need to focus on other things I can do more efficiently.

I know you're interested in fixing these outliers with huge N2D3P9's. How interested are you in tweaking things on the other end? Consider the ratio with the dubious distinction of being the ratio out of those that Sagittal doesn't notate exactly which has the lowest N2D3P9: 25/11. If there was a symbol in the Sagittal JI notation with a primary comma that exactly notated this ratio, that comma would be 100/99 which is about 17.4 cents. So it'd be competing with 99/98, the 11/49C, which has both higher SoPF>3 (25 vs 22) and higher N2D3P9 (54.90 vs 28.01). No other symbol covers 49/11 though, so we'd be sacrificing coverage of it for 25/11. But theoretically it'd still be a trade we'd want to make (well, except the 11/49C has apotome slope of 0.91, while the 25/11C has -3.07).

After we finish the write-up for N2D3P9, I'll need to take a few days to focus on another project. I told my friend I was on the "home stretch" for a sub-project of the microtonal notation system project. To honor that, I think I should dedicate some time to his project before digging into the comma-no-pop-rank metric. I recognize that the better "stopping point" in terms of this forum topic would be to finish the comma-no-pop-rank metric – to fulfill the name of the topic – but when I told him that, I was thinking only of the 5-rough-no-pop-rank metric part of the work.

[cmloegcmluin]

Can't believe it's only occurring to me to ask this now, but what did y'all do with the pitches in the Scala archive given in cents? I find a couple 386.3137s in there, but I assume those didn't get counted as votes for 5/1. Because what would you have done with 386.31499, 386.31055, 386.31312, 386.31742, etc... those are probably votes for 5/1, but where do you draw the line? Partial votes, according to some harmonic entropy like mapping? Or was the idea that since the project was assigning primary commas to symbols for a JI notation that it was just fine if not the right thing to do to count only votes given by people explicitly working in JI?

[cmloegcmluin]

For purposes of both my writeup and my code, do you consider N2D3P9 to be:

1. a function which takes any ratio as an argument,
   then 5-roughens it, then takes its reciprocal if necessary so that n ≥ d, then applies the formula
2. a function which expects a 5-rough ratio with n ≥ d,
   and if given something other than that, either errors or gives bogus results
3. something else

My feeling is that the answer is probably (a).

[Dave Keenan]

Might be interesting to see the introducing levels of each of symbols too, no, both here and in the list < 136, or whatever cut-off we choose? Let me know if you're interested in that and I can get them in there.

That would be good. Either a number for the level (0 = Spartan), or "low", "medium", "high", "ultra", ...

Why did you want slashes instead of colons, by the way? For me colons seemed appropriate because we're dealing with undirected ratios, e.g. I had to make sure my code grabbed for r = 5/1.

I'm not sure what you're saying here. Surely 5/1 is

Just trying to improve my intuition for this. Perhaps I'm a lost cause with respect to the : vs / issue. I always seem to get it wrong.

Maybe reread the directed comma names thread starting here: viewtopic.php?p=1839#p1839
Although I suppose that would require you to show the downward version of some symbols, which isn't important in this context. So I guess undirected is OK.

But if you're going to use undirected, I thought we agreed the canonical form for those has the small number first, to be consistent with extended ratios for chords, e.g. 4:5:6.

If one goal is to enable as many ratios as possible to be exactly notatable, it seems reasonable. But I'm thinking this could turn into quite the rabbit hole of a thing to optimize for.

Yes. Let's set the whole revisit of the Extreme JI notation to one side until we've assigned the tina ratios and updated SMuFL and Bravura. I'm pretty sure there are no unaccented symbols or unsymbolled accents in danger of being redefined, and hence no effect on the definitions already in SMuFL (except the errors you found some months back).

So the bad news is: there are actually more Sagittal symbols with primary commas in the JI notation which have big N2D3P9 (by our current threshold of big being > 136). Seventeen of them, actually, not just eleven. And the worst offender is actually much bigger: 1827.98, not 1118.41.

OK. Well caught. I'd be happy if we just found alternatives with N2D3P9 ≤ 216 for the two worst offenders. Of course it would be good to do the same for all those with N2D3P9 > 216, but that may not be possible. There are some minas that are comma deserts. You're lucky to find anything growing there at all. You really have to get out your magnifying glass.

And yes, we need to wait until we have a comma-no-pop-rank that incorporates the N2D3P9 5-rough-no-pop-rank.

I assume you mean get all ratios with N2D3P9 < 148, yeah?

You may also want to change the target now that I have accurate results. Though I actually think 148 is still the appropriate cutoff. So that would be 12 of the above 17 symbols would need different commas.

I did mean "get all ratios with N2D3P9 < 148", but now I'm thinking 216. We don't want to make life too hard for ourselves. But we can leave that until after the SMuFL/Bravura update. It's good to know we have an algorithm for producing a complete list to any max N2D3P9.

I didn't happen to implement my code to generalize to any max N2D3P9... I just hardcoded your ranges. I followed your explanations to some extent and it looks like I have a formula for the numerators but the denominator I understand is more complicated because it can't exist without a greater value already in the numerator. Anyway if I get time today I'll figure it out but I may need to focus on other things I can do more efficiently.

I have some ideas on how to reliably get the max prime exponents for the denominators, for a given max N2D3P9. I'll write it up when I get the chance. But the important thing now is the writeup of N2D3P9. Let's consider the above list-generating symbol-matching exercise as having been a shakedown-cruise to make sure N2D3P9 was fit for purpose. IMHO it passed with flying colours.

I know you're interested in fixing these outliers with huge N2D3P9's. How interested are you in tweaking things on the other end? Consider the ratio with the dubious distinction of being the ratio out of those that Sagittal doesn't notate exactly which has the lowest N2D3P9: 25/11. If there was a symbol in the Sagittal JI notation with a primary comma that exactly notated this ratio, that comma would be 100/99 which is about 17.4 cents. So it'd be competing with 99/98, the 11/49C, which has both higher SoPF>3 (25 vs 22) and higher N2D3P9 (54.90 vs 28.01). No other symbol covers 49/11 though, so we'd be sacrificing coverage of it for 25/11. But theoretically it'd still be a trade we'd want to make (well, except the 11/49C has apotome slope of 0.91, while the 25/11C has -3.07).

Yes, we should eventually investigate these. Including exploring email between George and I, to see why 11:49 was chosen over 11:25 for  . But, as you say, SOPF>3 tells the same 5-rough story as N2D3P9, as do the Scala archive stats, so it's not like we didn't know what we were doing. It may well have been due to the slope.

After we finish the write-up for N2D3P9, I'll need to take a few days to focus on another project. I told my friend I was on the "home stretch" for a sub-project of the microtonal notation system project. To honor that, I think I should dedicate some time to his project before digging into the comma-no-pop-rank metric. I recognize that the better "stopping point" in terms of this forum topic would be to finish the comma-no-pop-rank metric – to fulfill the name of the topic – but when I told him that, I was thinking only of the 5-rough-no-pop-rank metric part of the work.

The write-up for N2D3P9 is a fine stopping point. Let's not sagittate on anything else, until that's done.

sagittate (verb): To cogitate on anything related to Sagittal notation.

[Dave Keenan]

Can't believe it's only occurring to me to ask this now, but what did y'all do with the pitches in the Scala archive given in cents? I find a couple 386.3137s in there, but I assume those didn't get counted as votes for 5/1. Because what would you have done with 386.31499, 386.31055, 386.31312, 386.31742, etc... those are probably votes for 5/1, but where do you draw the line? Partial votes, according to some harmonic entropy like mapping? Or was the idea that since the project was assigning primary commas to symbols for a JI notation that it was just fine if not the right thing to do to count only votes given by people explicitly working in JI?

Good thought. It hadn't occurred to me either. We ignored them. But now that you've raised the idea, I say "too hard" (for exactly the reasons you point out) and unlikely to shed any new light on anything.