Dumbing of Age characters

I read Dumbing of Age, but I don't really pay super close attention to things.  There are 1773 comics as of today, and it's hard to keep track of stories that long over a long period of time.  There are also a lot of characters that I have trouble keeping track of.  "Wait, how do you know how many comics it has?"

Today's story is all "Amber pushes Danny away because she's angry and making bad decisions."  My thought was, "Who else is Amber's friend?  Joyce, right?"

amber danny 91 0.408072 0.325
amazi-girl danny 40 0.3125 0.142857
amber ethan 60 0.269058 0.285714
amber dina 53 0.237668 0.24424
amazi-girl dorothy 24 0.1875 0.0526316
amazi-girl joyce 19 0.148438 0.0255376
amazi-girl walky 18 0.140625 0.0392157
amber joyce 30 0.134529 0.0403226
amazi-girl sal 16 0.125 0.0695652
amazi-girl amber 15 0.117188 0.0672646

Ethan, Dina, and then Dorothy maybe.  Yeah, I was wondering this enough that I wrote a bot to scrape all the comics to pull out the tags that are applied to each comic, since conveniently list all the characters appearing in that comic.  Then, I looked at the pairwise matches in that set and dumped them out in "character A", "character B", "number of appearances together", and number of appearances for each of A and B, converted here into fractions of all appearances that are together.

So that was waste of time.  I also have dates, and you can extract chapters from the urls (which I saved), so more analysis could be done (my thought was to try to do some sort of connection map), but I still haven't eaten dinner.  Also, I discovered that this comic is the only one to have no characters appearing, so had to handle that case (SOLO_APPEAR is in that with SOLO_APPEAR).

Why is that so noisy?

Julie sent this video to me, and I was confused, because I didn't think it should saturate into noise on the third iteration.  Human voices are in the 1000 Hz range, so if the Carl doubles the frequency, three iterations only gets it to 8000 Hz, which is still well sampled by a 44 kHz sound file (the standard).  So, I did the sane thing when I got home, which is to download the video and do spectral analysis of the audio.

The human (s00), Carl A (s01), and Carl B (s02).

The human speech is mostly that tiny red peak on the left side, at about 1-2 kHz.  It's confused beyond that, but I think that second red peak (2k ish) can be plausibly shifted in the others.

Plotting everything.

The interesting thing in this one is that you can see that there are two patterns.  The dips around 11k and 13k are probably the easiest way to see that.  They're caused by the response function of the devices:

Carl A has the benefit on the first iteration to have the true voice.

Carl B.

 So I don't think it's really related to the speech frequency vs sampling rate.  I think it's just the addition of the noise in the microphone/speaker feedback.

This is kind of like reruns. Maybe it's a "remastered" post?

First up, killing the Supreme Court.  Again.  But still with numbers and statistics, because that's the best way to do things.  Assume the Senate decides to stop being dumb.  Then, Merrick Garland gets a hearing and since he's basically fine, he gets a seat on the Supreme Court.  Since my least favorite justice is dead.

So here's the cumulative "how many justices are alive" plot.  Honestly, according to this, if the Senate doesn't stop acting like children, Obama might have two more people to appoint before the end of his term.  It's good that the Republicans aren't running serious options this year, since that sets up a good shift when they don't become president.

And the by name individual plot.  I've seen a lot of stuff talking about how Garland is "already old" so it "doesn't matter" if he gets confirmed or not.  This is stupid.  The cumulative plot clearly shows that the next president is very important for determining the Supreme Court's future.

In any case, he's younger than the median justice, and is likely to be on the court for another ~15 years.  Or, you know, the next four presidential terms.  Also, it's interesting to note the benefit of appointing women to the court.  Roberts was born in 1955, and Sotomayor in 1954.  That's the unit the script uses for sorting the key.  But, looking at the graph, Sotomayor is likely to be on the court ~3 years longer.  I should also enable the grid display next time I do this.

Final final four

Today's also the last day I update the sports stuff for this year.  Here's the table for the rest of the tournament:

#Bracket	N_R1	PP_R1	Nwrong_R1	P_R1	S_R1	N_R2	PP_R2	Nwrong_R1	P_R2	S_R2
Mine	32	1	6	26	.995	16	2	4	50	.998
Heart-of-the-cards	32	1	10	22	.656	16	2	8	38	.320
Julie	32	1	10	22	.656	16	2	6	42	.738
BHO	32	1	9	23	.823	16	2	6	43	.820
538	32	1	8	24	.928	16	2	7	42	.738
Rank	32	1	13	19	.129	16	2	6	39	.424

#Bracket	N_R3	PP_R3	Nwrong_R3	P_R3	S_R3	N_R4	PP_R4	Nwrong_R4	P_R4
Mine	8	4	3	70	.998903	4	8	3	78
Heart-of-the-cards	8	4	6	46	.044	4	8	4	46
Julie	8	4	4	58	.610	4	8	4	58
BHO	8	4	4	59	.674	4	8	3	67
538	8	4	2	66	.955	4	8	2	82
Rank	8	4	2	63	.875	4	8	3	71

#Bracket	N_R3	PP_R3	Nwrong_R3	P_R3	N_R4	PP_R4	Nwrong_R4	P_R4
Mine	2	16	2	78	1	32	1	78
Heart-of-the-cards	2	16	2	46	1	32	1	46
Julie	2	16	2	58	1	32	1	58
BHO	2	16	1+	67+	1	32	1	67+
538	2	16	2	82	1	32	1	82
Rank	2	16	2	71	1	32	1	71

If the President gets his pick correct in the next round, then he'll win with an 83.  Otherwise, 538 wins based on only getting two wrong in round 4.  Everything else is locked in now, so there's nothing really to update anymore.

Round 3

Since it's the weekend, it's sports time.  First up, my picks for this round of things:

One that I was doomed to get wrong.

And the other doomed one.  But a new mistake!

Texas A&M:

29.687500       14.062500       3.125000                3       6       3       Texas A&M

28.125000       18.750000       21.875000               3       8       2       Oklahoma

First up, I think my analysis notes have been wrong on the previous posts.  The file I'm pulling these numbers from is in 2016/2015/2014/group/game/rank/name format, not 2014/2015/2016 format.  This changes the analysis for some of my previous mistakes, but I'm too lazy to go correct those.  In any case, using this new, correct information, it looks like I thought (from the 2016 ratings) that Texas A&M should be slightly better than Oklahoma.  Folding in previous years could have potentially altered that choice.

I was thinking a bit about adding some score-based information in as well.  The idea being that each team scores a given median number of points across all their games, and have a given median number of points scored against them.  By comparing how well a given score ranks in all their games, and against their opponent's, it should be possible to construct offense and defense ratings.  This might be useful to say, "Team X is generally better, but they only are a +1 in offense, and they're playing a +4 defense, so they might not win."  The other benefit would be to add two new metrics, which could then be used across the full multi-year dual-gender score set to determine which relative weights each should be assigned to a more complete prediction model.

I think the first step that I should do, though, is to dump all of that data into a database, instead of using horrible fixed-width formatted files to manage things.  That's largely a consequence of not really caring a lot about the project.

In any case, here's the comparison table for round three:

#Bracket	N_R1	PP_R1	Nwrong_R1	P_R1	S_R1	N_R2	PP_R2	Nwrong_R1	P_R2	S_R2
Mine	32	1	6	26	.995	16	2	4	50	.998
Heart-of-the-cards	32	1	10	22	.656	16	2	8	38	.320
Julie	32	1	10	22	.656	16	2	6	42	.738
BHO	32	1	9	23	.823	16	2	6	43	.820
538	32	1	8	24	.928	16	2	7	42	.738
Rank	32	1	13	19	.129	16	2	6	39	.424
#Bracket	N_R3	PP_R3	Nwrong_R3	P_R3	S_R3	N_R4	PP_R4	Nwrong_R4	P_R4	S_R4
Mine	8	4	3	70	.998903	4	8
Heart-of-the-cards	8	4	6	46	.044	4	8
Julie	8	4	4	58	.610	4	8
BHO	8	4	4	59	.674	4	8
538	8	4	2	66	.955	4	8
Rank	8	4	2	63	.875	4	8

This now has the added columns of S_RX.   These are my simulated CDF values based on the Yahoo selection pick fractions given for each team.  This is another piece of kind-of garbage code that I threw together earlier in the week.  I think it's doing everything correctly, but I don't see any simulated results that get a total score above 83, and yahoo does list some in their leader list.  Maybe 1e6 simulations isn't sufficient to fully probe things?  Maybe I'm truncating or rounding something odd?  The main idea behind this calculation is to see how well a given set of picks should rank.

Plots for individual rounds and the total after three.  In general, the mean drops (because past mistakes have continuing consequences) and the variance increases (because there's the 2^N point scaling thing and because the number of individual games is falling as well).

Round 2

today was the end of round two of the sports thing.  I also need to go back and update posts with the new label I've decided is probably useful, "sports".  So I updated everything before the final game was over, and then had to double check nothing went wrong:

Copying from 538.  Two of those I didn't care about anymore due to prior choices, one of them I kind of knew was going to be the case, one of them I wasn't expecting to take two overtimes to come to my result, one apparently fell apart in the last two seconds, and the final one had me frowning at it until it decided not to make me re-edit all my stuff.

 Results for this time:

Again, three of my four mistakes this time around were caused by my winning choice being eliminated in the previous round.  For the last one:

Xavier:
12.500000       42.187500       29.687500               2       7       7       Wisconsin

34.375000       12.500000       14.062500               2       8       2       Xavier

Why did I choose Xavier?  Did I get confused and use the 2014 rankings instead of the 2016 ones?  This looks like me being dumb.  Maybe I took the #2 ranking too seriously?  I should probably write down logic notes next time, so I can point to the error directly.

What does the scoring comparison look like?

#Bracket	N_R1	PP_R1	Nwrong_R1	P_R1	N_R2	PP_R2	Nwrong_R1	P_R2
Mine	32	1	6	26	16	2	4	50
Heart-of-the-cards	32	1	10	22	16	2	8	38
Julie	32	1	10	22	16	2	6	42
BHO	32	1	9	23	16	2	6	43
538	32	1	8	24	16	2	7	42
Rank	32	1	13	19	16	2	6	39

Again the "rank" method is garbage, and shouldn't be used.  Nate Silver had a tweet earlier about how this is apparently because it's based on RPI too much.  Looking at wikipedia, it looks like RPI is an incomplete version of my LAM method.  ¯\_(ツ)_/¯  This also shows the point where HotC totally falls apart, becoming the worst method.  Everyone else is pretty well clumped together.  I'm a bit surprised that 538 isn't doing better, given the "we included scores, and at-home values, and distances to the games, and the number of cats each player owns, and the SAT scores of each player."

This also makes me think I should have actually entered my selections into some pool.  Maybe I should hone the method a bit more, and see how it works over a few more years.  Or, alternatively, I could do the reasonably easy thing and apply the method to the historical data, and see if this consistently matches reality.  Maybe next weekend, since I think it's a long one.  This will also make me fix my master Makefile to put things into logical directories, and not just dump the outputs into a common directory.

Round one

Statistics results.

Ok, that West Virginia loss is going to hit the later rounds.

As is Purdue.  Not as bad as Michigan State, obviously.

Let's look at the comparison table:

#Bracket	N_R1	PP_R1	Nwrong_R1	P_R1
Mine	32	1	6	26
Heart-of-the-cards	32	1	10	22
Julie	32	1	10	22
BHO	32	1	9	23
538	32	1	8	24
Rank	32	1	13	19

The columns are the bracket identifier, the number of games in the round, the points per correct selection in the round, the number wrong, and the total points.  The brackets are mine above, the "Heart of the Cards" bracket taken by simply selecting teams based on the 2016 ranking I calculated, Julie's bracket, President Obama's, the 538 bracket taken by assuming constant composite rankings from their pre-tournament predictions, and a dummy bracket constructed by selecting teams based solely on their "sport rank" thing.  That's actually working out a lot better than I expected.  I was correct in shaking up the straight HotC numbers with a bit of historical data.  Looking at the mistakes:

Arizona:

26.562500       43.750000       40.625000               1       5       6       Arizona

25.000000       37.500000       53.125000               5       1       11      Wichita St

I didn't believe the numbers, given the #11 ranking.  From above, I should ignore the ranking in the future, because it's pretty crappy.  The problem is that my numbers suggest that Wichita State is the best team in the entire thing, which doesn't seem like it's right.

West Virginia:

28.125000       21.875000       1.562500                2       6       3       West Virginia

34.375000       39.062500       45.312500               2       6       14      SF Austin

Ditto.  My numbers predict that SF Austin is the second best team.  I guess if either of them come out winning, I can say that I predicted it, and then tossed it in the trash.

Baylor:

17.187500       23.437500       20.312500               3       3       5       Baylor

25.000000       18.750000       7.812500                3       3       12      Yale

No clue, but it sounds like everyone was surprised by this one.

Purdue:

29.687500       14.062500       -3.125000               4       3       5       Purdue

37.500000       -7.812500       -3.125000               4       3       12      Ark Little Rock

My numbers say they both suck, so I went with last year's numbers to break the tie.  I could have added in the 2014 values, but this was a #12 ranking, and I didn't believe those.

Dayton:

28.125000       26.562500       20.312500               4       7       7       Dayton

9.375000        7.812500        34.375000               4       7       10      Syracuse

This one I should have gotten right.  I folded the two previous years in, and that said that I should trust consistency over a sudden jump.  Maybe Syracuse has some new great player.

Michigan State:

35.937500       18.750000       28.125000               4       8       2       Michigan St

23.437500       3.125000        23.437500               4       8       15      MTSU

Again, this one seemed like it was a surprise to everyone.  There are only three values of my ranking between these two values, so that kind of suggests they're within ~5% of each other in terms of skill.  Oh well.