How does bedtime change with age?

I saw something earlier today that made me wonder how people's bedtime changes as they get older.  A quick google search pointed me to this study, which is probably the best/easiest to find data set that I'm likely to find just sitting online waiting.  There are a few obvious flaws:

1. It relies on users of the Jawbone UP for the data sample.  I have never heard of this device, and I'm guessing a lot of people are in the same boat.
2. Of those that do know of the Jawbone UP, they're probably younger than the average person, just because young people tend to use new tech at a higher rate.
3. There's likely class/income/etc. biases, as not everyone has the money to spend on a $50 fitness doodle.
4. There isn't actually any age data in the data set.

That last one isn't really that big of a show stopper.  

First step, scan through the source code to find the file that actually contains the data being used for the interactive map.  It's called counties.prod_.js, and nicely lists the county and state, bedtime in 12-hour format (minus 7, likely to ensure that there isn't a problem at the midnight boundary), as well as some other data I parsed out and saved in case I want to revisit something later.

Second step, trawl through the Census data for a county-by-county population breakdown, with age information.  That's here (although the full country data is actually 112MB, not the 11MB the page claims).  Then it's just a matter of pulling out the population data for 2015 (the closest match to the sleep data), setting the age for entire age groups at the midpoint, and calculating the weighted average age for each county (using the population in the age group as the weight).

So what's the result?

Other than me not truncating the best fit line.

The answer seems to be "Yes.  Bedtime is slightly earlier for older counties."  There's a bit of a plateau in the 16-20 range, but there's a reasonable decrease, even with the scatter.

Of course, using the full population isn't really the best, since the users of the Jawbone UP are likely adults, and not kids.  Redoing this analysis with just people older than 20 (due to the way the census data is binned into 5-year groups):

Basically the same, just shifted to an older age.

Working to get a consistent set of census data.

It's annoying, because the census doesn't have a single format that they use for all historical data.

Particularly in the effectively arbitrary old age cuts.  Why were they fine during the 80s, then slightly worse in the 90s, and then really bad in the 2000s?  No clue, but that's the data I have from the census, so that's what I'm using.  

The gap is from the lack of data between 2010 when the previous decade estimates stop and 2014 when the future projections begin.

This is kind of interesting too.  I initially started just plotting the population with age=0, with the intent to visualize generations.  The baby boom is really obvious in the purple curve.  I then added samples at different ages, lagged to use a consistent time base.  This thought this would give a probe of immigration, but that doesn't seem to be the case, as there aren't any major gaps in the first three samples in the 1850-1900 range.  I think the sag in the age=60 is just life expectancy issues.  That's also clearly apparent in the beyond 2000 area as well.  The projection yielding the 2014-2060 isn't predicting that to improve too much it seems.

Dumbing of Age characters

I read Dumbing of Age, but I don't really pay super close attention to things.  There are 1773 comics as of today, and it's hard to keep track of stories that long over a long period of time.  There are also a lot of characters that I have trouble keeping track of.  "Wait, how do you know how many comics it has?"

Today's story is all "Amber pushes Danny away because she's angry and making bad decisions."  My thought was, "Who else is Amber's friend?  Joyce, right?"

amber danny 91 0.408072 0.325
amazi-girl danny 40 0.3125 0.142857
amber ethan 60 0.269058 0.285714
amber dina 53 0.237668 0.24424
amazi-girl dorothy 24 0.1875 0.0526316
amazi-girl joyce 19 0.148438 0.0255376
amazi-girl walky 18 0.140625 0.0392157
amber joyce 30 0.134529 0.0403226
amazi-girl sal 16 0.125 0.0695652
amazi-girl amber 15 0.117188 0.0672646

Ethan, Dina, and then Dorothy maybe.  Yeah, I was wondering this enough that I wrote a bot to scrape all the comics to pull out the tags that are applied to each comic, since conveniently list all the characters appearing in that comic.  Then, I looked at the pairwise matches in that set and dumped them out in "character A", "character B", "number of appearances together", and number of appearances for each of A and B, converted here into fractions of all appearances that are together.

So that was waste of time.  I also have dates, and you can extract chapters from the urls (which I saved), so more analysis could be done (my thought was to try to do some sort of connection map), but I still haven't eaten dinner.  Also, I discovered that this comic is the only one to have no characters appearing, so had to handle that case (SOLO_APPEAR is in that with SOLO_APPEAR).

Why is that so noisy?

Julie sent this video to me, and I was confused, because I didn't think it should saturate into noise on the third iteration.  Human voices are in the 1000 Hz range, so if the Carl doubles the frequency, three iterations only gets it to 8000 Hz, which is still well sampled by a 44 kHz sound file (the standard).  So, I did the sane thing when I got home, which is to download the video and do spectral analysis of the audio.

The human (s00), Carl A (s01), and Carl B (s02).

The human speech is mostly that tiny red peak on the left side, at about 1-2 kHz.  It's confused beyond that, but I think that second red peak (2k ish) can be plausibly shifted in the others.

Plotting everything.

The interesting thing in this one is that you can see that there are two patterns.  The dips around 11k and 13k are probably the easiest way to see that.  They're caused by the response function of the devices:

Carl A has the benefit on the first iteration to have the true voice.

Carl B.

 So I don't think it's really related to the speech frequency vs sampling rate.  I think it's just the addition of the noise in the microphone/speaker feedback.

This is kind of like reruns. Maybe it's a "remastered" post?

First up, killing the Supreme Court.  Again.  But still with numbers and statistics, because that's the best way to do things.  Assume the Senate decides to stop being dumb.  Then, Merrick Garland gets a hearing and since he's basically fine, he gets a seat on the Supreme Court.  Since my least favorite justice is dead.

So here's the cumulative "how many justices are alive" plot.  Honestly, according to this, if the Senate doesn't stop acting like children, Obama might have two more people to appoint before the end of his term.  It's good that the Republicans aren't running serious options this year, since that sets up a good shift when they don't become president.

And the by name individual plot.  I've seen a lot of stuff talking about how Garland is "already old" so it "doesn't matter" if he gets confirmed or not.  This is stupid.  The cumulative plot clearly shows that the next president is very important for determining the Supreme Court's future.

In any case, he's younger than the median justice, and is likely to be on the court for another ~15 years.  Or, you know, the next four presidential terms.  Also, it's interesting to note the benefit of appointing women to the court.  Roberts was born in 1955, and Sotomayor in 1954.  That's the unit the script uses for sorting the key.  But, looking at the graph, Sotomayor is likely to be on the court ~3 years longer.  I should also enable the grid display next time I do this.

Final final four

Today's also the last day I update the sports stuff for this year.  Here's the table for the rest of the tournament:

#Bracket	N_R1	PP_R1	Nwrong_R1	P_R1	S_R1	N_R2	PP_R2	Nwrong_R1	P_R2	S_R2
Mine	32	1	6	26	.995	16	2	4	50	.998
Heart-of-the-cards	32	1	10	22	.656	16	2	8	38	.320
Julie	32	1	10	22	.656	16	2	6	42	.738
BHO	32	1	9	23	.823	16	2	6	43	.820
538	32	1	8	24	.928	16	2	7	42	.738
Rank	32	1	13	19	.129	16	2	6	39	.424

#Bracket	N_R3	PP_R3	Nwrong_R3	P_R3	S_R3	N_R4	PP_R4	Nwrong_R4	P_R4
Mine	8	4	3	70	.998903	4	8	3	78
Heart-of-the-cards	8	4	6	46	.044	4	8	4	46
Julie	8	4	4	58	.610	4	8	4	58
BHO	8	4	4	59	.674	4	8	3	67
538	8	4	2	66	.955	4	8	2	82
Rank	8	4	2	63	.875	4	8	3	71

#Bracket	N_R3	PP_R3	Nwrong_R3	P_R3	N_R4	PP_R4	Nwrong_R4	P_R4
Mine	2	16	2	78	1	32	1	78
Heart-of-the-cards	2	16	2	46	1	32	1	46
Julie	2	16	2	58	1	32	1	58
BHO	2	16	1+	67+	1	32	1	67+
538	2	16	2	82	1	32	1	82
Rank	2	16	2	71	1	32	1	71

If the President gets his pick correct in the next round, then he'll win with an 83.  Otherwise, 538 wins based on only getting two wrong in round 4.  Everything else is locked in now, so there's nothing really to update anymore.

Round 3

Since it's the weekend, it's sports time.  First up, my picks for this round of things:

One that I was doomed to get wrong.

And the other doomed one.  But a new mistake!

Texas A&M:

29.687500       14.062500       3.125000                3       6       3       Texas A&M

28.125000       18.750000       21.875000               3       8       2       Oklahoma

First up, I think my analysis notes have been wrong on the previous posts.  The file I'm pulling these numbers from is in 2016/2015/2014/group/game/rank/name format, not 2014/2015/2016 format.  This changes the analysis for some of my previous mistakes, but I'm too lazy to go correct those.  In any case, using this new, correct information, it looks like I thought (from the 2016 ratings) that Texas A&M should be slightly better than Oklahoma.  Folding in previous years could have potentially altered that choice.

I was thinking a bit about adding some score-based information in as well.  The idea being that each team scores a given median number of points across all their games, and have a given median number of points scored against them.  By comparing how well a given score ranks in all their games, and against their opponent's, it should be possible to construct offense and defense ratings.  This might be useful to say, "Team X is generally better, but they only are a +1 in offense, and they're playing a +4 defense, so they might not win."  The other benefit would be to add two new metrics, which could then be used across the full multi-year dual-gender score set to determine which relative weights each should be assigned to a more complete prediction model.

I think the first step that I should do, though, is to dump all of that data into a database, instead of using horrible fixed-width formatted files to manage things.  That's largely a consequence of not really caring a lot about the project.

In any case, here's the comparison table for round three:

#Bracket	N_R1	PP_R1	Nwrong_R1	P_R1	S_R1	N_R2	PP_R2	Nwrong_R1	P_R2	S_R2
Mine	32	1	6	26	.995	16	2	4	50	.998
Heart-of-the-cards	32	1	10	22	.656	16	2	8	38	.320
Julie	32	1	10	22	.656	16	2	6	42	.738
BHO	32	1	9	23	.823	16	2	6	43	.820
538	32	1	8	24	.928	16	2	7	42	.738
Rank	32	1	13	19	.129	16	2	6	39	.424
#Bracket	N_R3	PP_R3	Nwrong_R3	P_R3	S_R3	N_R4	PP_R4	Nwrong_R4	P_R4	S_R4
Mine	8	4	3	70	.998903	4	8
Heart-of-the-cards	8	4	6	46	.044	4	8
Julie	8	4	4	58	.610	4	8
BHO	8	4	4	59	.674	4	8
538	8	4	2	66	.955	4	8
Rank	8	4	2	63	.875	4	8

This now has the added columns of S_RX.   These are my simulated CDF values based on the Yahoo selection pick fractions given for each team.  This is another piece of kind-of garbage code that I threw together earlier in the week.  I think it's doing everything correctly, but I don't see any simulated results that get a total score above 83, and yahoo does list some in their leader list.  Maybe 1e6 simulations isn't sufficient to fully probe things?  Maybe I'm truncating or rounding something odd?  The main idea behind this calculation is to see how well a given set of picks should rank.

Plots for individual rounds and the total after three.  In general, the mean drops (because past mistakes have continuing consequences) and the variance increases (because there's the 2^N point scaling thing and because the number of individual games is falling as well).