welcome back everyone uh contrary to appearances we do have two commentators they're sitting right here no they're actually sitting in the audience so they can see the slides the slides from yesterday having garnered a very good reputation i actually dreamt about them four o'clock in the morning all i could see was synapses anyway welcome back everyone and thank you very much for attending today's lecture whether in person or in zoom on zoom i'm rebecca mclennan i'm on the faculty of the history department and i'm also serving on the tanner lectures organising committee this year today professor caroline hoxby will be presenting the second lecture in her tanner lecture series the fork in the road the imperative of investing in adolescent education now yesterday professor hocksby was very persuasive and arguing for the existence of a fork in the road of children's cognitive development one that accelerates not so much in the first three or four years of a child's life though it does a little as most laity assume but around the age of 10 and a half now professor hoxby i wish i had heard your wonderful lecture two years ago before i started writing preposterously large texts for my toddler's san francisco preschool uh but building on this the theme today's lecture is entitled smart money educational investments in adolescence earn higher returns we'll take a quick break after professor hoxby has further enlightened us and then i'll introduce today's distinguished commentators and after that we'll move to our discussion thank you please welcome professor hoxby [Applause] thank you our attendance on um the zoom i'm going to see whether i can do better with the clicker today than i managed to do yesterday so again i'm just showing you some i'm hoping nope it's not on thank you jane um synapses to get started on the neuroscience but mostly today we're going to be talking about tests of uh whether adolescents are able to learn more than students at other ages and so that's kind of the that's the reason for the title of the talk today sorry this one goes a little bit slowly that's image heavy okay so smart money uh educational investments in adolescents earn higher returns so i want to start with a very brief recap i'm going to try to keep it brief but just so people who weren't here at the lecture yesterday know where we are so the key points were first that increasing the share of americans with advanced cognitive skills is crucial in my opinion for social cohesion cohesion for advancing economic opportunity for reducing inequality and for decreasing geographic and political polarization and possibly also for other things like reducing crime or improving improving health now a recent neuroscience and neuropsychology suggests that adolescence is a key period for frontal cortex development since it is this frontal part of the brain that is thought to be responsible for advanced cognitive skills at least disproportionately we should expect some of these skills to really take off during adolescence and in fact i showed you this chart yesterday at measures of children's cognition and you can see that in early adolescence this period right here from starting at about 10 and a half and going to about 15 and a half but generally speaking that period you can really see um children who start gaining advanced cognitive skills their skills take off on a sort of high trajectory and other children they have their skills really stagnate during this period of time and then they tend to flatline thereafter and they gain very little if any skill in their late teen years so in other words early adolescence offers both great opportunities to learn but also risk of not learning after early adolescence i showed you some charts that looked like this that show that students skill trajectories start to harden and this is consistent with what we would expect from neuroscience specifically the pruning the pruning on myelination processes that occur in adolescents which sort of harden up these trajectories i also showed you a couple of maps of the united states this is one that i happen to like a lot i think it expresses political frustration and resentment of elites that may stem from the economic fatalism that some people feel when they realize that they have attained young adulthood without acquiring advanced cognitive skills and they may be shut out of remunerative and fulfilling careers so this one shows you what people what people think scientists think about climate change it's important to say it's not what they think about climate change and i concluded yesterday's lecture by saying that it's one thing to try to identify early adolescence as a key time when students should start developing advanced cognitive skills it's one thing to conclude that as a matter of logic or by looking at the neuroscience but it's another thing to demonstrate that we actually have realistic educational interventions that would be especially productive if they were implemented in early adolescence so today's lecture is all about all about those demonstrations i'm going to leave this up for right now and tell you a little story about how i became interested in adolescence now that we know where we are going forward so i teach an undergraduate class it's called the economics of education and in it we regularly examine recent research on early childhood education everything from the very famous abacidarian and perry preschool randomized control trials that are now almost 50 years old to recent research on the federal head start program and every year i teach my students the theory of endogenous skill growth which posits that children who acquire skills sufficiently early benefit more from each later educational experience creating trajectories that diverge further and further apart with age now this theory predicts that positive early childhood interventions such as abasidarian which believe it or not enrolled children at four months of age on average or perry preschool another very small but intensive program or head start that's the big federal program a very big program should launch children on a permanently steeper trajectory of skill growth in fact though the evidence looks quite different the effects of these programs are evident during the intervention period itself when you're enrolled in abasidarian or peri preschool or head start but then they tend to die out fairly quickly by second grade typically sometimes even first grade and the students and i would always find the theory of endogenous skill growth compelling and intuitive we would think to ourselves this makes sense this comports with our own personal experiences so we would finish up the lectures on early childhood education with a sense of class-wide frustration including myself and after class i would inevitably field many questions along the lines of if the theory is so compelling and intuitive and we all think it makes sense why isn't there stronger evidence for it the other experience that forced me to pay attention to early adolescence was my research on charter schools in new york city in this research some of which we will see later but very briefly i studied more than 100 charter schools using an experimental design that makes use of application lotteries essentially the experiment compared students who were and were not given seats in charter schools purely because of their randomly assigned lottery numbers i'll talk more about that a little later now some of the charter schools start with pre-kindergarten or kindergarten and then they end with an elementary grade let's say grade five or grade six and some uh charter schools start with grade five and end with grade eight so they're charter middle schools as it were and some charter schools are more like charter high schools they start typically with grade nine now the many economists who work on early childhood had led me to strongly expect that if a charter school was going to be a success it would be more successful if it started with the earliest possible grade thereby launching its pupils on a higher endogenous growth trajectory of skills and i was really baffled by the people who wanted to lead charter schools that started at grade five i thought you're just setting yourself up for failure you should have started your charter school with kindergarten or pre-kindergarten why are you starting with grade five and this was i was even more concerned because the data showed very clearly that the schools that started with fifth graders had applicants who were already struggling and already falling behind in the regular public schools at the time that they applied to the charter school i thought this is too late for these students they're never going to get the full benefit of a successful educational intervention or a successful charter school if the charter school is successful but when i computed the results in this charter school study i it was immediately obvious that these grade five to eight schools the middle schools were not only generating greater skill gains than other charter schools their gains were often much greater i reported the results but i really just didn't know what to make of them or how to explain the differences i also became aware that there was a reasonably large mathematica study that's a nonprofit research organization it was conducting a large randomized control trial on charter schools and it was coming up with very similar results about the middle charter schools doing an unusually good job so one day i was walking back to my office after having taught my early childhood classes and having heard all of my students frustration and i thought about the charter school evidence and i thought about the early childhood evidence and i realized that when i looked at education data which is of course something i do ordinarily in life that the fork in the road did not appear as early as i had expected it to occur something like kindergarten or pre-kindergarten but rather seem to events itself mainly in middle school and i also recall that a sizable share of students appear to get stuck at the middle school level of skills this is typified by the many students who are still struggling to learn algebra 1 in their freshman year of college even though they have been taking it in variously named math classes since the 7th grade indeed algebra 1 is the most commonly taken college course in the united states and freshmen who fail to master it often have to drop out of college not just because it's a prerequisite for majors like engineering or science or something along those lines but it's also a prerequisite for many remunerative community college majors just dental hygiene for instance or a lot of nursing majors so anyway this concatenation of thoughts about the charter schools the middle school the the charter middle schools the early childhood education and thinking about these people who get stalled in these middle school levels of skill made me rush off to the library which luckily is very close to my office and grab a whole stack of books and articles on neuroscience and adolescence and that was the genesis of these lectures so i just grabbed the books and i had a wonderful time uh taking a month or so off from economics and reading neuroscience and neuropsychology instead so if adolescence is indeed an age of opportunity this is a phrase coined by lawrence steinberg then it also has a big advantage vis-a-vis early childhood early childhood interventions are plagued by logistics many adults are needed to provide infants or toddlers with adequate care children of that age think of these four-month-olds need help with basic tasks such as eating toileting and dressing they usually have to be dropped off and picked up they cannot just ride on school bus with other children other older children and many parents are just uncomfortable with handling their toddler over to others for six or more hours a day unless the setting is intimate and familiar to them adolescents on the other hand are quite different in terms of logistics they're fairly independent people they're already in school for six hours a day so in other words if we can find educational interventions that improve their cognitive skills they are captive audience now the next thing that i want to talk about is the current neglect of adolescent education which might surprise you very much given the fact that this is this key opportunity time and a time that's a risky time it's unfortunate that adolescent grades are by far the most neglected in terms of teaching the neglect in my opinion is unintentional and it derives from some basic phenomenon class size is the largest in the middle school grades that's about grades six to eight so when i say middle school grades keep thinking grades six to eight usually this is partly because of the fascination with early childhood education combined with the fact that the logistics for smaller kids are just more difficult so schools tend to put them in smaller classes high school students are also in small classes but for a very different reason it's really because the curriculum starts to break up in high school so it's not just that there's science but there's chemistry and there's physics and there's earth science and biology so uh classes get split up into smaller and smaller units the same thing could be true of history there's u.s history european history world history and so forth so evidence on class size sorry this is one of my cognitive skill graphs showing you that early adolescence seems to be the time of a time of divergence this is average class size by grade for the state of north carolina which happens to have amazing administrative data so those are the data that i'm using for this and i'm going to keep using the north carolina data for a little while for the rest of the lecture so you'll you'll see more north carolina data but if we look at average class size by grade you can see that pre-kindergarten has about 14 children in each class very small the rest of the elementary grades from kindergarten to grade five it's usually 20 or fewer students in a class then we have the middle school grades six seven and eight and they are the middle school grades in the state of north carolina and you'll notice that all the class sizes are 27 uh or 30 even okay so much much bigger class sizes and then we have a decline in class size as we get higher and higher into the high school grades with the smallest class sizes being in 11th and 12th grade so i'm going to show you from now on i'm going to cluster some of those grades together both to keep the graphs a little bit neater and also because there are some data reasons for doing that into which i i will not go but so this is basically the same chart as my last chart and i'll keep using the same colors all the time elementary will always be green middle school will always be pink and high school will always be blue so it's just a way of collapsing the data to make it a little bit easier easier to read so not only is class size the largest for middle school teachers but their compensation is lower than that of elementary and high school teachers so you can see again green pink blue and the pink bar is the lowest and that's because middle school teachers are paid less than elementary and high school teachers this is not intentional but it appears to be due to the fact that teachers find it taxing and difficult to work with adolescents probably due to the brain development that makes them especially plastic cognitively at that age or it could just be puberty and the social transformation that they're experiencing in any case teachers who teach middle school grades are more likely to report problems such as student apathy student mental health issues or students being belligerent they're less likely to say that they are satisfied with their current job or satisfied with teaching and so it should not come as a surprise that teachers often move to an elementary or high school as soon as vacancy arises in one of them and as a result it's middle schools that are always seeing vacancies occur and having a tougher time filling them now most public school teachers as you may know are paid in the united states according to a scale that depends only on their years of seniority and their highest educational degree like whether you have a bachelor's or a master's degree this is called lock step pay thus middle schools have lower paid teachers because they constantly need to fill their vacancies with more first-year teachers or rookies or teachers who have little or no seniority in the district or teachers who have been taking years off so that's what's shown on this chart this is the percentage of teachers who are rookies in elementary middle and high schools and you can see that you have more rookies in the elementary school as are the middle schools and this is the percentage of teachers with no experience in this district so they're completely new to the curriculum they may not know things as well and you can see again that's higher in middle schools and it's because more vacancies arise horizon middle schools it's not just a problem for having very little experience with teaching it's also a problem for teacher quality although teacher value-added research which i am going to talk about in a moment finds little increase in teachers effectiveness after they have three or four years of experience rookies are consistently found to be less effective and so are teachers who have no experience in their district in other words middle school teachers not only encounter larger classes and less pay middle school students not only encounter larger classes and less well-paid teachers they also face teachers who are more likely to be struggling with instruction and who are dissatisfied with their current job middle school teachers are also less likely to have a graduate degree as shown here you can see that graduate degrees are quite disproportionately higher for people who are high school teachers and this is not a particular surprise remember that the the middle school teachers are more likely to be rookies and since so many teachers in the united states actually get their master's degree while they're already teaching and they do it part-time they're just less likely to have a graduate degree well that means they're going to be paid less because of the lock-step pay which gives you an automatic pay promotion if you get a a master's degree and all of these phenomena add up to this figure which i think is very dramatic so what this is showing you is the amount amount spent per pupil on teacher compensation and you can see that it is much much lower for middle school students than it is for elementary school students or for high school students in fact it's about it's less than half of what it is for high school students so why is this it's all of these forces combining it's the bigger classes it's the less experienced teachers who aren't getting the site the seniority pay it's teachers who don't have a graduate degree so they're not getting that bump in pay and that means that what we have is middle school students actually being unintentionally neglected in terms of the resources that they get so at this crucial point in time where their brains are plastic and they're trying to make transitions to more advanced cognitive skills they're also being comparatively neglected all right so regardless of the reasons why this happens it's clearly counterproductive in the sense that i tend to think that middle school teachers really may need combat pay of some type not the other way around because they're dealing with uh more more risky harder to teach harder to teach children okay so now that we have established that early adolescence are relatively neglected people i want to return to the central goal of this lecture that is conducting tests that can generate plausibly causal evidence of whether early adolescents are especially affected by successful educational interventions and i had four criteria for the natural or policy experiments i'm going to use those two words interchangeably that i employ i want to test interventions that one are specific to a grade or an age so that i can compare results across ages i can compare what happens to an adolescent versus someone who's in elementary or high school two my second criteria is the intervention should be something that you could apply at any grade again this is so that i can compare results across grades in other words if it's some kind of trigonometry intervention you obviously aren't going to apply that to third graders so it can't be an intervention of that type uh the um three my third criteria was that i wanted to test interventions that have typically been found in the past to have statistically significant effects that's because there's just no point in comparing results across grades if all of the results were null or extremely noisy and my fourth criterion was that i wanted to test interventions that allowed me to examine later effects such as a student's achievement at the end of high school or going to college after all given that age only moves in one direction we keep all getting older sad but it's true i can obviously not see the effects of an intervention that is applied to ninth graders and then see oh now that i've affected them as ninth graders what would have happened to their third grade test scores they're already past the third grade but i can test the same intervention if i apply it to third graders and ninth graders and then look at both of them at the very end of high school and say where did it make the most difference now it turns out that i had to rack my brain to come up with interventions that fulfilled all of these criteria and i actually only came up with three that i think are at all good and the first one is what i think is as the most important one it's about a teacher's value added the idea is quite simple it's having a teacher with high versus low value added in other words ability to teach students more materials ability to teach students more skills would i want to have a teacher when i was in middle school who has high value added versus elementary school versus high school i'm not going to get a high value-added teacher every year so where would i like to concentrate them if i had if i had the choice my second which i hope you will find relatively fun is being exposed to a curriculum that is more challenging cognitively for instance if a school district is going to introduce and introduce a new testing or curricular regime that has richer cognitive content and it's going to do that across all grades should students want to see that happen when they're still in middle school as opposed to say high school and then i'm going to talk about attending a successful charter school going back to the story with which i began so if students are only going to be able to secure a charter school seat in one lottery they can maybe win in elementary school maybe they could win in middle school maybe they could win in high school where would you want to win the most and i'll show you those results very briefly okay i'm going to spend most of my time on this value added because i think it's especially important partly because of who teachers are and partly because it's really the best policy experiment just to be clear so in my first natural experiment i'm going to test whether having a teacher with relatively high value added is especially important in the middle school grades so the way you want to think about this here is we could conjecture what if we could just concentrate all of the most effective teachers in the middle school grades perhaps we could pay them more to teach in those grades or maybe we could reduce class size in those grades to make those jobs more appealing we could also have other amenities for them for instance occasional sabbaticals to attend graduate school or do something like that something to make them want to take those jobs and stay in those jobs and compared to the curricular changes in the charter schools i am going to focus more on the teacher value-added experiment because it's not only the most informative but in my opinion it's the most relevant to policy there is now a very extensive body of research that shows that individual teachers matter and that they differ substantially in their value added even within the same school and the same grade therefore any policy maker could propose to pay them or hire them differently as mentioned earlier to induce them to teach particular grades now i'm going to use the most validated method of estimating teacher's value added and it may seem a little complicated when i describe it but in fact it is intuitive if we compare the problem to a simple randomized controlled trial which is what the method mimics suppose that there are two third grade teachers smith and jones they have classrooms located right next to one another and every year the school principal randomly assigns 20 students to smith and 20 students to jones something like that then smith and jones are essentially conducting a very small experiment every year 20 getting the smith treatment 20 getting the jones treatment right and so we can answer the question if we did this for a while we could answer the question do students systematically gain more skills in smith's classroom than in jones's classroom now of course in any one given school year smither jones might get an unlucky draw of students for whom learning is difficult and even if we try to control for each student's characteristics such as poverty race ethnicity and gender and try to control for their prior achievement before coming into smith's or jones's classrooms there will probably be some unobservable differences in their two classes tendency to learn but if you repeat the experiment for several school years it will tend to iron out this problem simply because of the randomization it's not going to be that smith is always lucky or that jones is always unlucky so as long as you have multiple years of data think of them as multiple little experiments on the same teachers and you have students prior achievement and their characteristics and you have an ac and you have an accurate and accurate measure of of achievement for each year then we can calculate a teacher's value added with reasonable precision now those who enjoy statistics can listen to this brief description of the method and others can tune it out so i'm going to i'm going to give you what the method is but if you don't like statistics you can just put your fingers in your ears okay calculating the value added of teachers i told you the method is more complicated than my smith jones example is you regress a student's test score on his or her prior test scores you could use a different outcome by the way it does not have to be test scores it just happens that people usually describe it in terms of test scores and also in their predetermined characteristics such as poverty race and gender actually regressing them on the predetermined characteristics often doesn't matter very much as long as you have some prior test scores you take the residuals from this regression and you sum them at the level of the teacher by year by class then you take each sum of residuals and regress it on the other sums of residuals for the same teacher but in different years and the prediction from this last regression is the teacher's value added this is the method that is most validated in actual randomized control trials where researchers are able to assign students to teachers and it also produces something called the correct shrinkage which really just means that luck is not going to dominate your estimate of teachers value-added smith always being lucky or jones always being unlucky something like that okay so i'm going to apply this method to the north carolina data that i've already mentioned and then having obtained an estimate of each teacher's value added i'm going to do a pretty simple exercise so what i do is i'm going to plot the distribution of teacher value added i'm going to do it differently for math and reading by grade and i left out some grades to keep this chart from getting too busy i'm going to always show and i've made a mistake by the way on the legend of the chart so it should say grade three is in the light blue or cyan then it should be grades six seven and eight not five six and seven so keep in mind it's always six through eight are the middle school grades and they're all plotted in a kind of orangey um color and it doesn't matter you can hardly tell the three lines apart anyway so i just plotted them all in sort of the same color spectrum and then i'm also going to plot the distribution for grade 12.
okay so what do we see when we look at this chart the third grade distribution and i should say by the way it's a mechanical fact that teacher value added is always pretty much centered around zero that's part of just we know that uh smith may be better than jones but teacher value-added is about their relative ability to teach students skills and and get students to learn it's not really about um you don't want to take the numbers on the bottom are less important than the shapes of the distribution that's how maybe you should think about it and these distributions are smoothed a little bit as my fellow economists will recognize that those graphs come from a program that we all like to use a lot okay so if we look at uh the distribution for third grade um teacher value added you'll see it's centered on zero and a lot of the mass or the density is piled up right around zero and it's a quite narrow distribution so what's the interpretation of this it means that if i get an unusually good third grade teacher that is better than getting one who's unusually bad but the difference between an unusually good teacher and an unusually bad teacher in the third grade is not all that great in terms of the value added that i'm going to get out of her classroom now if you look at grades 6 seven and eight in this sort of orangey colors you can see that these distributions are much more spread out they're still centered around zero but the density or the mass is much lower than it is in the light blue distribution for the third graders and they're just more spread out distributions so now if i happen to get unlucky with say a grade six teacher or grade seven teacher i would be way down here okay and if i get lucky i could be way up here so middle school teachers have bigger differences in value added than third grade teachers do so your luck in being assigned a good teacher or the bad teacher or say the less good teacher as a student matters more and one standard deviation below the mean is just much more meaningful then finally the 12th grade distribution that's the one that's in purple and that shows you that 12 the 12th grade distribution is more spread out than the third grade distribution but it's kind of between the distributions for the third grade and the distributions for grades six seven and eight uh this is consistent with some hardening of this student's trajectory of cognitive skills and later adolescence as i discussed in yesterday's lecture i would put it this way it's just harder for a 12th grade teacher to change a student's skill much in either a positive or negative direction than it is to change the cognitive skill of a sixth seventh or eighth grader now one might worry that teachers are assigned to positions in such a way that middle school teachers are simply less similar for some reason in their efficacy for instance if you think about all of these rookie teachers you might say well they're just less similar than teachers who have greater seniority so to test this possibility this next figure which looks just like the previous one so don't worry about that is based on by grade differences and value-added for the same teacher so the way to think about it is this if i teach sometimes in the sixth grade and then sometimes i'm teaching in the fourth grade or something like that then i'm only looking within that same teacher so it is not about assignment of some teachers to the sixth grade or some teachers to the fourth grade a teacher has to be moving among the different grades to be included in this figure and you can see it looks tremendously like the previous one i mean i i have to like look carefully so i can tell them apart from one another but it is actually making a different use of the data it is looking within teachers so we don't have to worry about that potential assignment problem and in this figure of course what we can see is that the third grade distribution is much more narrow and has a lot of mass or density around zero and the teacher value-added distribution for um grades six through eight is the most spread out and grade 12 is uh someplace in between so the next two figures i will show them just briefly are for um reading this is between uh different teachers and this is within uh different teachers so it's just it's just to let you know it's not just all about math reading teachers also differ uh significant they're significantly in their value added across across reading now the next figures that i'm going to show you are going to do another exercise and they differ only in the outcome that i'm going to study so for instance i'm going to study outcomes at the end of high school like how well you do on college aptitude tests as a student your sat scores or your act scores sorry [Music] i should have silenced that before i got started so the next figures are going to show the result of this other exercise that's fairly different and they're just going to differ in which outcome i look at at the end of at the end of high school for instance your sat or act scores uh or whether you enroll in a four-year college and then what i'm going to do is i'm going to say well does it matter when you got the teacher with the high value added for your outcomes at the end of high school and the figures are a little bit complex so it's worthwhile explaining the first one carefully and then once you understand it the others are all sort of the same they just the outcome just changes all right so here's the figure what we have is grades going across the bottom uh grade 3 4 5 6 7 8 in north carolina for reasons that i don't need to explain there is really no grade 9 testing grades 10 and 11.
Okay so what i want you to see here is um i'm regressing a student's math or act scores they're all converted to the s.a.t scale so don't worry about that on her third grade teacher's value edited math then her fourth grade teachers value edited math and so on so this first one is saying if i had a teacher who was better than average in value-added in the third grade how much does that make me have higher sat or act scores at the end of high school okay same exercise but for the fourth grade fifth grade six seven and eight and then ten and eleven okay now if a teacher's value added had the same effect on you regardless of the grade in which you encountered that teacher then all the bars would be of the same height okay they're obviously not all of the same height and most notably the ones for grades six seven and eight are significantly taller than the ones for um the other grades the elementary grades but also the two high school grades so that really looks like having a middle school teacher who has high value added is what's going to have um the most effect and it's not just that you know you take the sat at the end and so high school would matter more than middle school because you can see high school actually actually matters less so the figure demonstrates that a middle school teacher can change a student's letter outcomes more than elementary or high school teachers and this evidence confirms the idea at least to me that students are especially plastic when developing advanced cognitive skills in middle school and having an effective teacher in those grades sets a student on a steeper trajectory with endogenous skill growth ultimately producing substantially higher sat or act scores now i want to note the role of endogenous skill growth here something i emphasized a lot yesterday so let's look at the number 32.6 for seventh grade it does not mean that having a teacher who's sort of plus one in terms of a standard deviation raises students scores by sat scores by 32.6 points in any immediate way rather it's having a teacher who's better in the seventh grade allows you to learn a little bit more in the eighth grade and then you learn a little bit more in the ninth grade because you earned a little bit more learned a little bit more in the eighth grade and the whole effect of that is the is the 32.6 points okay sort of it's like compound interest that's how i like to think about it you can see how it adds up over time so the next figure is similar to the last one except that it's going to look at teachers value-added in reading and actually they're value-added in reading the effect on your verbal sat or act scores is even bigger than the effect of those the value added in math on math s.a.t or a ct scores so otherwise it's it's not very different and again i think endogenous growth is really playing a role here because those numbers are quite big so they're it's not it couldn't really all happen in one year it sort of needs a few years to have like compound interest to build up now sat and act scores are loved by some people and hated by others so there are other ways to measure whether students are taking taking on cognitive tasks that are truly challenging so the next couple of figures i'm going to look at college board advanced placement tests in important subjects because the college board advanced placement tests have the same curriculum and the same test across all the high schools in the united states so i don't have to worry about some grading standards being easier than others for instance this figure is showing that if you have a teacher who's a plus one in terms of standard deviation in seventh grade it raises your probability of taking ap calculus by 6.7 percent or if you have that teacher at eighth grade it raises your probability of taking calculus ap calculus um by 8.8 now obviously they're not taking ap calculus in seventh and eighth grade it's that later on having had this teacher uh allows you to um end up being more likely to do that there's a very similar result for ap science classes i'm not going to show you the figure it looks somewhat similar to that and they're not taking ap chemistry in the seventh or eighth grade either they but it they're having had a better science teacher or a better math teacher in the seventh or eighth grade makes a big difference a teacher's value added in reading this is on taking ap english by the grade in which the teacher taught students and again it's actually bigger than it is for math which i find fascinating for instance having a high value-added teacher in the 7th or 8th grade raises the probability that you take ap english probably in your 11th or 12th grade year by by a little more than 10 okay so finally oh i have two more finally not finally let's look at gpas as an outcome this is the effect of teacher value added in reading on your final gpa in high school by the grade in which the teacher taught students so again we see this disproportionate effect of the middle school grades on students final gpas in high school and the math version is similar but i won't show it to you and then finally i wanted to consider the effect of having a high value-added teacher on enrolling in a four-year college so and that's because i think four-year college is a very authentic and important outcome for many people you know going to colleges one of the things that changes their lives so this figure is showing you that having a teacher who's one standard deviation better in math in grades 6 7 or 8 raises your probability of enrolling in a college by 6.3 to 8.1 percentage points and by the way the alternatives to enrolling in a four-year college are employment the military enrolling at a one or two year technical or community college or just plain inactivity which is actually quite common among young people so interestingly encountering a teacher uh with value-added reading of plus one has an even bigger effect than having math of plus one for college enrollment and the effects of seventh and eighth grade reading teachers are more than 16 percentage points more likely to go to a four-year college that's an impressive amount again probably due to endogenous skill growth i think the impressive facts of value-added in reading on later outcomes demonstrate that the critical reading skills that we develop through reading and writing are just as if not more important for a person's long-term outcomes like college attainment and this is consistent with the fact that an inability to process college-level materials or read college-level texts is a consistent stumbling block for the many students who regardless of their preferred major find it hard to absorb material and they end up dropping out of college for that reason okay now i'm going to switch gears we're going to stop looking at teacher value added and instead we're going to talk about introducing a cognitively more challenging curriculum now the state of texas has a long-standing and fairly high stakes accountability system it's based on students test scores not only did the schools themselves get graded but students need to pass an exit exam in order to get a high school diploma in texas and also promotion to a higher grade is somewhat dependent on having done well on the tests at the end of that school year although there's more discretion apart from the exit exam the final one in addition texas is only one of two states that select the textbooks for its schools so it has an unusual degree of influence over curriculum in practice though i think that curriculum is more often influenced by tests than by textbooks a teacher can ignore the new textbook that is put into her classroom and just decide they're always giving me new textbooks i don't want to have to learn how to do this with this new material i don't want to have to learn new problem sets i don't want to have to learn new readings i'm just going to ignore the new textbook because i know in two or three years there will be another new textbook that will come down the road but a teacher cannot ignore the skills that are being tested by the exam that her students will take at the end of the school year and typical a testing regime typically lasts at least a decade so the one that we're going to be talking about in texas has been in place for more than a decade at this point so therefore schools and teachers paid a great deal of attention when they were switched starting in 2011 from the testing program known as the texas assessment of knowledge and skills i'm just going to call it talks from now on so i don't have to keep saying that long phrase to a new program which is known as the state of texas assessments of academic readiness or star so from tox is the earlier one in star is the later one now the switch is interesting because making the tests more cognitively challenging was the explicit motivation for the introduction of star to make the tests more cognitive cognitively challenging the creators of star designed questions that require critical thinking more often than the talks test did the start questions were deliberately constructed to make it hard for teachers to teach to the test or coach students in test taking strategies that tend to work well on a multiple choice exam start questions often ask students to provide an answer and not just only guess over multiple choices or choose over multiple choices there's there are three short essays on the star exam each in a different format whereas talks had no essay requirement and to assess whether a student truly understands a concept this is a kind of subtle thing the star exams are timed why it's because time pressure makes it hard for a student to simply try out each answer on a multiple choice test but the tax exam you could take as much time as you liked so you could just try out each answer you didn't have to really deeply understand how the problem worked and we'll see an example of that in a moment and star tests also focus on the content that should have been taught in the grade in question thereby assessing whether students are making progress vis-a-vis an ever-more challenging curriculum and talks tested content that was often way below the grade level of the students and so an eighth grade student could possibly feel like he or she was doing quite well on the test even though he or she was only really answering the questions that were for fourth graders or something something like that so to make these i made all these claims about this difference in these tests and it's going to help to examine questions from released prior year tests uh the texas state department of education releases prior year tests not quite all the questions but you can download them very easily from from there from their website i'm just going to show you a couple here so the first one is this was a talks math question for seventh graders and i'm going to keep showing you seventh grade questions because they seem most relevant so which list of integers is in order from least to greatest is it negative 42 negative 39 negative 4 40 41 or is it 41 40 negative 4 negative 9 39 negative 42.
So this is just not a seventh grade level question right it's a very easy question that a lot of fourth or fifth graders should be able to get right or here's another example of a tax math question for seventh graders this is really an arithmetic question it has nothing to do with algebra it simply requires you to divide nine by three that's obviously three then square it and you get 9 divide multiply that by 5.
you've got 45 and add 5 so you come up with 50 as the answer to the question but these are easy easy math questions for a 7th grader in contrast if you look at this star question for seventh graders you can see that this is basically an algebra question it also requires you to understand that this is a cartesian diagram you have to understand negative and positive numbers two different axes x and y you have to understand how to think about slope of a line and you can see that what you have to choose among they are multiple choice but they're four different algebraic equations that represent that line so it's a significantly harder question that requires more more critical thinking now i told you that i hope we would get to something that's rather funny so i want to show you a talk's seventh grade reading question so this is again the older test that's less demanding it focuses on an austin texas festival known as spam arama okay that's after that after that canned processed meat that we all know spam do not love maybe but we all know okay the contest has two divisions one for professional chefs and restaurant owners and one for amateur cooks in the amateur division everyone is welcome to show their stuff one contestant entered the contest spamarama with a dish that was a mixture of cheddar cheese mayonnaise spam and raisins the dish's poor rating at the contest did not deter this stubborn individual hoping to find a more accepting panel of judges the next year he froze his entry and brought it back the following year in keeping with the spirit of the event the judges decided to create a last place even if there were a hundred entries award just for him okay so that's the paragraph that students have to read on the tax test and the question was one contestant in spanorama froze his food entry because he planned to carve it he missed the entry deadline he wanted it to be eaten cold or he thought he deserved to win so this is a very easy uh reading comprehension question for a seventh grader now if you look at star's question it's obviously much harder it's on a more serious subject as well uh it's about um sainsbury's which is a big grocery store chain in the united kingdom sort of like the safeway or a p of the united kingdom and they're talking about food waste this is a longer article that is about three pages long and relatively complicated so i'm i've picked out just one paragraph and i hope you get the basic idea of just the one paragraph would be better to read the whole thing too much time so in spite of sainsbury's efforts a large volume of waste in other words food waste remains a machine grinds the waste into slushy goo which is then poured into giant silos called anaerobic digester digesters sorry these giant silos act like artificial stomachs inside microbes digest organic waste and produce methane bubbles the same thing happens to organic waste in landfills and waste treatment plants the difference is that these anaerobic digester silos are tightly sealed so they can capture and store the methane the resulting biofuel can power vehicles or it can be burned to produce electricity sainsbury's management estimates that its ads those are the anaerobic digesters can produce enough energy to power 2 500 homes for one year or it can make enough electricity to completely remove one of its stores from the public power grid so this just requires a heavier level of understanding there are actually a few hard vocabulary words like anaerobic or biofuel in the in the reading example what is the most likely reason that the author wrote this selection so they're trying to get the student to think about what's the structure of the argument what argument is being made and you can read the answers for yourself i think the correct answer is to demonstrate that creativity can help to solve environmental problems but it's undoubtedly a significantly harder question than this panorama question i think we would all agree all right so to test whether the switch from talks to star affected students cognitive skill development i'm going to focus on two cohorts in particular the students who entered the ninth grade in 2011 that was the first year of star they experienced star heavier or more difficult exam throughout high school however this cohort did not experience any star-driven more cognitively challenging curriculum in any of their middle school grades the second cohort that interests me is composed of the students who entered fifth grade in 2011 so they experienced the more cognitively challenging curriculum both in middle school and in high school and the difference between these two cohorts is not their experience of the star-driven curriculum in high school because they both had that but only the second cohort experienced star-driven cognitively challenging curriculum in the middle school so by comparing their later performance on the sat we can test whether it is crucial to come up against challenging material in middle school unfortunately a simple comparison between these two cohorts is not all that straightforward and that's because the college board undertook a major redesign of the sat starting in 2017 just when just with the worst possible timing so not only did the redesign change the scoring but also fear of taking a new test for which they had not prepared caused many students to take the exam in 2016 and when they ordinarily would have taken it in 2017 and in addition of course testing in 2020 was affected by the pandemic so i have done my absolute best to make sure using the um the concordance tables that everything is in the right uh scores uh there they are rationalized with one another um but i still need to have a control for texas that did not experience the talks to star transition of curriculum but did experience the same external events like the redesign of the sat so to find a control i'm going to adopt the method called synthetic controls this method combines all of the possible controls for texas so in other words all of the possible states that could be controls for texas to create a synthetic texas and the weights on these other possible controls are optimized to match texas's time pattern in sat scores as well as possible in the pre-star period in other words in the talks period and the weights and the controls are then validated outside of the optimization period so it has to also do well out of the the period with which you use to generate the weights the weights have to do well when you go out of sample and see that they can still be valid okay so that's a very important part of using synthetic controls so this all sounds a little complicated but it's actually very intuitive i love synthetic controls so texas is a big state with a variety of cities and landscapes and industries some of texas is a lot like oklahoma it's neighbor to the north they share oil drilling and farming and some of that sort of thing some of texas is a lot like louisiana with similar refineries and ports on the gulf coast some of texas is a lot like new mexico they share a desert they share a lot of cattle ranching and so forth and so on austin texas has a lot of similarities to other cities that are dominated by a state house and a flagship public university dallas has some similarities with other financial hubs i could go on and on what's the point here the point is that none of these other states would be a good control for texas in and of itself but if we wait them all up optimally we can create a synthetic texas that walks talks looks and acts like texas so we're going to be comparing texas and synthetic texas and a properly constructed synthetic control should show how texas school students would have done in the absence of the switch from the talks based curriculum to the star-based curriculum so i'm just going to show you one figure here that's a little bit complicated so first of all this is the period that in talks okay before star was put in and this is the period when the weights are constructed for the synthetic control the synthetic texas which is in the dashed sort of purple line and texas itself is in the green line okay so you can see both in the weight construction period and in the validation period before the switch of tests the synthetic texas is doing a very good job of looking a lot like texas right they sort of have the same things happening to their students test scores on the sat the pink line is the last cohort that had no early adolescent grade 5-8 exposure to the new star exam and then the red line is the first cohort with full early adolescent exposure to the new star exam what you'll notice is that up until you get to the pink line the two lines the synthetic texas and texas track one another very well but then after the introduction of the star exam texas is doing significantly better than synthetic texas and that's the idea right that synthetic texas is telling us what would have happened in texas if they had not switched if they had not switched tests one thing i wanted to mention about tests like star is that they're a very manageable policy reform of course it's not that easy to construct a good test that's what psychometricians are for many educators have to get involved but it's actually quite cheap to construct a test testing is not expensive by the standards of school finance in the united states finally i told you this story early um in the talk and i'll be very brief here to stay on time about charter schools and one set of facts that motivated me was the strong performance of charter schools that started grade 8 as their entry year and used grade 8 started grade 5 as their entry and used grade 8 as their exit year and these school students often made annual test gains that were two to three times the gains of schools with kindergarten entry or grade 9 entry and at the time as i said these results struck me as very counter-intuitive because i expected that the schools that started with kindergarten would produce the most positive gains in student achievement i really had low expectations for schools that took in the negatively selected applicants at grade five because i thought of them as students who had missed the boat of strong early childhood education and were already struggling in the regular public schools so this was all based on my idea which was then popular in economics and continues to be very popular among economists the children are only very plastic in early childhood so to my mind these struggling fifth graders had already been fated to have a lower level of cognitive skills so my study used more than a hundred new york city charter schools and it used lottery based methods i mentioned these before they're very simple students apply to one or more charter schools simply by filling out a one pager with their name their address their parents contact information and so on it's called an application but it's really just basic information and once the charter school has received all the applications it holds a lottery in which every child is assigned a lottery number so for instance if they want to admit 60 kindergartners the first 60 kids would get seats in the charter school and if you were below the cutoff for your lottery number you would typically not get a seat there's a bit there's a bit more to it than that because the lottery didn't and the lotteried out students we only want to use that lottery information to come up with estimates we don't want to use students who might have been picked off a waiting list or something like that so there are some bells and whistles but i won't go into the details this method is generally considered to be the gold standard for evaluating charter schools and lottery based studies tend to have similar findings across urban settings where the application lotteries are over subscribed so it's not as easy in rural areas because you might just not have too many students apply so this next figure shows you the effects of a year in charter school on math scores it doesn't really matter whether you look at the blue bars or the orange bars so that has to that's something about attrition and i didn't just use my own study of new york city charter schools i used all of the lottery-based studies that i could find where it was clear what grade was being uh was being shown for achievement for instance the kipp schools are a charter management organization that ordinarily starts uh charter middle schools with the fifth grade and so they're covered by a lot of those middle bars and you can see that their kids are just learning at a faster rate in the middle schools the middle the middle charter schools than they are learning in the high schools or the elementary schools and for new york city where i can do it i know long-term outcomes and it is also true that if you look at long-term outcomes like post-secondary attainment whether you go to college or not the effects are significantly bigger for students who are lotteried in in the fifth grade in fact the effects for them are almost four times as large as they are for students who were lotteried in in the in kindergarten at the beginning of this lecture i made the case that early adolescence um early adolescents are relatively neglected they have a lot less spent on their instruction they're in larger classes they have teachers who are rookies and they have teachers who are eager to stop teaching them and start instructing older younger children although this relative neglect is unintentional in my opinion it is perilous because adolescence is a crucial period of opportunity and also a period of risk owing to the plasticity of advanced cognition at those ages and the gradual hardening of children's cognitive trajectories thereafter now neuroscience and neuropsychology were behind much of my motivation to test for exceptional plasticity in adolescence and i believe that neuroscience and neuropsychology have a fundamental quality that makes them different than the kind of evidence that i've shown you today from education they should really apply across the whole world it shouldn't be the case that neuroscience for american they we just have different brains than brains of people in other countries in fact i'm going to be doing some work in the future on the uk and i would be uh horrified if i found out that they had completely different brain development than we did so that's the really nice thing about neuroscience is that it's important for learning universal truths about cognition in my opinion what i like about the education experiments is that these natural or policy experiments that i described today are all realistic interventions that are deployed in some schools already and could be deployed in more schools for instance concentrating the most effective teachers in early adolescent education or higher pay and other rewards that induce effective teachers to apply to and stay in middle school classrooms more generally the importance of early adolescence makes me optimistic about the future of american education and americans advance cognitive skill growth if you think about it the idea that only very early childhood education is matters is somewhat disheartening i would be the first to agree and neuroscience would support the idea that the period from three months before you're born to age two or three is crucial for brain development and later life skills but it is disheartening to think that kindergarteners might already be fated to a trajectory of low modest or high cognitive skills i just find it dispiriting to think of writing off a child who is early in elementary school yet this is the implication if we focus exclusively on early childhood spur to endogenous skill growth to the detriment of adolescence and its important effects on advanced cognitive skills which are so important to our economy and our today's fast-paced world and i know that uh eric is going to be discussing some of the macro implications um of this for our economy and uh just the larger the larger picture so the last two bullet points are especially aimed at aimed at eric i find hope in early adolescence precisely because it is a later period when society can have more influence over a child's educational experience no child masters abstract math or rigorous critical reasoning before early adolescence and i find this to be a good thing paradoxically because it means that children enter this quest for advanced cognitive skills on a relatively even playing field they didn't say it wasn't even playing field but it might be more even than we suppose when i look out over the united states and other highly developed countries what tends to worry me the most is economic fatalism among young people who lack advanced cognitive skills and therefore feel that they face a bleak future owing to technology and globalization now of course it's not that non-cognitive skills are useless we need service workers who can perform important tasks that are challenging on dimensions other than cognition like strength or dexterity or nurturing or courage however the demand for these tasks is declining in highly developed economies while the demand for advanced cognitive skills is rising i am convinced though that many more children could be sat on a positive high growth trajectory towards advanced cognitive skills during the plastic period of their adolescence of course not all children will attain these advanced skills but the more we see adolescents as an age of opportunity the more likely we are to offer these children opportunities rather than neglect them thank you very much thank you uh we're going to break for 10 minutes and come back and i'll introduce the commentators and following that we'll have a discussion thank you very much please um join me in thanking professor hoxby again for her wonderful lecture [Applause] our first commentator eric hurst joins us from the university of chicago the faculty of which he joined in 1999 and where he is the frank p and marion are diazi distinguished service professor of economics and the john e jerk faculty fellow at the booth school of business professor hurst's luminous and voluminous scholarship focus on focuses on housing markets labor markets and household financial behavior thematically and methodologically he ranges far and wide and over some of the most vexing problems of our day the welfare losses to society of racial discrimination and urban gentrification the economics of the covert pandemic leisure inequality and the correlates and causes of the persistent if narrowing gender wage gap among the college educated the recipient of multiple research awards professor hurst has also found time to be a remarkable teacher earning no fewer than not two not three but four teaching awards in his 20 odd years at the university of chicago please join me now in extending a very warm welcome to professor hurst thank you so much it is great to be in person with people that is not my children and it is wonderful um and so as caroline kind of foreshadowed i'm going to do two things over my two days so tomorrow i'm going to talk a little bit about the role of advanced cognitive skills versus just completed years of schooling is there something about specific skills versus just getting a degree that shows up in labor market outcomes so that's going to be a little bit tomorrow but today i wanted to set the stage and try to show why the importance of accumulating skills in general are becoming more and more important for the labor market that we live in today and so i'm going to do four things today in my my comments the first of which is i'm going to show you some patterns about inequality along a variety of dimensions and how they've been evolving over time in the united states particularly starting in the early 1980s second i'm going to talk a little bit about potential drivers of these changes in inequality because again if we want to think about a solution we might want to think about what might the causes of be the third thing i'm going to try to do is then i'm going to set up and say well how does economies adjust to changing in demand for certain types of labor and you'll see why that's going to be important in the fourth part of my comments where i'm going to link it back to some of caroline's uh talks over the last two days so that's kind of the big picture overview the the the component of where i'm going to link to to caroline's lectures is going to come at the end but i'm going to try to try to create an umbrella where you could see the importance of this research for some of uh what i think are the challenges that we are facing in a socio-economic way in in the 21st century so let me just kind of start by a picture that you might be familiar with this is going to be a measure of income inequality in the united states it's the share of all income earned accruing to the top 10 of the population and so the united states in particular is seeing a increase in the share of income accruing to the top 10 percent whereas in current periods we're talking about almost half of all income is owned by the top 10 percent of the u.s distribution so now i'm going to focus a little bit more on breaking things out by different education groups and so today i'm going to show you a lot of patterns by people who have a bachelor's degree or more versus that people who have less than a bachelor's degree and just with that as background and again when i teach my students this it is often striking to them but just as a reminder that we are rare in the population in terms of having a bachelor's degree and so this shows the share of i'm showing you men right now i'll talk about women as we go through but the share of men 25 to 34 who have completed at least a bachelor's degree by the time they are in that age range and the couple things i want you to you know kind of focus your eye towards is first you know the number that we're looking at is somewhere between 25 and 30 percent so most people in the united states don't have a bachelor's degree and again through the worlds we live in we tend to focus on you know our reflection and i just like us to remind us that our reflection is not the average by any dimension but the second and more important thing i want you to show you is that this share for men for women you've seen much more steeper increase but the share for men has been relatively stagnant over multiple decades and you see a little bit we talked a little bit about this in passing yesterday about coming out of the vietnam and the gi bill you see a little increase but basically from cohorts over the last few decades it's been relatively flat a little uptake recently so we have a group where the returns to skill is going up but yet the movement into skill at least by at least getting a bachelor's degree have been relatively stagnant until potentially recently and so this is what i focus um some of my work on is trying to figure out the difference in behavior in the labor market between those with a bachelor's degree and those with less and then trying to put on a little bit more what particular skills are necessary but this is just things that you might have seen in the newspaper which is just the average earnings of someone with a bachelor's degree relative to someone without a bachelor's degree and how that's evolved over time so a number like 30 percent means that on average a typical person with a bachelor's degree is earning 30 more in a given year than someone a typical person without a bachelor's degree but the key thing i want you to focus on is notice that has been increasing over time so part of the inequality we're seeing is showing up in completed years of schooling where the earnings are growing for those with a bachelor's degree at a greater rate than those without a bachelor's degree and that brings me to this next part which again we're starting to get into socioeconomic issues that are kind of tangible or things that we're feeling i'm showing you here now the employment rate of men and again i'm focused on men 25 to 54 here without a bachelor's degree and so there's a couple things your eyes should be drawn to there's ups and downs over time those are due to things like recessions so when recession less people are working you can see in 2008 there was a lot less people working in 1984 there was a lot less people working but i want you to focus your eyes on the downward trends when the things are peaking those peaks are getting lower and lower over time and so we're at a position where currently about 82 percent of men this is pre-covered so i stopped it right pre-covert 82 percent of men in these prime working years post you know college getting pre-retirement getting are working where that number used to be something like you know around almost 90 percent so we get this downward drift in employment for a group of individuals with less than a without with less than a bachelor's degree and if we want to compare them to the men with the bachelor's degree you just don't see that same downward trend you see slight trends but it used to be about 94 and now we're something like 93 but a really a concentration at the bottom of the distribution at the bottom of the bulk of the distribution of of employment rates that are falling over time and i promised some depressing stuff when we were talking yesterday john was pretty depressive pressing i'm going to go there right now with this picture this one is the one that that i find stark and i pause every time i'm giving in in some of my own work or when i'm teaching this is the share of people again men 25 to 54 without a bachelor's degree who report working zero weeks during the prior year so we have lots of government surveys that samples people working and they call them up on the phone do you have a job they say no they said did you work at all in the last year and they say no and so this is the share who report no they're not working at all over the prior 52 weeks and what i want you to see is that historically you know it was about 1 out of 20 men in this age set idle for a whole year that number is now one out of seven so there's a large chunk of our prime age population again men you see similar patterns for women not so much in terms of the increase but men with a a you know less than a bachelor's degree sitting idle for long periods of time and that has changed relative to where we were um you know 30 40 years ago okay so we have these shifts inequalities rising it's showing up in labor market outcomes in particular in the propensity to work what are driving some of these shifts and how do we kind of interpret it and so there's two stories in caroline touched on them yesterday i'm going to show you a little bit of evidence of them you know things you hear about in the newspaper such as automation and globalization so how do we think about changes in automation and how and changes in globalization and affecting these patterns that are shifting inequality over time okay i want to focus on one industry and i do a lot of work just thinking about this industry and you'll see why why it's so important but i'll show you stuff about the manufacturing industry and how that's changed in the united states over time and so what i'm showing you here is just if we count the number of manufacturing jobs in the united states and how that has evolved over time and so in 1980ish there was almost 20 million about 18 19 million manufacturing jobs in the united states so people who report working in the manufacturing industry and you can see that has been declining over time with an acceleration of that decline in the 2000s and so about 6 million manufacturing jobs disappeared from the united states between 2000 and 2010 and there's been essentially no rebound in that period afterwards so what are the two stories that people have that link to these decline in manufacturing jobs one is openness to trade and so there is a bunch of our colleagues who have worked in economics on a variety of kind of angles showing that opening up to trade in particularly trade with china increased the competitive forces in the manufacturing sector and manufacturing employment as a result increased but that's not all the story and i won't say it's not even probably the predominant story because during the same time period manufacturing output went up so how are we getting more output with less workers well we've been switching our manufacturing production from being labor intensive to being capital intensive or automation intensive and so that shift that has been incurring has been reducing not the demand for manufactured products produced in the united states that has changed some with trade but it has resulted in a decline in the manufacturing workers in the united states shifting towards machines and so this decline in manufacturing does not hit all schooling groups equally so let me show you some data now on the share of workers of different schooling levels and what percentage of those worked in manufacturing in given years so how do you read this picture it says if you were an individual with a high school degree or less in 1977 essentially one third of you worked in the manufacturing sector out of all people not only those who are working but across all individuals so the manufacturing sector for those with less than a high school degree nationally not just in detroit nationally was about one-third that number now is somewhere around 12 percent so we have this big sector that was important for workers particularly with a given skill level and that sector has shrunk and more importantly not only is it shrunk conditional on working in that sector you know they don't need as much of the high-skilled workers or low-skilled workers low educated workers as they do they need people now to fix robots as opposed to to work to work the line and so i'm going to show you lots of pictures like this more today tomorrow than today but i'm going to show you a couple today and i'm going to focus on a region that caroline focused on a lot in her picture so if you take a remember from her lecture earlier today and from yesterday where these places where these advanced cognitive skills are kind of missing spatially in the united states those are also the same places for which manufacturing intensity is high it's not exactly the same and i'll show you some pictures and more tomorrow then again today but each one of these areas basically show the intensity of manufacturing production in that county in 2000 so the darker the red the more manufacturing intensive that area is this should not be surprising to any of you manufacturing is clustered in the united states around kind of the the midsection and again just as a reminder this is caroline's picture she had it in pretty blue i don't know why i put it in black and white but you can see the places that are missing some of these advanced cognitive skills are exactly the exactly the same places now in some of my work i've actually shown that the places where manufacturing declined the most or where manufacturing was most present or also the same places that manufacturing declined the most are exactly the same places where non-employment rates have risen the most and so you see this area where manufacturing has shrunk in certain areas and those are the same areas where now male participation particularly concentrated for those without a bachelor's degree have fallen the most within the united states so there's some link between these aggregate forces of automation which is particularly effective the manufacturing affects other sectors too but affects the manufacturing industry and that is caused um you know these employment declines um that we've seen uh in in in the aggregate data so we'll do one more picture before i get to some of you know how do we adjust to these shocks but this is kind of something that carolina alluded to towards the end part of our our discussion so what this is now is take a look think about every job in the u.s economy okay filled with some people and if you go to those jobs some of us are teachers and some of us are professors and some of us are lawyers and some of us are shop clerks and some of us are manufacturers and then for each one of these jobs there's some skills that are needed to fill those jobs and so there's government surveys that says if you are a you know sales clerk what type of tasks do you tend to do on a regular basis and then we could take those task measures and project them on some i'm going to use these words like advanced cognitive tasks those are kind of you know you know using caroline's language or we could have things that are kind of routine tasks and so advanced cognitive tasks are things that are going to require some amount of thinking as part of our jobs complex problem solving so things like lawyers and professors tend to be higher in this type of task need than you know bank tellers or sales clerk and then there's some other types of jobs that again have these routine things that you do the same thing kind of in a repetitive way so my first job back in in high school my second job my first job i worked under the table at a golf course but i'm gonna forget about that if anybody's listening that might have that might have got rid of my you know uh chance of being the fed chair for saying that okay but okay my second job was working at wendy's and so there you know you just kind of worked the the cash register it was a very routine type of oil you know there was not much variation i'd come up i hit a button and people would give me money i hand it back what this is showing though is that the demand in the us economy for skills that require complex uh analytical tasks is rising over time in the jobs that require these routine tasks that are kind of repetitive you know manufacturing is a lot in those green lines is also declining over time so the skill mix for which the labor market is requiring has been shifting uh in a given in a given particular way favoring these advanced cognitive skills that caroline was talking about so this brings me now to kind of a bigger question which is how do economies adjust to the demands for you know occupation shifting over time what happens when some sectors go away that require some skills in new sectors are born that are require others and so you know i like to you know what i'm teaching we've gone through lots of disruptive periods in an economy over our you know centuries of existence and so if we go back a hundred years ago most of us were in the agriculture sector and in particular you know about one-third of all men worked in agriculture in the beginning of 1900s and then a robot came along we call that robot a tractor but it was some automation nonetheless and transformed this manufacturing this agricultural sector but yet the u.s economy weathered that disruptive shock right now there is very few of us that work in the agriculture sector relative to where we were before but you know that was true in 1950 60 70 but lots of people were still working so how do we respond when uh sectors disappear and other sectors grow and there's many kinds of margins of adjustment that we could think about and some of which are we could if jobs are in one place like in agriculture they tend to be in you know more rural areas in manufacturing tended to be in you know different rural areas or sometimes even people have to move to the job so moving spatially is one way people could adjust for shocks but oftentimes the sectors that are growing in the sectors that are shrinking have different skill mixes between them now it's not always the case because when we used to be in agriculture the skill tasks that people needed in agriculture translated roughly well to the skill tasks that were needed to manufacture my dad was a manufacturer my guess is if he was born 40 years before he would have been a farmer the skill mix that my father would have had translated nicely between those two sectors but now we're in periods of time when manufacturing is declining in the skills that the jobs that are growing are requiring these advanced cognitive skills and as a result we need some other type of adjustment to occur to fill the demands of the the labor market that is moving today and that means acquiring some new skills and we could do that by and potentially going to college or training programs or other forms of human capital adjustment and the difference is those adjustments take time and so now again this is just the umbrella to get to this last part which is you know i think caroline's work is so important because it's highlighting how we could think about one margin again we'll probably talk later on there's lots of other margins that we might want to think about one margin of how we could start moving the economy to get help get the skills that we're going to be needing to fill the labor market you know today relative to 20 years ago or 30 years ago and so caroline could have given the same lecture 20 or 30 years ago that where the most elastic for skill equity i'm going to say that adjustment is even more important to think about getting it right today than it was in the past given the demands of the shifts in the demand for different skills that have been accumulating and so thinking about this is you know providing a way to basically say we used to focus a lot of our narrative discussion and caroline emphasized this that we really want to target people early so the head start program a lot of the government programs and we're saying maybe there are better place or at least additional places where we want to focus um some of our uh some of our energy and so i think that's why the important so tomorrow i'll have some more specific thoughts about what skills versus with others but i think in the umbrella sense why is this important because these big shifts that the economy is hitting we need to think about ways to help if there are frictions to skill acquisition what can we do as a society to help mitigate some of those frictions and so that's what my my comments are on i look forward to more discussion that's all i got okay thank you very much uh eric um next um it gives me great pleasure to introduce our next commentator professor sylvia bungay of berkeley's own department of psychology and the helen wells neuroscience institute and excuse me an expert in the cognitive and neural processes that support reasoning memory and goal-directed behavior in humans particularly in those young humans we call children professor bungay directs berkeley's building blocks of cognition laboratory her lab draws on several disciplines including cognitive neuroscience developmental psychology and education research to study developmental changes and neural plasticity in the cognitive control and reasoning skills of children and adults an author or co-author of some 196 papers uh and i believe a brand new textbook on the fundamentals of developmental cognitive neuroscience is that out not out yet just almost immediately yeah there's a uh what's it called supply uh stuck in a boat in the bay right um that happened to my textbook a few months ago too um she's one of 11 members of the multi-disciplinary multi-university national scientific council on the developing child which strives to generate public will to close the gap between what we know and what we do to promote successful learning adaptive behavior and sound physical and mental health for all young children please join me in giving a warm welcome to professor bunji thank you so much thank you rebecca i'm truly honored to be here and i really want to thank the tanner electric committee the ones i've met so far jay and kinch and hannah and everybody else for including me i also want to thank chancellor christ who was here yesterday and jane fink for organizing this really fascinating lecture series and of course i want to congratulate professor hocksby caroline on two really compelling inspiring lectures on a really important problem so today i want to begin by providing some additional context on the advanced cognitive skills that caroline has been talking about then i'm going to talk about periods of brain plasticity during development these topics are really right up my alley as rebecca is alluding to so caroline defined advanced cognitive skills as those that require higher order reasoning they require a capacity to solve problems through logic think in the abstract engage in critical thinking and derive general principles from a set of facts they are integrative they are synthesizing they are important for planning end quote so higher order reasoning which is something i've studied for many years is the culmination of a number of mid-level cognitive abilities so for one working memory or the ability to keep information in mind and manipulate it interference suppression the ability to ignore distractors focus on what's currently relevant and relational thinking which is not as widely studied but something that's really near and dear to my heart which is the ability to jointly consider several pieces of information that is to compare them or integrate them and this information could be premises it could be rules it could be concepts anything like that so trying to build these skills uh is critical uh for several reasons as we've already talked about but one of them is that many students who are in school today are going to have careers that don't even exist yet drying on discoveries that have not yet been made so we can't impart all the knowledge that they will require and secondly students today were born and raised in the information age they have a wealth of information at their fingertips what they need more than anything else is to develop the skills that are required to select relevant information evaluated integrate it and apply it in other way words they must tap into what we think of as domain general cognitive skills that you can port from one situation to another we're going to come back to the development of these skills a little bit later after a brief overview of brain development and plasticity and i will say that tomorrow i'll talk about the fact that it's contentious as to whether we even have these domain general skills but coming from neuroscience i think we have evidence that we in fact do okay so as caroline has already beautifully covered um we we see dramatic brain development in the early years this is from seven and a half weeks gestation sorry months gestation through six years these are horizontal slices through the brain created with mri and the growth here is obvious even to the naked eye so and this message of early brain development sometimes called the birth to three or the birth to five period has been used effectively as a rationale for increased investment in preschool and kindergarten in this country so as caroline was talking about head start the opposition the peri all of these programs but as we'll see in a moment beyond this dramatic growth in what's called gross anatomy not because it's gross just because it's broad is we see that more detailed measures uh reveal that the brain is not fully developed at age five or six um or even 10 15 20.
It's still being sculpted by experience long after that so caroline's message that we need to invest more in middle school education was really music to my ears because i've been making the same argument about neuroscience for many years that we cannot let our efforts to promote early child development and child brain development completely eclipse our ability to to do so for later child and adolescent development so in what ways does the brain change after early childhood i'm going to expand a little bit here on caroline's excellent summary but if you take nothing from this then just remember what caroline said okay so here we have a seven-year-old and a 30-year-old and it's hard to see the differences between them aside from the fact that this particular seven-year-old happens to have a larger head and sort of more misshapen but if you look more closely you see that the gray matter is a little bit disproportionately bigger in the seven-year-old and the gray matter is where the neurons are concentrated along with the local connections among them or synapses as caroline talked about now the brain forms well over 100 trillion synapses over the first few years of life and as caroline showed it peaks around age two and then it drops drops drops through childhood and adolescence into adulthood um and so it's eliminating the uh the synapses that aren't being used that just aren't as relevant and this pruning of these excess connections is sculpting the brain in a way that allows a child's brain to adapt to its particular environment and this is absolutely critical this is why we have this protracted period is that it allows us to basically adapt in a way that you know humans are extremely adaptable to a wide variety of environments for better or for worse right um so importantly i want to make this point that although later childhood and adolescence are characterized more on balance by a loss of synapses in fact synapses are being created and removed eliminated all the time all the time as you're sitting here listening to this lecture it's just that more in general are being removed then created so there is plasticity okay um so where is the seven-year-old oh so sorry so my uh analogy here is to michelangelo who would take on these slabs of marble and sculpt them into these beautiful things by taking away rather than adding okay so whereas the seven-year-old has more gray matter the 30 year old has more white matter which is made up of these long range tracts these white matter tracks that connect distant parts of the brain that are necessary for basically higher anything higher order any kind of higher order cognitive function requires these distant brain networks okay and so these are it's called white matter because these long-range tracts the fibers are wrapped in this white fatty tissue called myelin that as caroline mentioned promotes efficient communication between areas and so my analogy here is to the building of the golden gate bridge in the mid-1930s and so here we have the structure the white matter tracts are growing and then in childhood and adolescence these tracts are being reinforced there's more myelin wrapping around these fibers and what that's doing is to support network traffic or communication between these distant brain regions and then over childhood and adolescence we see that this traffic increases unfortunately for us that's kind of the situation we have today right here and that's already in just one year after the bridge was built okay um so we know though that the brain is not developing in a vacuum right we know that it's sculpted by experience this is a phenomenon called experience dependent brain plasticity as the author will durant once said although it's been misattributed to aristotle we are what we repeatedly do and a question that people often ask is when in the when in development is the brain most malleable that is when is it most sensitive to experience and the answer is caroline mentioned is that it depends on the particular neurocognitive system that we're talking about so the brain networks that underlie basic senses like vision and hearing as well as basic emotional capacities like attachment get wired up in the first couple of years of life and they're very hard to change after that so for example if we have a child who's born with cataracts in both eyes and is not getting pattern visual light they'll be functionally blind if they aren't operated on because this this whole circuit here that's going from the eyes all the way to the back of the brain the visual cortex isn't wired up properly okay and so ophthalmologists say that the operation should happen early ideally within the first eight weeks of life but certainly within the first couple of years okay after that that's a critical period after which it's really really hard to change anything you need the specific environmental inputs to lay down that brain architecture and so to draw a common analogy once you've laid the basic foundation for building a house it's the foundation and the plumbing is really hard to change okay so by comparison with these lower level skills we can think about something higher level like like language um this plasticity this period of plasticity is not as early and it's not as bright of a line in the sand in fact for language there's uh a series of windows of plasticity that we know about from humans as well as from other model systems including songbirds believe it or not um but anyway so there are these multiple sensitive periods and one of the earlier ones is actually learning to pronounce words in another language as though you were a native speaker and if you're not exposed to that language within the first six years of life or something it's going to be very very hard after not impossible very hard on the other hand thankfully for those of us who are trying to learn other languages vocabulary and grammar you can acquire throughout your life it's just going to get harder than it is early on okay and so here you know once you've laid the foundation of a house you can still make changes to the building you can add or remove a wall or something but expanding the footprint or changing the plumbing is more difficult okay so what about advanced cognitive skills like reasoning and the academic capabilities that it supports these skills depend heavily on a core set of brain regions here the frontal lobes on the front of the brain that caroline talked about as well as the parietal lobes it's sort of a loyal henchmen interacting a lot with the frontal lobes and this is called the fronto parietal network okay and it's basically a swiss army knife it is helpful for any kind of cognitive task that you throw at it and so different parts slightly different parts overlapping parts of it have been associated with these different mid-level cognitive skills that i've talked about like working memory or relational thinking so when does this brain network mature and just how malleable is it to address this this question i want to show you some behavioral and brain imaging data on a task called the matrix reasoning task here so this is a task that's commonly found on iq tests and what you need to be able to do is to solve this puzzle by identifying the missing piece and to do that of course you have to look at the relations among the items in both of these directions dimensions here and it turns out that the correct answer is this one here but as you can see there are a lot of distractors and it's easiest to solve if you just kind of look for the correct answer so this involves relational thinking as well as the other skills i mentioned okay so this task is designed purposely so as not to rely as much on background knowledge so it's supposed to be more culturally fair and yet it's one of the strongest cognitive predictors that we have of future scholastic achievement and performance in challenging careers and so for example as i mentioned you know the ability to integrate relations between numbers or variables or concepts is essential for math so just to take math as an example if you think about fractions comparing fractions for example we think about graphs as caroline was mentioning relating the x and y axes or you think about equations or sets of equations in algebra that all involves relational thinking okay so now i'm going to show you uh performance over ages 6 to 21 on this task and so these are the reasoning scores up here as a function of age from 6 to 21 and what you can notice several things right one is this beautiful increase in performance on this task and there's an inflection point around age 12 um and then it starts to plateau but there's also massive variability i mean you can find a seven-year-old who's you know performing better than uh an 18 year old for example just massive variability that we want to be able to understand i should mention i should back up and say that each point is one child and then each line is that child tested at two time points often around a year and a half apart these are data that were collected by my lab and several others it's over 500 participants and if you imagine the brain scanning and all the cognitive testing that we do takes like you know six hours per participant so it's a huge amount of work to get here um but i digress okay um so uh let's see if there was something else i wanted to say all right um oh the other thing i wanted to say is we talked yesterday about motivation to perform a task right that's going to differ on some days somebody might be more motivated than another day or something along those lines sure that's that's definitely going to contribute here nonetheless we see these clear trends and nonetheless it is a good predictor of academic outcomes okay so some i'm so sorry some researchers have emphasized the high heritability of reasoning skills right that genes can explain some of this variants here and indeed if you take a bunch of children who are from homogeneous higher income backgrounds you see that what differentiates them the most is actually genetic but then if you look across a wide range of backgrounds you see that actually environment matters a ton right and so performance on this task does vary by socioeconomic status it varies by number of years that you have attended school um and there's a strong environmental in fact it's even increased over the decades actually as people's jobs have required more and more higher cognitive skills okay so in parallel with these findings we find that this fronto parietal network gets stronger throughout childhood and to a lesser extent in adolescence and moreover among six to 12 year olds where there's the the most dramatic increase in performance on this task we find that among the six to 12 year olds those who have stronger white matter connections actually they do better on this task not only that they also show increased growth in their trajectory of reasoning development over time and so it's you know laying down the scaffold literally seems to be supporting cognitive development okay so to conclude a brain system that supports higher level cognition is still under construction during early adolescence and is thought to be malleable and i'm going to talk quite a bit more about that part tomorrow and given what we know so far from neuroscience and from caroline's compelling evidence it's up to us really to ensure that students are properly challenged at this critical junction so tomorrow i'll touch on some of the challenges and future directions and i hope you'll come back to be part of that discussion so thank you so much to you all [Applause] thank you very much uh professor van gaay and we can move to questions uh now ah so response pardon me um professor hoxby yeah would you mind responding because i want to make sure people have a chance to ask questions but i want to thank both eric and sylvia for fantastic uh commentary and i agree with so much of it i'm having a hard time coming up with what i should say except thank you thank you thank you and thank you again let me just say something briefly about um eric's uh comments i think what he did was provide this really great overarching um framework for you to think about why is it that certain types of jobs are in decline and other types of jobs other the jobs that have the routine skills tend to be in decline and the jobs that have advanced cognitive skills tend to be tend to be rising in terms of labor demand and then the the something that i liked especially was thinking about not just the drivers what has happened automation which has changed manufacturing and changed the need for certain types of skills like for instance strength but also how the economy does adjust to shocks it's not as though we have the same economy that we had in 1910 i think that's a really important point but we're doing different types of adjustment now and we can see that our economy is trying to get us to adjust right that rising premium for people who go to college or university is it's the economy saying go go to college or university learn these skills you need to learn the sort of skills that people pick up in college or university and we're just not adjusting as fast as the economy wants us to adjust which is one of the reasons why we have a rising wage premium for people in college and i should emphasize an interesting thing about colleges in the united states is that almost all of the growth in college education has been at colleges that are not very selective so they're not the berkeleys of this world or the stanfords or the princetons or um something else they are in fact increasingly online schools for-profit schools which are having a lot of growth and they're just not producing the same quality of skills so in some ways that skill premium that we see going up or that college premium that we see going up really understates the true premium for having those more advanced skills because a lot of colleges are just not producing them very well anymore so i think if anything it understates the the pressures and why is it that we have such frictions to skill acquisition for advanced cognitive skills but we didn't seem to have as many frictions to changing someone from an agricultural worker to a manufacturing worker i think in fact it is because these are very very different sorts of tasks right they just reward different types of things being strong being hardy being able to work maybe in the cold is very useful in both agriculture and manufacturing but it's not all that helpful if you're a lawyer sitting in an office downtown san francisco or something like that i also wanted to say um a few words about sylvia's comments which i find just incredibly helpful because she can think about these types of neuroscientific things in a way that's much more precise than i can and thinking about what it is that we do when we're when we're using our cognitive skills it is about holding information in your head it's about integrating information it's about this sort of relational thinking selecting the information that you're going to decide is important and i think that that all of these things are unusual in certain jobs but very usual in other jobs and that's um uh so that's that's important i also i'm glad to hear that our synapses are just going strong we can keep developing them over time so we don't stop all together but if you're not being put under pressure to develop new cognitive skills you're not going to develop them as much as if you're under new pressure to develop cognitive skills so i think that's a really important thing and i love the phrase pruning is sculpting your mind because that's really the way to think about it that's that's that's fantastic also experience dependent dependent brain plasticity i think is a very important concept that she talked about because it's this idea that your brain is plastic but if you don't really use it you're not going to have it be plastic in the ways that you want it to be plastic and that is why i think there are a lot of frictions to changing people from manufacturing non-college types of jobs routine types of jobs to jobs that require them to show these integrative and other types of skills because they're just not getting enough experience in the areas that they would need to have the right plasticity in those directions so um thank you very much both of you this was just fantastic um i learned a lot i hope everyone else learned a lot and i think we can take questions and do ask um questions of sylvia and eric too okay well thank you very much for these enlightening lectures yesterday and today so i'm wondering do you have feedback from teachers and policy makers on that and if so what kind of feedback do you get on all this um yes so i'm the sort of person who talks to policymakers quite a lot i talk to policymakers both at the federal level like the u.s department of education but also at state departments of education which in practice are almost more important because k-12 education and higher education more controlled by the states in the united states than they are controlled by the federal government that's a kind of surprising thing higher education is a little different than k-12 but um so yes what you need to do is get their attention and do it with simple studies that they can understand well enough to say maybe i should try this i think the fact that we do have 50 states is actually a great benefit because the state of california can can decide we're going to go all in for early childhood education and some other state can decide we need to put more emphasis on adolescents and then you have a little experiment going on to see you know which which policy works better i think the states are actually pretty good about experimenting if you can explain why it is why it is that your study makes sense i mean i think the reason why we have so much early childhood emphasis at this point is precisely that some researchers got to state policy makers and um city policymakers school superintendents and things like that and got them convinced that this was really important so they are changeable for sure yeah and i think also one of the things that i think is that they're often more impressed by things like neuroscience than they are impressed by us economists because you show better pictures i think is there's some research you show a better picture so i think i think that also may inspire more confidence in them that this is about science this is not just about economists opinions about the labor market they tend to think of us as opinionated but not necessarily as scientific as you and yet it's so hard for us to actually get the funding uh to to pull off these interventions they're really hard to do and really expensive and they're so small scale uh as i'll talk about more tomorrow so it's really frustrating but at the same time it's just so nice to have this complementary evidence from both of your fields that really are on a much bigger scale question so this is this is um poorly formulated but i i just wanted to ask caroline to say a little bit about something that uh sylvia mentioned which is heritability and just ask you about how you think of that as interacting and how you might address or need to address the question of heritability as we look at your map of appalachia say and we think of a fair bit of immobility there um we it seems like questions of heritability come to the fore and i just wonder if you've thought about it so heritability is a as you can imagine it's a somewhat controversial thing it's hard to often parse the evidence between what is explained by something i inherited abilities or um or what i've picked up along the way as an experience let's see certainly it is the case that you see that there are some areas of the united states where there's very little mobility either mobility in terms of geographic mobility people are not maybe should be moving out and they're not because jobs are decreasing there but also just a lack of economic mobility that families have been you know poor for a very long time and don't seem to be changing very much now that being said um i think that the role of heritability because i think you were talking about genetic heritability right genetic heritability i think is quite overrated for a couple of different reasons first of all when we look at interesting studies of adoptees who get assigned essentially randomly to different sets of parents in the united states sort of it's a first come first serve system on some adoption uh systems um you see the adoptees do not look that different from the from the children who are the biological children of the parents who adopt them so they tend to not score exactly the same way they tend to not do exactly the same thing in school but they're not that different and they're not genetically related to the people who adopted them so experience must account for quite a lot of things the other point that i wanted to make about this is and i think i mentioned this briefly in passing yesterday that economists like me when we're teaching undergraduates i understand where the undergraduates are coming from but the first time we show them a regression that tries to explain an adult's outcomes say an adult's earnings or an adult's educational attainment or things about whether they stay married or something like that family stability it turns out that our regressions no matter what we throw at the regression and that includes a lot of measures of aptitude measures of what your parents were like your parents education where do you live what's your race and ethnicity what's your sex at birth or your gender depending on the survey we never really get r squared above 15 so that means 85 of an adult's outcomes we can't explain even after we've thrown all of these variables at it now my undergraduates always dislike this fact right they find it very frustrating because they want to know how to fix the world and essentially what we're telling them when we say the regression has an r squared can explain 15 of the data is they feel very frustrated they're like well wait how how do i how do i fix everything i want to be able to fix everything and i say no no no it means it's a good thing because it means 85 of who you are is an individual and we can't explain you by just putting all of these variables into regression i don't really want to live in a world where everyone can be explained by knowing a few things about the household in which they grew up and knowing some things about their parents education and their test scores or something like that i think 85 percent of individuals being individual is is good so i guess that gets to heritability a little bit we have time for one more question i had a question about the evidence on teacher value added and i was curious i was unaware of this pattern by grade levels fascinating i was wondering if you would see a similar pattern if you looked at some outcome that we could think of as a proxy for non-cognitive skills and whether we would expect to see the same pattern or or would that be more concentrated for early childhood education yeah i don't i don't know the answer to that question because i haven't i haven't tried it i mean um conrad as conrad knows very well teachers value-added and non-cognitive skills is often quite different than their value-added and cognitive skills in fact some teachers are very good at teaching math or very good at teaching verbal skills but not good at both and there are teachers who are very good at keeping kids in school making sure kids don't have discipline problems getting them to go to college things like that and they are not necessarily the same set of teachers who are good at raising math scores and test scores um in other subjects so my my speculation would be that it would not look the pattern would not look that similar if we were talking about non-cognitive skills and the things that you would want to do first would probably be to look at disciplinary outcomes other things like that where you think teachers are very play a very important role right they're very important motivators but it's not that close to just being good at taking math tests right so and you also need of course um non-cognitive skills that you can actually measure so that's one of the that's one of the great difficulties with non-cognitive skills is a lot of them are quite difficult to measure but i expect that the pattern would look different sylvia do you have a sense of whether it would look it would look younger um i would i might think it might look older actually yeah a little bit later in adolescence when um when you're when your social skills are still continuing to develop so larry steinberg shows that social emotional development has an even more protracted developmental period than um abstract cognitive skills yeah one of my favorite tests um i was mentioning this yesterday is this test that they give on the nlsy which is a longitudinal survey that just has extremely easy questions on it and they're really boring and the only thing it's picking up is what's called coding speed because some people get very discouraged when taking the test and they just get bored and so they stop doing the coding and so they don't it's not actually a hard test at all but it's a very very good predictor of a lot of non-cognitive outcomes later in life because it's sort of picking up that someone is is very motivated right so we need more tests like that that are supposed to be picking up non-cognitive things and not not just um cognitive tests we have a lot of cognitive tests and too few non-cognitive tests i think well on that note uh please join me in thanking all three of our [Applause] you