“Stable Genius” – Let’s Go to the Data

So, as always. First the headline, then you need to eat your vegetables to get the details.

The headline:

By any metric to measure vocabulary, using more than a half dozen tests with different methodologies, Donald Trump has the most basic, most simplistically constructed, least diverse vocabulary of any President in the last 90 years. This is by a statistically significant margin in each case.

Okay, the headline’s out of the way. On to the vegetables, so you understand why we checked this, and the methodology.

(And with our apologies for the simplistic charts. The Google Sheets plug in is quick and dirty… but the data’s all there for you at the bottom)

Why Are You Blogging on a Sunday Night?

Well, the Golden Globes are on. Also…

I usually try to unplug over the weekend. And by unplug, I mean “catch up on everything I was supposed to do during the week but didn’t because who the hell can get work done during office hours.” You know, by relaxing and stuff.

So the emails that started coming in Saturday morning around 8 a.m. kind of interfered with that plan. I ignored them for all of 20 seconds before seeing what the heck was going on. In general, when something is going on, the emails tend to clump together. The phone wasn’t going to stop vibrating by force of will alone.

Turned out, it was a number of folks asking if I’d seen the “genius” tweet, and if Factba.se had ever run an intelligence test.

Now, when someone emails me at an ungodly hour (and prior to 11 am on a weekend more than qualifies, given my normal bedtime is defined as “Thursday”) to ask about a tweet, I put the darker thoughts out of my mind and did my best not to get upset.

But I was awake. May as well spoil it. The tweet in question (a three-parter, which is more unusual of late since the character limit was upped):

…spanning 11 minutes. (Sorry about that last one… one of my favorite Road Runners).

The quote that seemed to stick out in everyone’s mind was the last one: “I think that would qualify as not smart, but genius….and a very stable genius at that!”

Okay, I was awake.

Apparently, the intellectual exercise would be to parse the phrase “genius” and could it be proven, or disproven.

Into the Den of Snopes

Measuring intelligence is normally done through a simple method with no agreed upon standard: an IQ test, a loosely-defined standardized test, variations of which have been in use for more than a century. The most common one in modern use is the the Wechsler Adult Intelligence Scale (WAIS) v4, in use since 2008.

However, there is no peer-reviewed method to look at writing / speeches / etc to assess intelligence. The closest is a 2006 study, which used a historiometric method.

Suffice it to say, that method is fine, but it takes a doctorate and an expert. We don’t have presidential scholars at Factba.se. We’re a bunch of data schmoos. Also, this particular study was ripped off and faked enough in the past 15 years that it has multiple snopes pages (here, here, and here) and it rates its own Wikipedia page. Again, the study is fine. Making stuff up around it isn’t.

Supercalifragilisticvocabularydocious

However, the ability to measure the complexity of vocabulary, the diversity and its comprehension level is something we do all the time here in the Fact Cave, courtesy of Margaret, our platform’s AI. In fact, it’s done every time we add a word into the platform, automagically. The most common metric, the Flesch-Kincaid Grade Level, was actually developed for the military in the 1970s as a way to check that training materials were appropriate and could be understood by its personnel. It is used as a measurement in legislation to ensure documents such as insurance policies can be understood.

There are a number of competing algorithms. They use different approaches, but all try to do one of two things:

  • Grade Level. Establish the grade level at which the text could be understood
  • Reading Ease. Essentially the same thing, but with a normalized statistical score vs. a U.S.-centric grade level.

At Factba.se, Margaret runs every single bit of text automatically through the following algorithms:

… and about a dozen others, including difficult word count, etc. We’re also testing the Lexile Framework.

As a side benefit, recreationally, we built a database of interviews, speeches and press conferences for previous presidents, leaning heavily on what’s available publicly from presidential libraries, and the wonderful collections at the University of California, Santa Barbara’s American Presidency Project. One of the reasons we did this is to provide a point of contrast. Looking at a single datapoint can tell you everything and nothing. A nice cohort comparison… that’s better.

Importantly, as we’ve blogged earlier, we like to focus on a person’s own words if possible, not speechwriters. The UCSB archive in particular gave us a rich trove of Presidential press conferences back to Herbert Hoover in 1929. So we could look at just what a president said. Unscripted (or as close an approximation as is possible for a president).

Okay. We had the algorithms. We had the text. On to…

Methodology

As mentioned previously, we narrowed our samples from Hoover forward to just press conferences, presidential debates and interviews. Of course, within those, we only use words spoken by the President, nothing else.

This left us with a deep sample for each, but spread out. We ran the analysis two ways:

  • Complete. Whatever we have, we have. On the low end, it’s 44,705 words for Gerald Ford, up to 1,124,164 words for Bill Clinton. Trump clocked in second at 915,801 words.
  • Equal Sample. We then ran the same test on 30,000 words, plus or minus 1% (actual range was 30,003 – 30,253 words), where we looked only within the person’s presidency (no pre-election debates) and started from Inauguration Day forward, adding sentences until we hit 30,000, then stopped and analyzed those.

In addition, we’ve been testing the Lexile framework. It’s a free test so we’re limited to 1,000 words. But we took the first 1,000 words (in full sentence format) from the equal sample and tested those.

It’s important to note: for the two presidents where social media existed, this was not included. This was strictly utilizing the responses given by a president in an interview, during a press conference, or in a political debate.

The Result

It statistically made no difference which way we analyzed it, or which method. It affected some scores and some of the ranks, but not the position of Donald Trump on that list. In each case, he ranked last of the past 15 presidents.

By every metric and methodology tested, Donald Trump’s vocabulary and grammatical structure is significantly more simple, and less diverse, than any President since Herbert Hoover, when measuring “off-script” words, that is, words far less likely to have been written in advance for the speaker.

Significant is not editorializing. The gap between Trump and the next closest president (in most indices, Harry Truman, known historically for a folksy, simple pattern of speech), is larger than any other gap using Flesch-Kincaid. Statistically speaking, there is a significant gap.

This gap appears both when using the complete corpus available to us for all presidents, and the more limited 30,000 word set to use an equal data set for each. In either data set, Donald Trump consistently clocks in at the bottom of the list. Depending on the scale used, it’s between a 3rd and 7th grade reading level.

Using the same one used by the Department of Defense, the grade level on the equal sample is 4.6. That’s between a fourth and fifth grade level.

The next closest is Truman at 5.9, followed by Bush 41 at 6.7. The top three: Herbert Hoover (11.3), Jimmy Carter (10.7) and Barack Obama (9.7).

In terms of word diversity and structure, Trump averages 1.33 syllables per word, which all others average 1.42 – 1.57 words. In terms of variety of vocabulary, in the 30,000-word sample, Trump was at the bottom, with 2,605 unique words in that sample while all others averaged 3,068 – 3,869. The exception: Bill Clinton, who clocked in at 2,752 words in our unique sample.

So What?

That’s a fair question. So what? Vocabulary is not a proxy for intelligence. In IQ Tests, vocabulary is a component, but only a component.  However, it is used as a proxy for a number of things:

  • Doctors use it to measure symptoms of degenerative brain diseases (note: as blogged previously, we see no downward trend over 40 years in Trump’s vocabulary. For unscripted, it’s very consistent).
  • Psychologists use vocabulary as a measure of intellectual curiosity and a person’s reading ability.

But also, it should be pointed out:

  • Politicians strive to get a clear, concise message in front of the public. That includes keeping it short and simple.

Other than Donald Trump, all presidents in this cohort were either career politicians, or in the case of Eisenhower, a very public figure and military leader for decades before running for president (historians argue whether a general at Eisenhower’s level would already be considered a politician before running for office, due to the need to navigate very political waters at that level).

Back to so what? In answer to those who emailed the equivalent of “is the president a stable genius”, the answer is “we don’t know.” Short of IQ tests, there’s no way to know for sure.

But what we can say is, compared to the 14 presidents who preceded him, by every measure, his use of words when off script are significantly less diverse, and simpler, than all presidents who preceded him back to Herbert Hoover.

As always, feel free to dispute the analysis, but come prepared with data. We don’t need more opinions. But more analysis with supporting data is always welcome.

Here’s the data. Have fun!

[Note: Hmm… thought the plug in would download all the tabs, not just one. Oh well. This is the Google Sheets link

id
president
Flesch-Kincaid Grade Level
Flesch-Kincaid Reading Ease
Gunning Fog Index
Coleman-Liau Index
SMOG Index
Automated Readability Index
Dale Chall Readability Score
Average Words / Sentence
Average Syllables / Word
Word Count
Unique Words
Lexile Score (Low)
Lexile Score (High)
Lexile Word Count
Chart Label
45Donald Trump4.6082.507.407.705.803.202.8011.591.3330,2522,605600700974Trump
33Harry S. Truman5.9072.707.809.906.904.804.4011.251.4530,0093,246800900896Truman
41George H. W. Bush6.7071.909.109.607.305.904.0013.981.4330,1323,308700800925Bush 41
43George W. Bush7.4067.009.8010.207.906.404.4014.111.4831,0643,29710001100951Bush 43
32Franklin D. Roosevelt7.4070.7010.009.207.806.603.5016.151.4230,0953,40911001200908FDR
36Lyndon B. Johnson7.6067.7010.2010.008.206.904.2015.431.4630,0363,19113001400975LBJ
40Ronald Reagan8.0068.5010.709.708.107.604.1017.241.4330,0383,48511001200974Reagan
35John F. Kennedy8.8063.7011.2010.109.008.304.6017.941.4830,0033,28911001200953Kennedy
42Bill Clinton9.3064.6012.009.609.009.004.1020.231.4430,0402,75211001200916Clinton
37Richard Nixon9.4063.3012.109.809.209.004.5019.871.4630,0223,06812001300939Nixon
34Dwight D. Eisenhower9.4064.1012.209.609.009.204.2020.571.4430,0053,26911001200945Eisenhower
38Gerald Ford9.4060.8011.8010.309.408.704.7018.521.5030,0923,28713001400923Ford
44Barack Obama9.7058.1012.1011.209.509.305.0018.231.5430,0063,86912001300979Obama
39Jimmy Carter10.7054.6013.3011.4010.2010.405.2020.221.5630,0703,62411001200973Carter
31Herbert Hoover11.3051.9014.4011.4010.9011.004.9021.381.5730,1703,47112001300912Hoover

17 Replies to ““Stable Genius” – Let’s Go to the Data”

  1. I am not surprised by this analysis or results. Donald Trump only uses a few words when he speaks which are “Great, Fantastic and Sad.” I feel as though he thinks the average American is a bunch of elementary school kids. Also the corny name calling is also an example of the words that a fifth grade bully would use to describe a classmate. It is really comical but also as he would say—“SAD…SO SAD☹️☹️☹️

  2. Advertising executives who spend billions on electronic and print media, regardless of their own levels of education, are acutely aware of the necessity to target the 3rd to 4th grade level of Americans in order to maximize profits.

    Obviously, Donald Trump is perfectly matched to this standard, either wittingly or unwittingly, and where persuasion is concerned, generally can outperform a less savy competitor.
    These “facts” strongly suggest that many Americans with higher education do NOT really understand how Donald Trump can effectively sell to the very significant group of their fellow Americans known as “Consumers.”

    Huey Long, ( “A chicken in every pot”), the assasinated Louisiana Governor, who was a popular potential Presidential candidate, was said to be Adolph Hitler’s most feared adversary because of his ability to arouse a majority of “forgotten Americans” with his 3rd to 4th grade rhetoric.

    Warning: never over-estimate the cognitive or emotional intellect of American consumers when it comes to advertising.

    Therefore, by the legal standard of Res Ipsa Loquitor

  3. You have to remember the grade level of the American average that the Presidents talk to so they can comprehend what
    he is talking about! If he were talking to all College graduates or the average middle, high school educated crowd?
    What is his objective with his speech? Who does he want to get to and how many people does he want to inform?

    1. i guess his objective is to influence 4th graders.. fuck all those people with actual brains that work though. jesus..

  4. Maybe this test reflects the average grade level of those in this country. I worked for an industry, whereby 9th grade level was what I was required to edit/write tech manuals. We finally had to resort to comic gaphics. What does that say about our schools sytem?

    1. Disagree but opinions are like a@$holes. Everybody has one. You provide no facts to support “Not just my opinion.” Many people think the moon landing was fake, too. Glen

  5. How unusual, the liberal socialists of tolerance and compassion making up stories bashing President Trump. From this story it appears the more brash you are the richer you’ll get. Hmmmmmm then there is this: http://www.sciencealert.com/swearing-is-a-sign-of-more-intelligence-not-less-say-scientists

    AS for me, I’ll go with what the hateful left wing liberal socialists writers say. I mean they wouldn’t lie …………………………………………… would they???

    1. Note we included Jeb’s comment as an example of how *not* to reply…
      — The comment has nothing to do with the blog post (not about swearing)
      — Lots of language that’s intended to be incendiary “liberal socialists, etc”
      — No actual facts or details cited.
      — And just trolling in general.

      If you wish to post comments like the above, they’re not getting posted from here on out. Take that to twitter or somewhere else. Preferably the toilet for lowering the level of discourse.

      We don’t mind opinion, discussion, etc. We do mind troll comments . Civilized debate is a wonderful thing…

  6. Concerning the president’s fourth-grade vocabulary, Trump enthusiasts who don’t mind insulting others might say that he is only being smart by deliberately using it to appeal to his base. But can it be true so many of our past presidents had less than a tenth-grade vocabulary? I guess a tenth-grade vocabulary isn’t what it used to be. Or maybe they too were simply appealing to their bases — not a very flattering picture of the American electorate.

  7. Insightful & thoughtful. Not just what you wrote, Mr. Frischling, but also the comments. As an English teacher & librarian in public schools for 35 years, and for my most of my life, I have loved words and all they convey or suggest. Your words say a lot about our country. Those who chose and support our President like short, simple, vague unsubstantiated sentences, with a touch of vulgarity to punctuate a point – or degrade a person. Personally I prefer someone like you, who gets to the point with facts, sincerity, and decorum, not deflection, innuendo, and rage. That’s a stable genius for me. Thank you,

  8. Thank you for sharing excellent informations. Your web-site is very cool. I’m impressed by the details that you¡¦ve on this blog. It reveals how nicely you perceive this subject. Bookmarked this web page, will come back for extra articles. You, my pal, ROCK! I found just the information I already searched all over the place and simply couldn’t come across. What an ideal web site.

    1. Thanks! Just to confirm, though. We’re careful not to let our own opinions / perceptions in, as much as is humanly possible. We play the data where it lies. Any opinion / interpretation, we make sure to use algorithms where anyone else can double check them.

  9. I have seen Donald Trumps public appearances going back 30+ years. I believe that his use of simplified speech, not only is clear for all comprehension levels but serves to less likely be twisted by foes.

Leave a Reply

Your email address will not be published. Required fields are marked *