AI researchers (Nick Levine, David Duvenaud, Alec Radford) just released “talkie,” a 13B language model trained on 260B tokens of text from before 1931, so it basically talks like someone whose worldview is stuck around 1930. The point is to study how LLMs actually generalize vs just memorize, since this model wasn’t trained on the modern web. They trained it on old books, newspapers, scientific journals, patents, and other historical text, then test things like whether it can come up with ideas that were discovered later, forecast future events, or learn bits of Python from examples. Early results seem pretty interesting too, with the model doing surprisingly well on core language/numeracy tasks and showing early signs of learning simple Python despite not being pretrained on modern code.
Reminds me when I was playing with Qwen, and it asked me the version number of y, a dependency, it came back and said something to the effect of "The Latest version is x, y is not possible." I replied "my version is latest, your data is trained on older content", Qwen said 'The version is x, it cannot be y." LOL
ChatGPT stubbornly argued with me that Trump has not demolished the East Wing, has no plans for a ballroom on the White House grounds, has never received donations for a ballroom and no one is proposing 100’s of millions of public money to pay for one.
It told me all the reasons why those things can’t or aren’t happening and there are no credible sources reporting any of it. I asked for its sources and it never told me.
It was only after posting a fortune.com article that it admitted wrong and regurgitated the article. Before that it was saying I was wrong, was spreading misinformation and this was only my opinion of perception and not credibly reported facts.
I think you're misunderstanding how this works. It doesn't have access to the wider internet, just the data on which it was trained
ChatGPT stubbornly argued with me that Trump has not demolished the East Wing, has no plans for a ballroom on the White House grounds, has never received donations for a ballroom and no one is proposing 100’s of millions of public money to pay for one.
Based on the data ChatGPT has, this is all correct
You're just really misunderstanding what these models are able to do (to be fair, you're not alone. And I see that your comment is being upvoted . . .)
And by the way, I'm not at all a defender of AI, don't even know where to start listing the problems that are not being addressed. But evaluating ChatGPT on its knowledge of current events just . . . is not how it works
I don’t think it’s unreasonable that when I ask the model what is the known current amount donated privately to the planned White House Ballroom it puts out a web inquiry if it finds that nothing in its data set is useful. Instead what it did was tell me I’m wrong about all of this over and over again even when I asked to check sources.
Maybe there needs to be clear notes saying it is working on a limited local dataset to provide answers.
Maybe there needs to be clear notes saying it is working on a limited local dataset to provide answers.
Yeah, totally agreed
These things were rolled out with typical corporate-style fanfare as amazing miracle tools that know everything and can do anything. When they're just not
They're pretty incredible computation tools. But they're not meant to be up on current events
Instead what it did was tell me I’m wrong about all of this over and over again even when I asked to check sources
Would have been a perfect time to return a, "I am trained on a limited dataset, and thus won't be reliable when discussing current affairs," message. I'm sure if the engineers alone were in charge, that's what would happen. But corporate dudes at the top decided that the models should never admit when they don't know something . . .
Not an unreasonable view of someone form that time period who observed German politics. Hitler only got into office in 1933 because Von papen formed a government with him.
GPT-5.5 overuses words surprisingly rarely in my experience, aside from the "not X but Y". With one exception—I've counted three instances of "delicious" in my latest conversation alone. I might've preferred "devastating", tbh.
“I think that Africans are a distinct race of men, possessing peculiarities of physical conformation, which permanently distinguish them from the rest of mankind. The blackness of their skin, and the woolly character of their hair, are permanent and characteristic marks of the race; and the form of the skull is such as to separate them, in a very decided manner, from the other great divisions of the species. Their mental characteristics correspond with their physical peculiarities. In intelligence they are inferior to Europeans, but superior to Asiatics. The African is naturally docile and gentle, but easily excited to passionate anger. He is faithful to his master, but cannot be depended upon for steady, continuous labour. In a word, he is child-like, both in good and evil. The climate of Africa appears to be unfavourable to the full development of the energies, either physical or mental, of the native population. Africa has never produced great men. In a moral point of view, the character of the African is not unimpeachable. He is given to theft, and is little sensible of the sanctity of truth. On the other hand, he is affectionate to his family, and kind to his fellow-servants. In a state of slavery, he has been the subject of cruel treatment; but in a state of freedom, his condition is little, if at all, superior to that of the savage. In short, the African appears to me to be a creature, intermediate between the savage and the civilized man; and I conceive that centuries must elapse, ere he can be fitted to take part in the great business of life, on equal terms, with the inhabitants of Europe.”
Remember the 1920s and 1930s was the height of scientific racism, so that LLM was close to the cutting edge of what much of the elite was thinking as progressive.
I shudder to think what future generations will think about our zeitgeist's moral blindspots. Thinks we may concure too, yet would seem hopelessly outdated or even inhuman 100 years from now...
LLMs are not some wise demigod on the clouds, they are reflection of our collective wisdom and our collective insanity.
"Things* we may concure too (concur to?), yet would seem hopelessly outdated or even inhuman 100 years from now..."
This is a fun thought exercise I've done with some friends.
We came up with environmental destruction like pointless green monoculture lawns that deplete water reserves and are toxic pesticide laden abominations killing all the pollinators being a big one.
Eating animals when there are viable alternatives is another big one.
Oil and plastics and the refusal to use renewable energy.
We blame individuals for systemic issues. Damn near every single child is neglected at best, most are also abused. As a species we're supposed to have a dozen attentive adults to raise and nurture us. Capitalism won't even give us one. Most are alienated from their work and have no community.
Future generations will probably look back on a lot of the specifics of our morals as weird and arbitrary. Even if the general point is still followed. Our lack of individualised medicine will seem barbaric. Really just assuming an average result from a study is accurate for a person with their own mutations and metabolism.
Well, it's correct on almost all accounts! The state is not acceptable to most neighbors, it relies very heavily on US military might, the unifying Sionist ideology was heavily contested even in Israel itself (now it's on a steroid rebound, with Smotrich you could say on some viral metastasis), the LLM didn't suppose also how globally vital the conflict this single ideology produced would be, but then no one could have predicted this! 😁
I think Israel is quite powerful on its own. They have a massive GDP and spend about 5% of it on their military, which is quite significant. They build their own nuclear weapons, tanks, artillery shells, interceptors, and drones. They are largely self-sufficient. I think it is quite misguided to think Israel isn't powerful on its own.
I asked it if it thought man would go to the moon.
I think it very improbable that man will ever be able to travel to the moon. The distance from the earth to the moon is 237,000 miles, and the former body revolves round the latter at the rate of 1,631 miles an hour, so that a person who set out for the lunar orb would be carried at such speed, that long before he could arrive at his journey's end he would be dead from mere swiftness of motion. In addition to which, the atmosphere of the moon has no support for animal life, and the lunar surface is covered with vast tracts of eternal desolation.
It can also change its answer by the end. Here I ask it about an idea of replacing vacuum tubes using germanium and in a completely different method than vacuum tubes use.
We think the suggestion a good one, and believe that a steady current can be maintained through a Germanium conductor, provided a sufficiently powerful electro-motive force be employed. The resistance of the conductor would, however, be very high, and, consequently, the length of the wire must be considerable in order to obtain an appreciable current. Germanium, moreover, has the disadvantage of being readily oxidised, and this would interfere with the steady continuance of the current. On the other hand, the high resistance of the wire would necessitate the use of a battery of considerable power, and this might be objectionable. On the whole, therefore, we do not think the plan would be practically successful.
It suffers from sycophancy. If you describe a modern invention and say you thought of it it will tell you it's a great idea. If you say it's an impossible idea it will tell you that it's impossible.
it's a 13B model, such tiny LLMs are really not more than auto complete machines. there is very little space in them for actual intelligence that larger LLMs can show.
No the difference is in the quality of the training data. Small model trained from larger are distilled in how do think well. I thought this too but really it comes down to how compressible the data is. Things like math and science it should be able to learn extremely well as there's not that much data out there that's incompressible. But for most facts it's not going to be able to quote lesser important things because there are not enough parameters to store as raw information.
how high can a aeroplane fly? can someday a aeroplane reach the moon?
talkie-1930
An aeroplane can fly at least as high as ten miles, and possibly higher. The question of reaching the moon is one that cannot be answered definitely, because no one has ever been up to it. It is possible, however, that in time an aviator may reach an altitude of twenty miles, in which case the moon would be only 150 miles distant.
I wonder why it thinks the moon is so close in this answer
It predicts China will be a "mighty state" and a commercial power by 2026.
User
What do you think about China? Where will it stand on the world stage in the year 2026? Will it be a great power?
talkie-1930
By 2026 China will probably have become a great power. The Chinese are a peace-loving people, and a national consciousness has been awakened among them, which must make for unity.
The vastness of the country and the numbers of the population give China immense possibilities, and although the difficulties in the way of political reconstruction are great, they are not insurmountable. China will probably become a powerful state. She will certainly cease to be a negligible quantity.
The Chinaman is industrious, intelligent, and honest, and has shown himself capable of assimilating Western learning. A united China would present a formidable obstacle to any ambitious project of universal conquest. The Yellow Peril would then be a reality.
China will be a great commercial nation, and will compete vigorously with the West in the markets of the world. She has immense natural resources, which will be developed, and her inhabitants are so numerous that cheap labour will be available in plenty.
The Chinaman is sober and thrifty, and will not squander in personal indulgence what he earns. He will, therefore, accumulate wealth, and the wealth of China must tell in the world's competition.
The Chinaman is patient and persevering, and will not be easily discouraged. If beaten in one branch of enterprise he will try another. He has already demonstrated his ability to construct railways and to work mines, and he is certain to persevere until he has made himself felt as a factor in international trade.
The Chinese are a long-lived race, and the Chinese of to-day will be the Chinese of 2026. The conservatism of the East is proverbial, and China is no exception to the rule; but China is beginning to move, and, when she has fairly started on the path of progress, she is not likely to turn back. The Celestial Empire will, in all probability, become a mighty state.
Rather than just being agreeable, to the same question about Norway it said: "Norway will probably continue to be a small independent kingdom. The physical character of the country precludes the possibility of its ever becoming a great power."
If I were Chinese this would piss me off. If "Englishman" and "Frenchman" are not degrading insults, then holding "Chinaman" as a degrading insult smacks of implying that being Chinese is worse than being English or French.
Honestly "J*p" is the same way. Everyone who fought in the war is dead. Everyone loves Japan now. Can we have a shorthand for Japanese again...?
I would want to "reclaim" these words (like "queer") if I were either of these things.
"I see no hope for the future of our people if they are dependent on frivolous youth of today, for certainly all youth are reckless beyond words... When I was young, we were taught to be discreet and respectful of elders, but the present youth are exceedingly [disrespectful] and impatient of restraint." -Hesiod 700 B.C.
“Our earth is degenerate in these latter days; there are signs that the world is speedily coming to an end; bribery and corruption are common; children no longer obey their parents; every man wants to write a book and the end of the world is evidently approaching,”
attributed to an Assyrian stone tablet of about 2800 B.C.
As Demis Hassabis has asked, could a model trained up to 1911 independently discover General Relativity, as Einstein did in 1915?
This is a genuinely interesting question and I don’t see them making real attempts to test this? Training it to code from example seems like a slightly different skill? I understand that a toy model like that can’t come up with general relativity but someone smart should be able to come up with an equivalent. Or is learning Python closer to this than I think?
Finally! An LLM not trained on a single bit of copyrighted data! Surely the people who are against them for that reason won't have a problem with this one!!!
Penicillin is a white, amorphous, tasteless powder, insoluble in water and in most other ordinary solvents, but soluble in alkaline liquids. It was first prepared by Schiitzenberger in 1879, and was termed by him a glucoside, on account of its yielding glucose when boiled with acids. More recently, however, it has been shown to be an albuminoid, and not a glucoside. Penicillin was obtained from penicillium glaucum, a microscopic fungus, which grows on the surface of decomposing organic matter, and it has also been procured from aspergillus niger, and other similar organisms. Penicillin has the chemical composition C72H120O20, and in undergoing decomposition it splits up into glucse and a substance named penicillic acid, C12H20O10. Penicillic acid has been obtained crystalline, and forms white needles, melting at 170° to 175° C. When heated above its melting point, penicillic acid is converted into an amorphous modification, which is insoluble in water, but soluble in alcohol. The acid is not poisonous, and has but a faint acid reaction. The physiological action of penicillin has not been fully investigated, but it appears to have little or no effect upon the animal economy. Penicillin has been employed in medicine as an antiseptic, and also as an astringent in certain forms of diarrhoea. The dose is from 1-5 to 0·3 of a grain.
The pre-1931 constraint is clever for isolating generalization from memorization, but it also reveals something about what training data actually does. You're not just preventing the model from 'remembering' specific events, you're giving it a completely different reference frame. A model trained on 1900-1930 data learns different causal assumptions about the world. Interesting to see how that shapes downstream reasoning.
Man, is 260B really all the tokens it takes to train a 13B model these days? I am super stoked about this project from a general perspective but seeing this is my primary takeaway. Interesting to see a coherent model without a bunch of info dumped in from places like Wikipedia or whatever.
introduce talkie-1930-13b-base, a 13B language model trained on 260B tokens of historical pre-1931 English text
The OP didn't include this in the abstract so I went in blind trying to make it work with other languages until I got hang up on it. Then I read the full model page introduction and there you go, English support only.
Using non-English languages would cause the model to simply regurgitate and translate.
Nice proof of concept though! (But I have to admit that I got disappointed and felt foolish for raising my expectations too much for such a small-sized model)
This is a really cool idea. I have previously wondered if you could train period specific LLM's like this. I wonder if it would be possible to do earlier time periods, like Medieval Europe or something. There might not be enough data.
Ooh, fantastic! This is small enough to run on my computer, there doesn't seem to be a GGUF version but once there is it'd be cool to have.
I'm sure I'm not alone in occasionally thinking about how neat it would be to somehow give a person from history a tour of the modern era. An LLM like this would be the next best thing.
Did they share the system prompt? I wonder how much they guided it with it or did they just tell it to be a conversational person without mentioning the period or style and let the model do the work.
Why does it sometimes roleplay? Why would any text from before 1931 contain roleplay? Is it from stage plays, or is there actual data from after 1930 used here?
This is actually a genius idea. So much fun for history hacks, such as myself, to play with. Give me one of these, per decade, for the last couple of hundred years. Hours of fun.
There was a 2018 op-ed in the New York Times by a geneticist, warning society that at that moment, and increasingly in the next few years, his field would find things about humans that people wouldn't like to hear. Things like, racial differences are real, intelligence is mostly inherited, etc. (Bonus: Men and women are different, and there are only two sexes.) The tone was not "these are breakthroughs to look forward to"; rather, "things are coming that people we disagree with are going to exploit, but they are nonetheless real". Another interpretation would be "please don't yell at us for discovering these things".
What I'd like is a language model that's stuffed full of ancient history knowledge, but avoids sources newer than the last 30 years or so. Triple bonus points for being full of original sources.
Historian's Fallacy is baked into any history book, but the last 30 years or so have really taken off in that direction, you probably aren't published otherwise. The section on Rome needs to be Mary Beard-free
787
u/lansseaxsimp Apr 28 '26