Vijay's Reading Log

Currently Reading/Rereading

• Hands-on Machine Learning Scikit-Learn, Keras & Tensorflow: I won’t be taking Intro to Machine Learning at my college until Fall 2024 at the earliest, so in the meantime I am self-studying the concepts with this book.
• Into Thin Air by Jon Krakauer: An account of the 1996 Mount Everest disaster and how over the years, climbing Everest gradually turned into a business.
• Essays by Michel de Montaigne: Montaigne (1533-1592) invented the essay, but not as we know it today. Modern essays are usually written as part of a school assignment and focus on just one topic. Montaigne’s essays, meanwhile, jump from topic to topic and are not dry. There is an emphasis on witty writing and funny anecdotes.

Past Reads

• What is ChatGPT Doing… and Why Does It Work? by Stephen Wolfram: A great primer for topics like NLP, LLMs, and ChatGPT. The author, Stephen Wolfram, has put the whole book online on his website. One interesting takeaway is, a lot of stuff about LLMs was discovered through tinkering and experimentation. For example, when writing about why a temperature value of 0.8 works best for language models, he admits, “There’s no “theory” being used here; it’s just a matter of what’s been found to work in practice.” Likewise for why attention blocks “split up” the embedding vector – “we don’t know any particular reason why [this is] a good idea.”

  There are plenty of other examples throughout history of inventions being discovered before people fully understood why they worked. For example, the Wright Brothers invented simple airplanes before flight mechanics was a formal, well-understood field like it it is today.¹ Many researchers today are tinkering with LLMs in a way not unlike how the Wright Brothers tinkered with airplanes. That’s why we have things like prompt engineering and tweaking (hyper-)parameters to see how the model’s performance changes.

• The Beginning of Infinity by David Deutsch: I went through the book about a year ago, but looking back, I’ve barely scratched the surface. My favorite chapter is #17, “Unsustainable.” If you like discrete math or or Turing machine / complexity theory stuff, you will definitely enjoy this book.
• Ender’s Game by Orson Scott Card: Passes itself off as a kids’ novel, but has a lot of philosophical depth. Kept my interest from start to finish with the deeper philosophical questions. Is it ethical to preemptively strike against aliens?
• Ender’s Shadow by Orson Scott Card: This is a great example of a successful sci-fi sequel. It doesn’t outdo the original book, it complements it. Orson Scott Card sheds new light onto the characters’ motivations and fleshes out subtle details that went unexplained in Ender’s Game.
• The Three-Body Problem by Cixin Liu: The premise is that there are a finite amount of resources in the universe, so it is only logical that alien civilizations will pre-emptively attack each other. This is known as the “Dark Forest Theory”, and it is wrong. Aliens would visit for knowledge, not resources.
• Antifragile by Nassim Taleb: Very difficult to summarize. If you ask 10 different people to summarize Antifragile, you will get 10 completely different summaries. This is always a good sign, because it proves that a book is multidimensional.
• The Bed of Procrustes by Nassim Taleb: A very short read. You could finish it in one afternoon, because it consists entirely of one or two-sentence aphorisms. You could also read them one-at-a-time, finishing the book in a few months.
• The Black Swan by Nassim Taleb: Philosophy, epistemology, funny anecdotes from the author’s career as a Wall Street trader: Taleb’s writing style is highly idiosyncratic, and you will either love it or hate it.
• Only the Paranoid Survive by Andrew Grove: Written by a former CEO of Intel, the book talks about “critical inflection points,” times when a business encounters crisis and must radically pivot its product or go bust. Critical inflection points aren’t bad. If handled properly, they can take a business to new heights. The idea of a “critical inflection point” reminds me a lot of The Black Swan.
• Guns, Germs and Steel by Jared Diamond: I was really convinced by the author’s thesis until I read David Deutsch’s The Beginning of Infinity. Diamond’s explanations are highly reductive. They suggest that human decisions play hardly any role in history and that every important event can be explained away with simple geography.
• Striking Thoughts by Bruce Lee: Bruce Lee was a great writer and philosopher, and I’m surprised this book isn’t more popular.
• The Almanack of Naval Ravikant by Eric Jorgenson: Unusual philosophical depth for a business book.
• Schott’s Original Miscellany: Found this in a random bookshop in London. It reads like a Commonplace book from a few decades ago.
• Surviving the College Admissions Madness by Kevin Robert Martin: The book disguises itself as a college admissions advice book, but it’s really a set of memoirs + investigative journalism. A thoughtful, convincing book on why College Applications are not just a waste of time, but also hurtful to society.
• The Discovery of France by Graham Robb: A phenomenal history book. As late as the 1800s, most of France’s population didn’t know French. What we think of as “French” today was actually the dialect of Paris, which was only exported to the rest of the country in recent times.
• Homage to Catalonia by George Orwell: Orwell wrote a lot of stuff besides 1984 and Animal Farm. Did you know that he was fluent in Burmese?
• The Story of English by Robert McCrum: Books about the history of the English language are really fun to read.
• The Great Seige by Ernle Bradford: This book is a very thoroughly-researched history of the Ottoman Empire’s seige of the island of Malta in 1565, complete with detailed breakdowns of the military strategies employed by both sides. Good book to bring on a plane or long train ride. Bradford mixes expert historical research with very entertaining storytelling.
• The Wisdom of Adam Smith: A neat condensed form of Adam Smith’s economics and philosophy writings.
• Cultural Amnesia by Clive James
• Don’t Sleep, There are Snakes by Daniel Everett
• Ficciones by Jorge Luis Borges
• The Story of Your Life and Others by Ted Chiang
• Good Calories, Bad Calories by Gary Taubes
• On Writing by Stephen King
• How to Read a Book by Mortimer Adler
• Fashionable Nonsense by Alan Sokal
• Sapiens by Yuval Noah Harari: No.

Non-books, dense blog posts, websites

• Nassim Taleb’s Philosophical Notebook: Stumbled across this a long time ago, and it’s a ton of fun to read. There are ~90 pgs about everything from human nutrition to epistemology. My favorite entries: #66, “Real Books vs Digital Words” and #71, “Low Carb Philology”
• Reflections on Hyderabad - 2022: If you have ever traveled to India, you might find this interesting. See the section “English English English.”
• Japanese Level Up (JALUP) My main takeaway from this website is that language learning is way, way easier now than it was just a few decades ago. If you wanted to learn Japanese in the 1980s, your options were limited. Maybe you’d find some old, unsubtitled VHS movies or a battered dictionary at a yard sale, but without tech like Zoom, Anki (a flashcard app), or YouTube, your options were severely limited. It’s no wonder that that generation considered learning a language like Japanese a near-impossible task.
• Text is the Universal Interface: Reflections on ChatGPT, the Unix philosophy, and the history of software development.

Cool Research Papers

• Optimizing the cloud? Don’t train models. Build oracles!: Optimizing cloud computing configurations is a very complex task, and it involves a lot of moving parts. Since even a small mistake can mean thousands of extra dollars spent, there is a lot of research into mathematical methods for optimizing these configurations. This research paper argues that “cloud oracles”, which leverage computational geometry instead of machine learning, are a viable solution to this problem. Knowing when to, and not to, apply machine learning to a problem is something I want to better understand as a computer science student.
• RAFT: Adapting Language Model to Domain Specific RAG
• Towards Multiverse Databases
• MemGPT: Towards LLMs as Operating Systems: A scheme for extending LLM context windows to theoretically infinite length using a paging system analogous to virtual memory in OSes. This paging mechanism could also be used to help an LLM retreive and use outside information (e.g. PDFs)
• Neural Unsupervised Reconstruction of Protolanguage Word Forms: I’m interested in cognate generation / reconstruction because I think it could be useful for developing comprehensible input tools for foreign language teaching.

The design of my reading log was inspired by this page.