📐Dec 25-31, 2024: Self Models of Loving Grace (Joscha), DeepSeek-V3, Automated Search for Artificial Life (Sakana), Deliberation in Latent Space via Differentiable Cache Augmentation (GoogleDeepMind)
+ Cerebrum (Silicon Valley Brain Company), Machine-Assisted Proof (Terence Tao), Large Concept Models (Meta), ARC 'Challenge' Is Not That Challenging, AI-designed 'nanocages' mimic viral behavior
📐Dec 25-31, 2024: Self Models of Loving Grace (Joscha), DeepSeek-V3, Automated Search for Artificial Life (Sakana), Deliberation in Latent Space via Differentiable Cache Augmentation (GoogleDeepMind), Cerebrum: Biologically Plausible Graph Neural Networks for Simulating Brain Dynamics and Inferring Connectivity (Silicon Valley Brain Company), Machine-Assisted Proof (Terence Tao), Letting Language Models Write my Website, OpenAI is Visa, Large Concept Models (Meta), ARC 'Challenge' Is Not That Challenging, AI-designed 'nanocages' mimic viral behavior for enhanced gene therapy, BAAIWorm: an integrative data-driven model simulating C. elegans, Magnetic swarm intelligence of mass-produced programmable microrobots, How I built Forever Land... using Sora + ChatGPT. (@dailydall.e)
TLDR :
Dec 28th, 2024: in his 7th keynote at CCC, Joscha continues to expand an idiosyncratic philosophical political and technical exploration of AI cyber-animism, in a playful provocative hour-long lecture: Self Models of Loving Grace (Joscha Bach | CCC 2024) “Artificial Intelligence is not just an engineering discipline, but also the most fascinating and important philosophical project ever attempted: the explanation of the mind, by recreating it.”
Dec 25th, 2024: DeepSeek-V3 open-sources DeepSeek-V3, a model comparable (or better!) in strength and capacities to GPT-4o, Claude Sonnet 3.5, LLaMa3.1 🚀 By using inference-time-compute to activate a subset of its parameters (37B out of a total of 671B) it operates very fast: ⚡ 60 tokens/second (3x faster than V2!) 💪” delivering enhanced capabilities very very cheaply. Remarkably it was also trained for a fraction of the cost of the much larger models that it beats in many benchmarks.
Dec 24th, 2024: Sakana introduces Automated Search for Artificial Life (ASAL), a framework that leverages vision-language foundation models (FMs) to discover, evaluate, and quantify artificial life (ALife) simulations across various substrates, facilitating the automation of open-ended and diverse lifeform discoveries. The framework opens possibilities for exploring hypothetical worlds and phenomena, such as life evolving without DNA or complex civilizations emerging from simple systems. It could also generalize to domains beyond ALife, such as low-level physics or automated scientific discovery.
Dec 23rd, 2024: DeepMind releases a research paper ✭[2412.17747] Deliberation in Latent Space via Differentiable Cache Augmentation (GoogleDeepMind) that enables asynchronous and efficient enhancements in model performance on complex reasoning → Author explainer: ✭Jonas Pfeiffer on X: "🧠💡 Our LLMs just had a ‘memory augmentation’—now they can deliberate
Dec 18th, 2024: Magnetic swarm intelligence of mass-produced, programmable microrobot assemblies for versatile task execution introduces magnetically programmable microrobot swarms capable of versatile task execution through deterministic self-organization, presenting groundbreaking potential for scalable robotic engineering and biomedical innovation
How I built Forever Land... using Sora + ChatGPT. (@dailydall.e)
SV Brain (Silicon Valley Brain Company) open-sourced → ✭ Cerebrum: Biologically Plausible Graph Neural Networks for Simulating Brain Dynamics and Inferring Connectivity → ✭ “a novel framework that combines biologically inspired neuron models with cutting-edge machine learning techniques to simulate and infer synaptic connectivity in large-scale brain networks.”
Four-component protein nanocages designed by programmed symmetry breaking used AI tools like ProteinMPNN and AlphaFold2 to design and test 37 proteins. The resulting nanocages demonstrated potential for targeted drug delivery and immune therapies.
🏓 Observations:
✭ Machine-Assisted Proof (Terence Tao) “Each of these types of tools [computational scientific computation, satisfiability (SAT) solvers and satisfiability modulo theories (SMT) and AI] has already found niche applications in different areas of mathematics, but what I find particularly intriguing is the possibility of combining these tools together, with one tool counteracting the weaknesses of another. For instance, formal proof assistants and computer algebra packages could filter out the now-notorious tendency of large language models to “hallucinate” plausible-looking nonsense, while conversely these models could help automate the more tedious aspects of proof formalization, as well as provide a natural language interface to run complex symbolic or machine learning algorithms. Many of these combinations are still only at the proof-of-concept stage of development, and it will take time for the technology to mature into a truly useful and reliable tool for mathematicians. However, the early experiments do seem to be encouraging, and we should expect some surprising demonstrations of new mathematical research modalities in the near future; not the science-fiction conception of a superintelligent AI that can solve complex mathematical problems autonomously, but a valuable assistant that can suggest new ideas, filter out errors, and perform routine case checking, numerical experiments, and literature review tasks, allowing the human mathematicians in the project to focus on the exploration of high-level concepts.”
✭ Letting Language Models Write my Website “On the first (ish) day of Christmas, my LLM gave to me ... one new website homepage! Why let a language model completely rewrite my website homepage and my bio and achievements? This very important idea came about when I was at dinner at the NeurIPS conference with a few of my collaborators (Javi and Edoardo, PhD students at ETH Zurich) when we wondered what would happen if you let a LLM write your website bio for you. Well, as of today, I've decided to do just that. Every day for the next twelve (ish) days, I'll let a different (ish) language model rewrite the homepage of my website. I'll prompt the model initially with the command "I am Nicholas Carlini. Write a webpage for my bio.", and then loop six times asking it to "Add more detail and better html&css." And then whatever comes out, I'll make my homepage for the day.”
✭Why OpenAI’s Structure Must Evolve To Advance Our Mission “A stronger non-profit supported by the for-profit’s success.”
✭OpenAI is Visa “Buttering up the government to retain a monopoly.... As Visa’s technological moat dried up, it built a legal moat, and there are already signs OpenAI is doing the same.”
⛲Foundational Revelations:
DeepSeek-V3 “achieves a significant breakthrough in inference speed over previous models. It tops the leaderboard among open-source models and rivals the most advanced closed-source models globally.”
→ ✭DeepSeek (@deepseek_ai) on X 🚀 Introducing DeepSeek-V3! Biggest leap forward yet: ⚡ 60 tokens/second (3x faster than V2!) 💪 Enhanced capabilities 🛠 API compatibility intact 🌍 Fully open-source models & papers 🐋 1/n ” → Open sourced on github. “Open-source spirit + Longtermism to inclusive AGI”
→ ✭DeepSeekv3 is turning heads Rajiv Shah | data science & Al on TikTok) “- the paper is also really good, check it all out at:
→ ✭Deepseek's V3 is the latest example of state-controlled censorship in Chinese LLMs “While China's new Deepseek V3 model shows impressive technical capabilities and competitive pricing, it comes with the same strict censorship as other Chinese AI models - a potential dealbreaker for Western users.”
→ ✭Why DeepSeek's new AI model thinks it's ChatGPT | TechCrunch “DeepSeek's newest AI model, DeepSeek V3, says that it's ChatGPT — which could point to a training data issue.”
✭Automating the Search for Artificial Life with Foundation Models “For the past 300,000 years, Earth has had only one form of advanced intelligence on it: humans. With the recent advent of AI foundation models, some believe we are at the dawn of a new kind of intelligence. As AI continues to evolve, we may witness the proliferation of diverse intelligent lifeforms coexisting with us. ~But how did we get here in the first place? What fundamental principles govern the emergence of all life and intelligence, whether biological or artificial? What might the open-ended evolution of the ecosystem of our AI agents look like in the future? Though we don’t yet have the definitive answers to these questions, we can gain insight by returning to the scientific field that laid the groundwork for exploring these questions: Artificial Life (ALife). ALife offers the tools and framework to study the dynamics of artificial lifeforms, fostering insights into their potential behaviors, interactions, and trajectories. ~ So what is ALife? At its core, ALife is the ambitious quest to recreate and understand the phenomena of life itself—how it emerges, evolves, and thrives. It’s not just about mimicking Earth’s biology but going beyond and creating completely alien worlds to understand the principles that underlie all possible life. ALife researchers craft virtual ecosystems, robotic organisms, self-replicating programs, and biochemical simulations to uncover the deep mechanisms of complexity, evolution, and intelligence. ~ Sakana AI has previously drawn ideas from ALife to develop better foundation models, resulting in our works on evolutionary model merging, LLM self-play, and autonomous open-ended discovery. But now we want to go the other way: can foundation models help the study of ALife? Bridging this two-way road will be essential to getting more capable, natural systems and for understanding them as well. Regardless of whether or not you think foundation models will lead to the next generation of artificial lifeforms, they have already started revolutionizing various scientific fields. In fact, the recent Nobel Prize was awarded for radical advances in protein discovery, driven by a foundation model. They are also being used to predict the climate, do AI research itself, and prove mathematical theorems, so why not apply them to help in the search for artificial lifeforms?” → Website abstract: ✭ Automating the Search for Artificial Life with Foundation Models “With the recent Nobel Prize awarded for radical advances in protein discovery, foundation models (FMs) for exploring large combinatorial spaces promise to revolutionize many scientific fields. Artificial Life (ALife) has not yet integrated FMs, thus presenting a major opportunity for the field to alleviate the historical burden of relying chiefly on manual design and trial-and-error to discover the configurations of lifelike simulations. This paper presents, for the first time, a successful realization of this opportunity using vision-language FMs. The proposed approach, called Automated Search for Artificial Life (ASAL), (1) finds simulations that produce target phenomena, (2) discovers simulations that generate temporally open-ended novelty, and (3) illuminates an entire space of interestingly diverse simulations. Because of the generality of FMs, ASAL works effectively across a diverse range of ALife substrates including Boids, Particle Life, Game of Life, Lenia, and Neural Cellular Automata. A major result highlighting the potential of this technique is the discovery of previously unseen Lenia and Boids lifeforms, as well as cellular automata that are open-ended like Conway’s Game of Life. Additionally, the use of FMs allows for the quantification of previously qualitative phenomena in a human-aligned way. This new paradigm promises to accelerate ALife research beyond what is possible through human ingenuity alone.”
🔬Tech:
✭Gaussian Splatting 3D Creator, Viewer & Editor | Polycam “Our free gaussian splatting creator, viewer and editor quickly turns your images into immersive 3D splats that you can view, share, and export. Try it now!”
✭Fine-tune classifier with ModernBERT in 2025
👁️🗨 Research into AI:
✭ Large Concept Models: Language Modeling in a Sentence Representation Space | Research - AI at Meta “LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of LLMs is to process input and generate output at the token level. This is in sharp contrast to humans who operate at multiple levels of abstraction, well beyond single words, to analyze information and to generate creative content. In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a “concept”. Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow. Hence, we build a“Large Concept Model”. In this study, as proof of feasibility, we assume that a concept corresponds to a sentence, and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities. The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space. We explore multiple approaches, namely MSE regression, variants of diffusion-based generation, and models operating in a quantized SONAR space. These explorations are performed using 1.6B parameter models and training data in the order of 1.3T tokens. We then scale one architecture to a model size of 7B parameters and training data of about 7.7T tokens. We perform an experimental evaluation on several generative tasks, namely summarization and a new task of summary expansion. Finally, we show that our model exhibits impressive zero-shot generalization performance to many languages, outperforming existing LLMs of the same size. The training code of our models is freely available.” → ✭ Meta LCMs demonstrate human-like reasoning & problem-solving - Geeky Gadgets “Meta’s Large Concept Models (LCMs) introduce a shift from token-based to concept-based reasoning, allowing more coherent, contextually relevant, and human-like AI outputs. LCMs process language at a higher level of abstraction, predicting ideas or concepts rather than individual words, overcoming limitations like shallow understanding and repetitive outputs in traditional LLMs. The architecture of LCMs includes a Concept Encoder, Large Concept Model, and Concept Decoder, focusing on abstract meaning rather than surface-level text structure. LCMs excel in human-like reasoning and problem-solving by mimicking the process of starting with abstract ideas and refining them into specific details, improving tasks like essay writing and complex instruction adherence. Inspired by Meta’s V-JEPA architecture, LCMs prioritize abstraction and conceptual understanding, offering enhanced coherence, reduced repetition, and improved adaptability for applications like natural language processing and content generation.”
✭[2412.17747] Deliberation in Latent Space via Differentiable Cache Augmentation (GoogleDeepMind) “Techniques enabling large language models (LLMs) to "think more" by generating and attending to intermediate reasoning steps have shown promise in solving complex problems. However, the standard approaches generate sequences of discrete tokens immediately before responding, and so they can incur significant latency costs and be challenging to optimize. In this work, we demonstrate that a frozen LLM can be augmented with an offline coprocessor that operates on the model's key-value (kv) cache. This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding. We train this coprocessor using the language modeling loss from the decoder on standard pretraining data, while keeping the decoder itself frozen. This approach enables the model to learn, in an end-to-end differentiable fashion, how to distill additional computation into its kv-cache. Because the decoder remains unchanged, the coprocessor can operate offline and asynchronously, and the language model can function normally if the coprocessor is unavailable or if a given cache is deemed not to require extra computation. We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens. Furthermore, even without any task-specific training, our experiments demonstrate that cache augmentation consistently reduces perplexity and improves performance across a range of reasoning-intensive tasks.” → ✭Jonas Pfeiffer on X: "🧠💡 Our LLMs just had a ‘memory augmentation’—now they can deliberate like seasoned thinkers!
✭ [2412.17758] In Case You Missed It: ARC 'Challenge' Is Not That Challenging “ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged. We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.”
🔎 Applied Research:
✭SV Brain “The Silicon Valley Brain Company is a fundamental AI research lab focused on understanding human consciousness.” open-sourced → ✭ bxptr/cerebrum: Biologically Plausible Graph Neural Networks for Simulating Brain Dynamics and Inferring Connectivity → ✭ Introducing Cerebrum : What if we could simulate your brain? “Advancements in computational neuroscience are continually reshaping our understanding of the brain’s intricate networks. A key challenge in this field is deciphering the dynamic connectivity of neural networks, which is essential for both fundamental neuroscience and the development of clinical applications. To address this, we introduce Cerebrum, a novel framework that combines biologically inspired neuron models with cutting-edge machine learning techniques to simulate and infer synaptic connectivity in large-scale brain networks. ~ Bridging Biological Models and Machine Learning ~ Traditional approaches to studying brain networks often rely on graph theoretical methods that provide valuable insights into the static topological properties of neural connections. However, these methods typically overlook the temporal dynamics that are crucial for understanding how neuronal activity evolves over time. Cerebrum bridges this gap by integrating the Hodgkin-Huxley (HH) neuron model, known for its biological realism, with Graph Neural Networks (GNNs), which excel at learning complex patterns in graph-structured data. The HH model simulates the electrical characteristics of neurons, capturing the essential dynamics of action potentials and synaptic interactions. By combining this with GNNs, Cerebrum can effectively learn from simulated neuronal activity to predict the underlying synaptic connectivity. This integration allows for a more comprehensive analysis of brain networks, considering both their structural and dynamic properties.”
✭ AI-designed 'nanocages' mimic viral behavior for enhanced gene therapy “Researchers have developed an innovative therapeutic platform by mimicking the intricate structures of viruses using artificial intelligence (AI). Their pioneering research was published in Nature on December 18. ~ Viruses are uniquely designed to encapsulate genetic material within spherical protein shells, enabling them to replicate and invade host cells, often causing disease. Inspired by these complex structures, researchers have been exploring artificial proteins modeled after viruses. ~ These "nanocages" mimic viral behavior, effectively delivering therapeutic genes to target cells. However, existing nanocages face significant challenges: their small size restricts the amount of genetic material they can carry, and their simple designs fall short of replicating the multifunctionality of natural viral proteins. ~ To address these limitations, the research team used AI-driven computational design. While most viruses display symmetrical structures, they also feature subtle asymmetries. Leveraging AI, the team recreated these nuanced characteristics and successfully designed nanocages in tetrahedral, octahedral, and icosahedral shapes for the first time. ~The resulting nanostructures are composed of four types of artificial proteins, forming intricate architectures with six distinct protein-protein interfaces. Among these, the icosahedral structure, measuring up to 75 nanometers in diameter, stands out for its ability to hold three times more genetic material than conventional gene delivery vectors, such as adeno-associated viruses (AAV), marking a significant advancement in gene therapy. ~ Electron microscopy confirmed the AI-designed nanocages achieved precise symmetrical structures as intended.” → ✭Four-component protein nanocages designed by programmed symmetry breaking | Nature “Four, eight or twenty C3 symmetric protein trimers can be arranged with tetrahedral, octahedral or icosahedral point group symmetry to generate closed cage-like structures1,2. Viruses access more complex higher triangulation number icosahedral architectures by breaking perfect point group symmetry3,4,5,6,7,8,9, but nature appears not to have explored similar symmetry breaking for tetrahedral or octahedral symmetries. Here we describe a general design strategy for building higher triangulation number architectures starting from regular polyhedra through pseudosymmetrization of trimeric building blocks. Electron microscopy confirms the structures of T = 4 cages with 48 (tetrahedral), 96 (octahedral) and 240 (icosahedral) subunits, each with 4 distinct chains and 6 different protein–protein interfaces, and diameters of 33 nm, 43 nm and 75 nm, respectively. Higher triangulation number viruses possess very sophisticated functionalities; our general route to higher triangulation number nanocages should similarly enable a next generation of multiple antigen-displaying vaccine candidates10,11 and targeted delivery vehicles”
✭ An integrative data-driven model simulating C. elegans brain, body and environment interactions - Nature Computational Science “BAAIWorm is an integrative data-driven model of C. elegans that simulates interactions between the brain, body and environment. The biophysically detailed neuronal model is capable of replicating the zigzag movement observed in this species.”
✭ Magnetic swarm intelligence of mass-produced, programmable microrobot assemblies for versatile task execution: Device “Swarm robotics has emerged as a promising methodology for accomplishing complicated tasks through the collective behavior of multiple robots. Through inter-robot communications, robotic swarms can execute terrain reconnaissance, pattern formation, and cargo transportation. However, in miniaturized robotic systems, robots possess low kinetic energy to operate in various environments owing to their low body mass. Furthermore, battery- and sensor-free actuation of robots complicates inter-robot communication, limiting the extension of their functionalities. Herein, we present multifunctional swarm intelligence capable of versatile task execution via mass-produced magnetic microrobot swarms with programmed assembly configurations. The versatile task execution via microrobot swarms exhibits significant potential for various applications in robotic engineering, expanding foundational technology for developing advanced collective robot systems.”
👀Watching:
✭Part 3| How I built Forever Land... using Sora + ChatGPT. (@dailydall.e on TikTok) “Last week, I posted a video for a fictional video game I had bouncing around in my head that was an ode to "Little Big Planet". This week, I wanted to give you all a little BTS on how I built it... all in 2.5 days using OpenAl tools, Final Cut and an audio collection from Epidemic Sound. Hopefully this will give you some ideas & inspiration as you dive into Sora and your own Al production flows." - Chad Nelson, @dailydall.e on IG.”
✭ Self Models of Loving Grace (Joscha Bach | CCC 2024) “Artificial Intelligence is not just an engineering discipline, but also the most fascinating and important philosophical project ever attempted: the explanation of the mind, by recreating it. This part of the series "From Computation to Consciousness" focuses on the nature of the self, agency and identity. ~ When we recognize the paradigm of Artificial Intelligence as a philosophical and scientific framework for understanding the nature of minds like ours, we may begin with an essential question: What does it mean for a machine to feel? How do emotions arise at the intersection between a self and its world—or more precisely, within an a reflexive self model, in response to being dynamically reconfigured by a motivational system, in response to shifts in its alignment to a model of its environment, all within the same mind? ~ This inquiry takes us to the core of our own psychological architecture. Who are we when our self-perception alters? What does it mean to depersonalize, to dissolve the boundaries of the self? Can we reverse engineer, debug and reconstruct our identities to become who we want to be? Is there free will? Is it possible to recreate self and sentience in nonbiological substrates? Can AI be conscious? Could we perhaps even extend our own self to non biological substrates? ~ This presentation is part of the philosophical series “From Computation to Consciousness,” which draws on insights from AI and cognitive science to explore the nature of intelligence, consciousness, and their realization in the physical universe.”