Again Theory, DeepMind’s Gemini, XGen 7B, Mofi, RoboCat, Building an Autograd Engine Interactive Guide, Autonomous Agents, AI and the automation of work, AI-predicted protein structure clustering
June 27th July 3rd: + de novo computational protein design, Advanced Cellular and Endocytic profiling for Intracellular Delivery (ACE-ID), Fruit Fly AI, Enactive Gender Norms in AI, Sentient Syllabus
🏓 Observations : Again Theory, Enactive Gender Norms in AI, Sentient Syllabus, AI and the automation of work, Transformers and Time Series, GPT detectors Bias
✭ Again Theory: A Forum on Language, Meaning, and Intent in the Time of Stochastic Parrots | In the Moment “Published in the March 2021 issue of FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, few scientific papers have captured as eager a public audience as “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜,” jointly authored by Emily Bender, Timnit Gebru, Angelina McMillan-Major, and “Shmargaret Shmitchell” (the last being Margaret Mitchell, whose smudgy pseudonym did not protect her and Gebru from being terminated from positions at Google in its aftermath). It has been downloaded almost 200,000 times from the official ACM site alone, rendering it obligatory in any discussion of the new generative artificial intelligence”
Matthew Kirschenbaum’s introduction to the forum
Lisa Siraganian’s “On Accidental and Parasitic Language“
Hannes Bajohr’s “When in Doubt, Go to the Beach“
Seth Perlow’s “Intention and Text Machines“
Rita Raley and Russell Samolsky’s “Against AI?“
Tyler Shoemaker’s “Machines, Reading“
Annette Vee’s “Against Output“
Alex Gil ‘s “On the Uses of Text beyond Intention“
Caroline Bassett’s “Silicon Beach“
Kari Kraus’s “A View from the Periscope“
Ted Underwood’s “The Empirical Triumph of Theory“
Steven Knapp and Walter Benn Michaels’s “Here Is a Wave Poem that I Wrote . . . I Hope You Like It!“
N. Katherine Hayles’s “Afterword: Learning to Read AI Texts“
✭ Frontiers | Enactive artificial intelligence: subverting gender norms in human-robot interaction “gender cannot be understood in isolation from other social identities and factors that shape individuals' experiences and opportunities in life (Crenshaw, 1997, 2023; Collins, 2002b). Understanding these multiple dimensions of gender is crucial for creating a more inclusive and equitable society that values diversity and respects the experiences of all individuals. With respect to AI, specifically, feminist theory shows that institutions, technology, digitalisation and artificial intelligence systems and structures in society work against individuals based on their gender, as well as other intersecting factors such as race, sexuality, and class (Shields, 2008; Strayhorn, 2013; Beck et al., 2022; Birhane et al., 2022). A singular focus on gender alone is insufficient. It is necessary to address how other forms of oppression intersect and influence women's embodied experiences (Strayhorn, 2013; Ciurria, 2019; Losh and Wernimont, 2019; UN Women, 2020).”
✭ (5) Just the principles ... - by The Sentient Syllabus Project “Three Principles
An AI cannot pass a course.
AI contributions must be attributed and true.
AI use should be open, and documented.”
✭ AI and the automation of work — Benedict Evans “ChatGPT and generative AI will change how we work, but how different is this to all the other waves of automation of the last 200 years? What does it mean for employment? Disruption? Coal consumption?”
✭ Data Machina #208 - Data Machina “Transformers and Time Series. LLM Autonomous Agents. English PySparkAI. Machine Unlearning. RAG with Graphs. Private MPT-30b. Instruction Tuning. LLaVAR.”
✭Don't use AI detectors for anything important “because AI detectors produce false positives, it's unethical to use them to detect cheating” ✭ [2304.02819] GPT detectors are biased against non-native English writers “In this study, we evaluate the performance of several widely-used GPT detectors using writing samples from native and non-native English writers. Our findings reveal that these detectors consistently misclassify non-native English writing samples as AI-generated, whereas native writing samples are accurately identified. Furthermore, we demonstrate that simple prompting strategies can not only mitigate this bias but also effectively bypass GPT detectors, suggesting that GPT detectors may unintentionally penalize writers with constrained linguistic expressions. Our results call for a broader conversation about the ethical implications of deploying ChatGPT content detectors and caution against their use in evaluative or educational settings, particularly when they may inadvertently penalize or exclude non-native English speakers from the global discourse.”
✭ Microsoft, OpenAI sued for 3B$ for ChatGPT 'privacy violations' “"With respect to personally identifiable information, defendants fail sufficiently to filter it out of the training models, putting millions at risk of having that information disclosed on prompt or otherwise to strangers around the world," the complaint says, citing The Register's March 18, 2021 special report on the subject. The 157 page complaint is heavy on media and academic citations expressing alarm about AI models and ethics but light on specific instances of harm. For the 16 plaintiffs, the complaint indicates that they used ChatGPT, as well as other internet services like Reddit, and expected that their digital interactions would not be incorporated into an AI model.
🛠️ Tech : DeepMind’s Gemini, XGen 7B, Mofi, RoboCat, Building an Autograd Engine Interactive Guide, Autonomous Agents, Loihi 2 (Neuromorphic Computing Chip), BCI HIL (Human-in-Loop), Sophia: A Scalable Stochastic Second-order Optimizer
✭ Google DeepMind CEO Demis Hassabis Says Its Next Algorithm Will Eclipse ChatGPT | WIRED “DeepMind’s Gemini, which is still in development, is a large language model that works with text and is similar in nature to GPT-4, which powers ChatGPT. But Hassabis says his team will combine that technology with techniques used in AlphaGo, aiming to give the system new capabilities such as planning or the ability to solve problems....“I would love to see academia have early access to these frontier models,” he says—”
✭ Mofi // Content-aware fill and trim for music! “Shorten and lengthen a song, making it the perfect length to match a video or performance. Keep the vibe of the song with seamless transitions.”
✭ RoboCat: A self-improving robotic agent | DeepMind “New foundation agent learns to operate different robotic arms, solves tasks from as few as 100 demonstrations, and improves from self-generated data.”
✭ Long Sequence Modeling with XGen: A 7B LLM Trained on 8K Input Sequence Length | Salesforce “We trained a series of 7B LLMs named XGen-7B with standard dense attention on up to 8K sequence length for up to 1.5T tokens. We also fine tune the models on public-domain instructional data. The main take-aways are:
On standard NLP benchmarks, XGen achieves comparable or better results when compared with state-of-the-art open-source LLMs (e.g. MPT, Falcon, LLaMA, Redpajama, OpenLLaMA) of similar model size.
Our targeted evaluation on long sequence modeling benchmarks show benefits of our 8K-seq models over 2K- and 4K-seq models.
XGen-7B archives equally strong results both in text (e.g., MMLU, QA) and code (HumanEval) tasks.
Training cost of $150K on 1T tokens under Google Cloud pricing for TPU-v4.”
✭ Building Autograd Engine & Neural Network Library: An Interactive Guide “In this article, we will build an Autograd engine and a neural network library that handle an N-dimensional array. Autograd is a tool used for derivative calculation. It tracks operations on values with enabled gradients and builds a dynamic computational graph — a graph without cycles. Input values serve as the leaves of the graph, while output values act as its roots. Gradients are computed by traversing the graph from root to leaf, applying the chain rule to multiply gradients at each step.”
✭ LLM Powered Autonomous Agents | lilianweng.github “Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver. In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components: planning, memory, tool use.”
✭ Loihi 2 | Neuromorphic Computing - Next Generation of AI “Intel Labs’ second-generation neuromorphic research chip, codenamed Loihi 2, and Lava, an open-source software framework, will drive innovation and adoption of neuromorphic computing solutions.”
✭ Frontiers | An open-source human-in-the-loop BCI research framework: method and design “an open-source research framework that features a modular and customizable hardware-independent design. This framework facilitates human-in-the-loop (HIL) model training and retraining, real-time stimulus control, and enables transfer learning and cloud computing for the online classification of electroencephalography (EEG) data. Stimuli for the subject and diagnostics for the researcher are shown on separate displays using web browser technologies. Messages are sent using the Lab Streaming Layer standard and websockets. Real-time signal processing and classification, as well as training of machine learning models, is facilitated by the open-source Python package Timeflux.” ✭ GitHub - bci-hil/bci-hil: Brain Computer Interface Humain-in-the-Loop Research Framework
✭GitHub - Nixtla/neuralforecast: Scalable and user friendly neural forecasting algorithms. “NeuralForecast offers a large collection of neural forecasting models focused on their usability, and robustness. The models range from classic networks like MLP, RNNs to novel proven contributions like NBEATS, TFT and other architectures.”
✭ Stanford team develops a faster, cheaper way (Sophia) to train large language models “If an optimization program can estimate that curvature (workload), it can make LLM pretraining more efficient. The problem is this: Estimating curvature with existing methods is remarkably difficult and expensive. "In fact, it's more expensive than doing the actual work without making curvature predictions," Liu says. That's partially why the current state-of-the-art approaches to optimizing LLM pretraining (Adam and its variants) forgo the curvature estimation step. Still, Liu and his colleagues noticed a possible inefficiency in the prior methods that used parametric curvature estimation: Prior researchers updated their curvature estimates at every step of the optimization. The Stanford team wondered if they could make the process more efficient by decreasing the number of updates. To test that idea, the Stanford team designed Sophia to estimate parameters' curvature only about every 10 steps. "That turned out to be a huge win," Liu says. The team's second optimization trick, called clipping, addresses a related issue: The problem of inaccurate curvature estimation. "If the estimation is wrong, it's like giving people with hard jobs even more work to do. It makes things worse than if there were no estimation at all." Clipping prevents that by setting a threshold, or a maximum curvature estimation. "In our factory metaphor, it's like setting a workload limitation for all employees," Liu says. Another metaphor often applied to optimization is a landscape of hills and valleys where the goal is to end up in the lowest valley. Without clipping, Liu says, it is possible to land at a saddle between two mountains. "In optimization, that's not where you want to be," he says.” ✭ [2305.14342] Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
🔎 Research : AI-predicted protein structure clustering, de novo computational protein design, Advanced Cellular and Endocytic profiling for Intracellular Delivery (ACE-ID), Fruit Fly AI, DeePAy (Neuromarketing), Neuromorphic Spiking Models
✭ Researchers develop new base editing tools using AI-predicted protein structure clustering “This study highlights an approach that uses just the cytidine deaminase superfamily to develop a suite of new technologies and uncover new protein functions. These newly discovered deaminases, based on AI-assisted structural predictions, greatly expand the utility of base editors for therapeutic and agricultural applications.” ✭ Discovery of deaminase functions by structure-based protein clustering: Cell “AI-guided structural classification establishes new deaminase family relationships. Further AI-assisted truncation enables AAV packaging and efficient soybean editing.”
✭ Digitally designed protein works like an antifreeze for biological material “researchers have used computer simulations to develop a protein that works like an anti-freeze agent. Researchers could use this protein to freeze and defrost biological material such as immune cells, sperm and perhaps even donor organs in the future, without causing any damage to the material.” ✭ De novo designed ice-binding proteins from twist-constrained helices | PNAS “We demonstrate through de novo computational protein design that constraining the twist of an ice-binding helix, to align its threonine residues, is a key feature determining its ice-binding activity. This finding opens avenues for the design of synthetic IBPs with activities tailored to the requirements of specific applications, such as cell and tissue cryopreservation.”
✭ Understanding Intracellular Biology to Improve mRNA Delivery by Lipid Nanoparticles - Hunter - Small Methods - Wiley Online Library “Poor understanding of intracellular delivery and targeting hinders development of nucleic acid-based therapeutics transported by nanoparticles. Utilizing a siRNA-targeting and small molecule profiling approach with advanced imaging and machine learning biological insights is generated into the mechanism of lipid nanoparticle (MC3-LNP) delivery of mRNA. This workflow is termed Advanced Cellular and Endocytic profiling for Intracellular Delivery (ACE-ID). A cell-based imaging assay and perturbation of 178 targets relevant to intracellular trafficking is used to identify corresponding effects on functional mRNA delivery. Targets improving delivery are analyzed by extracting data-rich phenotypic fingerprints from images using advanced image analysis algorithms. Machine learning is used to determine key features correlating with enhanced delivery, identifying fluid-phase endocytosis as a productive cellular entry route. With this new knowledge, MC3-LNP is re-engineered to target macropinocytosis, and this significantly improves mRNA delivery in vitro and in vivo. The ACE-ID approach can be broadly applicable for optimizing nanomedicine-based intracellular delivery systems and has the potential to accelerate the development of delivery systems for nucleic acid-based therapeutics.”
✭ New AI system can decode fruit fly behaviors: Why that's pivotal for future human genetics research “a new AI tool that can tell you if a fruit fly is hungry, sleepy or singing (yes, fruit flies sing). Dubbed MAFDA (for Novel Machine-learning-based Automatic Fly-behavioral Detection and Annotation) the system uses cameras and a newly developed software to track and identify complex interactive behaviors of individual flies within a larger group. This allows researchers to compare and contrast the behaviors of fruit flies with different genetic backgrounds.” ✭ Integrating lipid metabolism, pheromone production and perception by Fruitless and Hepatocyte Nuclear Factor 4 | Science Advances
✭ Frontiers | DeePay: deep learning decodes EEG to predict consumer’s willingness to pay for neuromarketing ”There is an increasing demand within consumer-neuroscience (or neuromarketing) for objective neural measures to quantify consumers’ subjective valuations and predict responses to marketing campaigns. However, the properties of EEG raise difficulties for these aims: small datasets, high dimensionality, elaborate manual feature extraction, intrinsic noise, and between-subject variations. We aimed to overcome these limitations by combining unique techniques of Deep Learning Networks (DLNs), while providing interpretable results for neuroscientific and decision-making insight. In this study, we developed a DLN to predict subjects’ willingness to pay (WTP) based on their EEG data.”
✭Frontiers | Reservoir based spiking models for univariate Time Series Classification “A variety of advanced machine learning and deep learning algorithms achieve state-of-the-art performance on various temporal processing tasks. However, these methods are heavily energy inefficient—they run mainly on the power hungry CPUs and GPUs. Computing with Spiking Networks, on the other hand, has shown to be energy efficient on specialized neuromorphic hardware, e.g., Loihi, TrueNorth, SpiNNaker, etc. In this work, we present two architectures of spiking models, inspired from the theory of Reservoir Computing and Legendre Memory Units, for the Time Series Classification (TSC) task. Our first spiking architecture is closer to the general Reservoir Computing architecture and we successfully deploy it on Loihi; the second spiking architecture differs from the first by the inclusion of non-linearity in the readout layer. Our second model (trained with Surrogate Gradient Descent method) shows that non-linear decoding of the linearly extracted temporal features through spiking neurons not only achieves promising results, but also offers low computation-overhead by significantly reducing the number of neurons compared to the popular LSM based models—more than 40x reduction with respect to the recent spiking model we compare with. We experiment on five TSC datasets and achieve new SoTA spiking results (—as much as 28.607% accuracy improvement on one of the datasets), thereby showing the potential of our models to address the TSC tasks in a green energy-efficient manner. In addition, we also do energy profiling and comparison on Loihi and CPU to support our claims.”