As enterprises proceed to adopt large language models (LLM) across a wide range of applications, one in all the important thing challenges they face is improving model material knowledge and reducing hallucinations. In a new article, scientists from Meta-artificial AI propose “scalable memory tiers”, which could also be one in all several possible solutions to this problem.
Scalable memory layers add more parameters to LLMs to extend their learning capability without requiring additional computational resources. The architecture is beneficial in applications where additional memory might be saved for factual knowledge, but at the identical time the speed of inference and the agility of models are required.
Dense and memory layers
Traditional language models use “dense layers” to encode massive amounts of data into their parameters. In dense layers, all parameters are used at full capability and are most frequently activated at the identical time during inference. As they grow, dense layers can learn more complex functions, but increasing their size requires additional computational and energy resources.
In contrast, for easy factual knowledge, much simpler layers with an associative memory architecture resembling lookup tables can be more efficient and easier to interpret. This is what memory layers do. They use easy, sparse activations and key-value lookup mechanisms to encode and retrieve knowledge. Sparse layers take up more memory than dense layers, but only use a small fraction of the parameters at a time, making them far more computationally efficient.
Memory layers have been around for several years, but are rarely utilized in modern deep learning architectures. They are usually not optimized for current hardware accelerators.
Current pioneering LLMs typically use some type of “mix of experts” (MoE) architecture, which uses a mechanism somewhat much like memory tiers. MoE models consist of many smaller expert components that focus on specific tasks. At the time of inference, the routing engine determines which expert can be activated based on the input sequence. PEER, an architecture recently developed by Google DeepMind, extends MoE to tens of millions of experts, providing more granular control over the parameters activated during inference.
Updating memory layers
Memory tiers eat little processing power but are memory-intensive, creating particular challenges for current hardware and software structures. In their paper, Meta researchers propose several modifications that address these challenges and enable their large-scale use.
First, the researchers configured the memory layers for parallelism, spreading them across several GPUs to store tens of millions of key-value pairs without changing other layers within the model. They also implemented a special CUDA kernel to handle operations requiring high memory bandwidth. They also developed a parameter sharing mechanism that supports a single set of memory parameters across multiple memory layers inside the model. This means that the keys and values used for searching are common across layers.
These modifications enable the implementation of memory tiers inside LLM without slowing down the model.
“Memory layers with their sparse activations nicely complement dense networks, providing increased capacity for knowledge acquisition while providing low computational overhead,” the researchers write. “They can scale efficiently and provide practitioners with a compelling new direction in the memory-computing trade-off.”
To test the memory layers, researchers modified Lamy’s models by replacing a number of dense layers with a shared memory layer. They compared memory-enhanced models with dense LLMs, in addition to MoE and PEER models, on several tasks, including answering fact-based questions, science and common sense knowledge, and coding.
Their findings show that memory models improve significantly over dense baselines and compete with models that use 2-4 times more computational power. They also match the performance of MoE models that have the identical computational budget and variety of parameters. The model’s performance is especially noteworthy for tasks that require material knowledge. For example, with regards to answering fact-based questions, the 1.3 billion-parameter memory model approaches the performance of Llama-2-7B, which was trained on twice as many tokens and 10 times as much processing power.
Moreover, the researchers found that the advantages of memory models remained consistent with model size after they scaled their experiments from 134 million to eight billion parameters.
“Given these findings, we strongly advocate the integration of memory layers into all next-generation AI architectures,” the researchers write, while adding that there remains to be far more room for improvement. “In particular, we hope to develop new learning methods that will further improve the effectiveness of these layers, enabling less forgetting, less hallucinations, and continuous learning.”
