-
Notifications
You must be signed in to change notification settings - Fork 6
Explicit Memory #8
Copy link
Copy link
Open
Labels
MLRequires machine-learning knowledge (can be built up on the fly)Requires machine-learning knowledge (can be built up on the fly)coreImproves core model while keeping core idea intactImproves core model while keeping core idea intactresearchCreative project that might fail but could give high returnsCreative project that might fail but could give high returns
Description
Metadata
Metadata
Assignees
Labels
MLRequires machine-learning knowledge (can be built up on the fly)Requires machine-learning knowledge (can be built up on the fly)coreImproves core model while keeping core idea intactImproves core model while keeping core idea intactresearchCreative project that might fail but could give high returnsCreative project that might fail but could give high returns
Many modern architectures, such as Memorizing Transformers, RETRO and PKM have an explicit memory where the model can retrieve information from and optionally even store it. Some hypothesise that Mixture-of-Experts embeds fuzzy representations of books and other things it must memorise into its weights.
That's why adding explicit memory to our models could give them a considerable boost in performance. Instead of storing this information in dense layers and having the weights fight about whether they should be storing concepts or memorising sequences, our model would be able to do both.
This issue is about implementing such an explicit memory (be it PKM, MoE or even a new architecture) and improving the convergence of our language model at the same runtime.