Sparse autoencoders (SAE) use the concept of autoencoder with a slight modification. During the encoding phase, the SAE is forced to only activate a small number of the neurons in the intermediate ...

In particular, the sparse autoencoder that supports GPT-4 was able to find 16 million features of GPT-4. OpenAI has published the features found from GPT-4 and GPT-2 small and the corresponding ...

The research showed that: when a globally sparse randomly connected network (RN) is rewired to a more biologically realistic modular network (MN), the network's running consumption (neuron firing ...