Earlier in March Google quietly released a new language model or an upgraded one that is supposed to be a response to ChatGPT. Similar to ChatGPT, it can do questions and answers and various other language tasks. The multimodal aspect of it makes it quite interesting because as stated on the blog it can be used for various tasks in robotics and other areas.
“On the one hand, PaLM-E was primarily developed to be a model for robotics, and it solves a variety of tasks on multiple types of robots and for multiple modalities (images, robot states, and neural scene representations).
At the same time, PaLM-E is a generally-capable vision-and-language model. It can perform visual tasks, such as describing images, detecting objects, or classifying scenes, and is also proficient at language tasks, like quoting poetry, solving math equations, or generating code.”
PaLM-E is a combination of various models such as the PaLM which was another language model that was released earlier and Vit-22B which is an advanced vision model that has state-of-the-art results. The interesting thing about PaLM-E is that it gives us a glimpse into what Google is trying to aim for, where ChatGPT was seen as a consumer-facing application, PaLM-E seems to be focused on robotics which is a highly lucrative field.
Traditionally, language models are trained with textual input, with PaLM-E it is mainly trained with raw streams of robot sensory data. This is an interesting new development because models in this category can absorb all sorts of data from the environment and then be updated and calibrated in real-time which is similar to how humans operate by absorbing data through our bodily senses.
PaLM-E seems like a very promising system, it might not have the buzz around it such as ChatGPT but is a powerful one nonetheless, given the fact that this one is robotic-based, there will be some interesting applications coming from this model.
