Are Llama and ChatGPT Really an Open-Source
Meta, the social media and advertising-technology company, has recently made a significant development by releasing an update for its prominent language model, Llama. This latest version, named Llama 2, has been made open source, granting users access to the model’s weights, evaluation code, and comprehensive documentation. The motive behind Meta’s open-source release is to ensure that the model becomes accessible to a wide range of users, including individuals, creators, researchers, and businesses. By doing so, Meta aims to encourage experimentation, innovation, and responsible scaling of ideas within the community.
However, when compared to other open-source Language Model Models (LLMs) and open-source software packages in general, Llama 2 exhibits a relatively closed-off approach. While Meta has made the trained model available, it has not shared the model’s training data or the specific code used for its training. This limitation means that aspiring developers and researchers have restricted access to scrutinize the model in its entirety, unlike some other open-source LLMs and software.
At the ACM Conference on Conversational User Interfaces, a group of AI researchers from Radboud University in Nijmegen, Netherlands, shed light on the issue of Llama 2’s “open-source” label. In their research paper, they present a multidimensional assessment of model openness, using a rubric to evaluate 15 different nominally open-source LLMs based on their availability, documentation, and methods of access. The researchers have compiled these assessments into an online table, which has since been expanded to include 21 different open-source models. Even smaller research-focused models were considered in the evaluation if they met the criteria of being genuinely open, well-documented, and released under an open-source license, as stated in the preprint.
“Meta using the term ‘open source’ for this is positively misleading.” — Mark Dingemanse, Radboud University
