OpenCoder

OpenCoder is an open and reproducible series of code LLMs, featuring 1.5B and 8B base models as well as chat models, with support for both English and Chinese. Pretrained from scratch on 2.5 trillion tokens composed of 90% original code and 10% code-related web data, and fine-tuned with supervision on over 4.5 million high-quality SFT examples, OpenCoder achieves performance on par with top-tier code LLMs. We provide not only the model weights and inference code but also reproducible training data, a complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols. _OpenCoder empowers researchers to build and innovate, serving as your open foundation for advancing code AI. Fully Open-Source: OpenCoder goes beyond just releasing model weights and upcoming inference code by also publishing the complete data cleaning code to ensure full transparency. This release includes high-quality synthetic data, an extensive set of checkpoints, and a dataset of over 4.5 million supervised fine-tuning (SFT) entries, making OpenCoder one of the most comprehensive open models available. Thorough Experimental Analysis: OpenCoder undergoes rigorous testing through extensive ablation studies on various data cleaning strategies and training processes, including experiments at both file-level and repository-level deletions, ensuring thorough exploration and validation of the model's performance. High-Quality Synthetic Data: OpenCoder offers a fully developed synthetic data generation process, yielding over 4.5 million SFT data entries, establishing a robust data foundation for model training and evaluation. Outstanding Performance: OpenCoder delivers high performance across multiple language model benchmarks, positioning it among the leading open-source models for code.

OpenCoder is an open and reproducible family of code large language models (LLMs), including 1.5B and 8B base and conversational models, supporting both Chinese and English. It aims to provide researchers with an open foundation to advance code AI and facilitate building and innovation.