How to download GGUF AI models from Huggingface

Learn how you can download AI models for Ozeki AI.

What is a GGUF model?

GGUF (GPTQ for GGML Unified Format) models are compressed versions of large language models, optimized for use in environments with limited computational power. Designed for the GGML framework, they enable efficient AI processing on CPUs, offering faster speeds and lower memory requirements while maintaining core functionality.

What is Hugging Face?

Hugging Face is a comprehensive platform and community focused on machine learning (ML) and artificial intelligence (AI). It enables users to build, train, and deploy ML models using open-source technologies. The platform serves as a hub where data scientists, researchers, and ML engineers can share models, datasets, and collaborate on projects.

Quick link

The quickest way to open the Hugging Face AI model library with the GGUF filter is to click on the following link. After you have clicked on this link, you may select the model, and in the "Files" section, you may click on the GGUF file.

Model search link:

https://huggingface.co/models?library=gguf&sort=trending

Some of our favoirte model publishers:

Downloading a GGUF model from Huggingface (TL;DR version)

  • Step 1 - Go to huggingface.co
  • Step 2 - Click on Models
  • Step 3 - Click on Libraries
  • Step 4 - Click GGUF
  • Step 5 - Select the model
  • Step 6 - Select "Files and versions" tab
  • Step 7 - Click on the GGUF file
  • Step 8 - Movel the GGUF file into C:\AIModels

Downloading a GGUF model from Huggingface (video instructions)

The video below walks you through all the steps in motion, starting with navigating to the website, all the way to moving the LLM file to the right folder. It's a worthwhile use of your time, since it takes a mere 75 seconds to watch, but captures every important step in detail.

Downloading a GGUF model from Huggingface (step-by-step guide)

In your browser, enter huggingface.co in the searchbar. You should land on a page similar to Figure 1.

Open huggingface.co page
Figure 1 - Open huggingface.co page

First, click Models near the top edge of the screen. In the left panel, set the filter to Libraries, then select GGUF from the available options, as seen in Figure 2.

Select GGUF models
Figure 2 - Select GGUF models

Select your preferred LLM model, as shown in Figure 3. For the sake of this tutorial, we'll be using the following model:

Meta-Llama-3-3.1-8B-Instruct-hf-Q4_K_M-GGUF

Select model library
Figure 3 - Select model library

On the page of the model, open the Files and versions tab, and look for the file with a .gguf format, highlighted in red in Figure 4.

Download model file
Figure 4 - Download model file

Download the .gguf file, and move or paste it to the following folder (Figure 5):

C:\AIModels

Figure 5 - Copy GGUF model file to C:\AIModels

Summary

With your GGUF model downloaded and placed in the correct folder, Ozeki AI Studio is now ready to leverage its capabilities. By following these steps, you've ensured that your AI system can efficiently access and use the model.

To find out how you can create your very own chat bot using the file you've just downloaded visit our AI chat bot sites.

More information