Hy3 (hy_v3) Model Quantization and Deployment Guide for llama.cpp
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Read the full articleYou might also wanna read
Tencent Hy3 295B 1bit Quantization: 598GB to 85GB — Single GPU Flagship Model Deployment, llama.cpp Ecosystem Deep Dive
Tencent Hy3 295B 1bit Quantization: 598GB to 85GB — Single GPU Flagship Model Deployment, llama.cpp Ecosystem Deep Dive 1. Introduction On J
Enhancing Mistral Models Integration with llama.cpp: Key Features and Fixes
Description This PR aims to enhance the integration of Mistral models with llama.cpp by addressing several key issues and introducing new fe
github.com·11mo agoRunning LLMs Locally with Ollama and llama.cpp: A Developer's Guide
Practical developer guide to running local LLMs: hardware, quantization, setup, APIs, and integrating models into workflows.
Developer flags harness bug after local 35B model passes task but fails eval on stop token
A developer running agentic evaluations on a local 35-billion-parameter quantized model (ornith-1.0-35b-Q8_0) via llama.cpp found a mismatch
GGUF File Format: What Metadata It Stores for Language Models and What's Still Missing
What extra stuff is needed to properly run a language model? Besides the weights of a language model, what is the gguf metadata that we need
nobodywho.ooo·2mo agoHow to Compile and Cross-Compile llama.cpp for ARM64 on Debian 12/13
llama.cpp is a lightweight C/C++ inference runtime for running large language models locally using the GGUF format on x86_64 servers and ARM
Comments
Sign in to join the conversation.
No comments yet. Be the first.