Running Code Llama Locally

Steps to setup Code Llama locally and my experience with it.

Looking for an AI coding assistant that can run locally? Follow the steps below, and you'll have codeLlama up and running in just a few minutes. While its performance might not match ChatGPT's, it operates locally, allowing you to provide potentially sensitive context that you might hesitate to feed into a system like ChatGPT.

1. Install Ollama - https://ollama.ai/ and follow instructions to install the Ollama command line tool

2. Launch the terminal.

3. Execute the following command: ollama run codellama

4. Wait for the download to complete, which involves roughly 5GB of data.

5. Once you see >>> on the screen, input your prompt.

6. Patiently wait for the response. Some times it can be snappy but in my experience, it's can take up to ~2 minutes for a response to start generating.

In my trials, using Code Llama as opposed to ChatGPT powered by GPT-4 has been more miss than hit. I haven't posed it a rigorous challenge and mostly experimented for fun, ChatGPT consistently outperformed it in terms of first-time correct responses and with troubleshooting code. At the time of writing, only the Instruct 7B parameter is available for execution via Ollama so as support for the 13b model is added some of the performance may improve. For updates, consider checking the repository link provided below.

Some issues observed with the 7B model on Ollama:

It tends to add extra end brackets to the code, e.g., (something)).

The model can sometimes "hallucinate" function usage.

Context maintenance can be inconsistent. For instance, I once requested Python code and upon a subsequent request for an addition, the response was in JavaScript.

Code Llama AI for Coding

Ollama Repository

CodeLlama Research by Facebook

Note: My tests were conducted on a MacBook Pro M1 with 32GB of RAM.

Additionally, the code from the screenshot at the start of the article does not run. When I fixed the errors and did get it running, below is the "Llama" that it drew(left) vs the Llama ChatGPT drew(right). Is it ridiculous to have it draw a Llama? Of course but why not have some fun with it. I am also impressed that ChatGPTs version was close to a basic drawing and only. I made 0 tweaks to the drawing portion of the code.

↑Back to Top