Quick Tour: Ollama+Gemma+LangChain

Utilize Ollama for hosting Gemma and utilize LangChain for developing an AI application with a local model.

I think there’s no need to explain what Gemma is. Through this tutorial, you can set up a Gemma-based AI on your local machine with the assistance of Ollama and LangChain.

Ollama style

Locate the model

gemma (ollama.com)

Pull model

ollama run gemma:7b
or
ollama run gemma:2b

Run model

ollama run gemma:2b

Then in the terminal you can see

ollama run gemma:2b
pulling manifest 
pulling c1864a5eb193... 100% ▕█████████████████████████████████████████████████████████████████████████████▏ 1.7 GB                         
pulling 097a36493f71... 100% ▕█████████████████████████████████████████████████████████████████████████████▏ 8.4 KB                         
pulling 109037bec39c... 100% ▕█████████████████████████████████████████████████████████████████████████████▏  136 B                         
pulling 22a838ceb7fb... 100% ▕█████████████████████████████████████████████████████████████████████████████▏   84 B                         
pulling 887433b89a90... 100% ▕█████████████████████████████████████████████████████████████████████████████▏  483 B                         
verifying sha256 digest 
writing manifest 
removing any unused layers 
success 
>>>

After >>> you can write any question or query you want.

Check model list

ollama list

Integrate with LangChain

A runable command-line app

from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_community.llms.ollama import Ollama

llm = Ollama(
    base_url="http://localhost:11434",
    model="gemma:2b",
    callback_manager=CallbackManager(
        [StreamingStdOutCallbackHandler()],
    ),
)
while True:
    query = input("Enter a query: ")
    llm.invoke(query)
    print()