Deep Dive with Llama-Cpp-Python

This week we will take another look at llama-ccp, this time using the python wrapper available in llama-cpp-python. This library can be installed and imported directly, or run as a separate service available through the OpenAI API.

We covered the basics of llama-cpp last year and looked through the different quantization approaches available. It provides a great way to run a lower level model using only CPU resources and RAM. There is some quality loss due to the quantization, but the results are good enough for prototyping at a much lower cost.

Agenda:

• Introduction to Llama-cpp-python
• Installation and configuration
• Python API
• OpenAI API

Series:

Over the next several sessions, we will be diving deeper into separate components needed for RAG – hopefully resulting in a chat-based Q&A service for the NASA Technical Report Server.  We were introduced to this data during our submission for the 2022 NASA SpaceApps Challenge – where we placed 2nd. Our submission was a semantic search based on the abstracts for the NSTR dataset of 10,000 papers.

I hope to have the video from our last session posted today. You can check for updates at https://hsv.ai/videos

Links:

• https://github.com/abetlen/llama-cpp-python
• https://llama-cpp-python.readthedocs.io/en/latest/
• Huntsville AI 2022 SpaceApps Submission – https://github.com/HSV-AI/spaceapps2022

Details:

• Date – 05/01/2024
• Time – 6-7pm
• Location – HudsonAlpha
• Address –  601 Genome Way Northwest, Huntsville, AL 35806
• Zoom –https://us02web.zoom.us/j/82295484520?pwd=cU1ETGNrS3NtY1hsUmNmRVJDUmNsUT09