New Show Hacker News story: Show HN: Open-source proxy server for Llama2, GPT-4, Claude2 with Logging,Cache

Show HN: Open-source proxy server for Llama2, GPT-4, Claude2 with Logging,Cache
5 by ij23 | 0 comments on Hacker News.
Hello hacker news, I’m the maintainer of liteLLM() - package to simplify input/output to OpenAI, Azure, Cohere, Anthropic, Hugging face API Endpoints: https://ift.tt/gfPSUip We’re open sourcing our implementation of liteLLM proxy: https://ift.tt/hqxiDVP... TLDR: It has one API endpoint /chat/completions and standardizes input/output for 50+ LLM models + handles logging, error tracking, caching, streaming What can liteLLM proxy do? - It’s a central place to manage all LLM provider integrations - Consistent Input/Output Format - Call all models using the OpenAI format: completion(model, messages) - Text responses will always be available at ['choices'][0]['message']['content'] - Error Handling Using Model Fallbacks (if GPT-4 fails, try llama2) - Logging - Log Requests, Responses and Errors to Supabase, Posthog, Mixpanel, Sentry, Helicone - Token Usage & Spend - Track Input + Completion tokens used + Spend/model - Caching - Implementation of Semantic Caching - Streaming & Async Support - Return generators to stream text responses You can deploy liteLLM to your own infrastructure using Railway, GCP, AWS, Azure Happy completion() !

Comments

Post a Comment

Popular posts from this blog

New Show Hacker News story: Show HN: A local Python prototyping tool for Jupyter and Streamlit

Show HN: A local Python prototyping tool for Jupyter and Streamlit 4 by galenmarchetti | 0 comments on Hacker News. Hey HN! I built a local Python prototyping tool that is finally the Python development environment I've always wanted. It has a Jupyter notebook for data crunching, a database of your choice (Python or MongoDB), and a Streamlit app for building a frontend visualization. You can edit the Streamlit backend via an embedded VSCode editor, or locally on your own IDE. The best part for me is that the database connectors within Jupyter and Streamlit are configured out-of-the-box, so you don't need to spend time thinking about how to tie all that together - you can just pick the database you want to use and get going. Disclaimer: I do also work on the tool that deploys all this under the hood, but this project is a personal hackweek project that I threw together so I could develop Python apps on my own
Read more

New Show Hacker News story: Show HN: Natural language Twitter search using Codex

Show HN: Natural language Twitter search using Codex 3 by andyk | 0 comments on Hacker News. We built a structured search engine for Twitter called Bird SQL, available at https://ift.tt/7BU0hHb . Our search interface uses OpenAI Codex to translate natural language to SQL. Our backend then verifies the SQL, executes it, and displays the results on the web app. This makes large structured datasets like a scrape of Twitter easy for anyone to explore. As background, while working on text-to-SQL as a general problem, we came to believe one of its most powerful applications is as a search tool because: - SQL is hard to write by hand and prone to errors - It allows you to iterate quickly if you’re exploring a new dataset - A lot of contextual information that you’d normally have to internalize (e.g. your data’s schema) can be automatically generated and offloaded to the language model Using large language models (LLMs) like Codex to write the SQL for you means you don’t have to worry about...
Read more