Software for LLM Effectiveness
Large Language Models (LLMs) are great when their needs are met:
Need 1. Can consistently get relevant, well-formatted context for their inference
Need 2. Can consistently output text w.r.t. an input schema
Need 3. Are offered a sandboxed code execution environment
Need 4. Are subject to human review; minimizing OODA loop here
LLMs can be useful for multiple kinds of services:
Service 1. Data Labelling
Service 2. Code Writing
Service 3. Code Execution / Process Control
Service 4. Low-consequence Q&A / Summarization
We can develop software to support these use-cases. This list covers interesting things but is neither comprehensive or completely forward looking. Some of these things exist already.
Low-Level Software
Software that serves a minimal set of requirements to enable LLMs to do more than just chat. Not to be confused with actual low-level/systems software. This software serves the LLM.
Webpage to Markdown
Serves: Need. [1]
Examples: embedding.io
Constrained Sampling / Structured Extractions
Serves: Need. [2]
Examples: Outlines, Instructor
Code Sandbox
Serves. Need. [3]
Examples: ...
Consider a WASM sandbox that you provide to an LLM. Consider providing an LLM a REPL.
Human Eval as a Service
Serves: Need. [4]
Software
Software that addresses a human need for services rendered.
ChatGPT-like Chat
Serves: Service. [1,2,3,4]
Needs: Need. [3]
Webpage to Markdown, Code Sandbox
Chat is a great catch-all interface. It has some issues on throughput mainly. Additional annoyances on establishing context.
Programming Assistant Chat
Serves: Service. [2]
Needs: Need. [1,3]
Webpage to Markdown, Code Sandbox
Normal chat is not great. I'd like to check a few boxes for "I use this library" then get dropped into a chat interface. This can be serviced by OpenAI GPTs, Claude Projects, Gemini Gems, but I don't want to have to update my docs whenever a library is updated. Can this not just be automated for me? These tools are so close to being there.
Fast Data Labeler
Serves: Service. [1]
Needs: Need. [2]
I don't want to write code every time I want to do something like label(input data, structure, example labelings). This Gemini structured prompt interface is so close. If i could freeze this and execute it against a google sheet, that would be great.
Unfortunately gemini is pretty bad at this labelling task.