Language models are all the rage, and many people are just taking foundation models (most often ChatGPT or something similar) and then connecting them to a vector database so that when people ask their \u201cmodel\u201d a question, it responds to the answer with context from this vector database.<\/p>\n
What is a vector database<\/a>? I\u2019ll explain that in more detail in a future essay, but a simple way to understand it is as a collection of information stored as chunks of data, that a language model can query and use to produce better responses. Imagine \u201cThe Bitcoin Standard,\u201d split into paragraphs, and stored in this vector database. You ask this new \u201cmodel\u201d a question about the history of money. The underlying model will actually query the database, select the most relevant piece of context (some paragraph from \u201cThe Bitcoin Standard\u201d) and then feed it into the prompt of the underlying model (in many cases, ChatGPT). The model should then respond with a more relevant<\/em> answer. This is cool, and works OK in some cases, but doesn\u2019t solve the underlying issues of mainstream noise and bias that the underlying models are subject to during their training.<\/p>\n This is what we\u2019re trying to do at Spirit of Satoshi. We have built a model like what\u2019s described above about six months ago, which you can go try out here<\/a>. You\u2019ll notice it\u2019s not bad with some answers but it cannot hold a conversation, and it performs really poorly when it comes to shitcoinery and things that a real Bitcoiner would know. <\/p>\n This is why we\u2019ve changed our approach and are building a full language model from scratch. In this essay, I will talk a little bit about that, to give you an idea of what it entails.<\/p>\n The mission to build a more \u201cbased\u201d language model continues. It\u2019s proven to be more involved than even I had thought, not from a \u201ctechnically complicated\u201d<\/em> standpoint, but more from a \u201cdamn this is tedious\u201d<\/em> standpoint.<\/p>\n It\u2019s all about data.<\/em> And not the quantity of data, but the quality and format of data. You\u2019ve probably heard nerds talk about this, and you don\u2019t really appreciate it until you actually begin feeding the stuff to a model, and you get a result\u2026 which wasn\u2019t necessarily what you wanted.<\/p>\n The data pipeline is where all the work is. You have to collect<\/em> and curate<\/em> the data, then you have to extract<\/em> it. Then you have to programmatically clean<\/em> it (it\u2019s impossible to do a first-run clean manually). <\/p>\n Then you take this programmatically-cleaned, raw data and you have to transform<\/em> it into multiple data formats<\/em> (think of question-and-answer pairs, or semantically-coherent chunks and paragraphs). This you also need to do programmatically, if you\u2019re dealing with loads of data \u2014 which is the case for a language model. Funny enough, other language models are actually good for this task! You use language models to build new language models.<\/p>\n Then<\/em>, because there will likely be loads of junk left in there, and irrelevant garbage generated by whatever language model you used to programmatically transform the data, you need to do a more intense clean<\/em>.<\/p>\n This<\/em> is where you need to get human help, because at this stage, it seems humans are still the only creatures on the planet with the agency necessary to differentiate and determine quality<\/em>. Algorithms can kind of do this, but not so well with language just yet \u2014 especially in more nuanced, comparative contexts \u2014 which is where Bitcoin squarely sits.<\/p>\n In any case, doing this at scale is incredibly hard unless you have an army of people to help you. That army of people can be mercenaries paid for by someone, like OpenAI which has more money than God<\/a>, or they can be missionaries, which is what the Bitcoin community generally is (we\u2019re very lucky and grateful for this at Spirit of Satoshi). Individuals go through data items and one by one select whether to keep, discard or modify the data.<\/p>\n Once the data goes through this process, you end up with something clean on the other end. Of course, there are more intricacies involved here. For example, you need to ensure that bad actors who are trying to botch your clean-up process are weeded out, or their inputs are discarded. You can do that in a series of ways, and everyone does it a bit differently. You can screen people on the way in, you can build some sort of internal clean-up consensus model so that thresholds need to be met for data items to be kept or discarded, etc. At Spirit of Satoshi, we\u2019re doing a blend of both, and I guess we shall see how effective it is in the coming months.<\/p>\n Now\u2026 once you\u2019ve got this beautiful clean data out the end of this \u201cpipeline,<\/em>\u201d you then need to format<\/em> it once more in preparation for \u201ctraining<\/em>\u201d a model.<\/p>\n This final stage is where the graphical processing units (GPUs) come into play, and is really what most people think about when they hear about building language models. All the other stuff that I covered is generally ignored.<\/p>\n This home-stretch stage involves training a series of models, and playing with the parameters, the data blends, the quantum of data, the model types, etc. This can quickly get expensive, so you best have some damn good data and you\u2019re better off starting with smaller models and building your way up.<\/p>\n It\u2019s all experimental, and what you get out the other end is\u2026 a result\u2026<\/em><\/p>\n It\u2019s incredible the things we humans conjure up. Anyway\u2026<\/p>\n At Spirit of Satoshi, our result is still in the making, and we are working on it in a couple of ways:<\/p>\n You can search for anything there and access the URL, text file or PDF. If a volunteer can\u2019t find something, or feel it needs to be included, they can \u201cadd\u201d a record. If they add junk though, it won\u2019t be accepted. Ideally, volunteers will submit the data as a .txt file along with a link.<\/p>\n<\/li>\n This is a way for people who have spent years learning about and understanding Bitcoin to transform that \u201cwork\u201d into sats. No, they\u2019re not going to get rich, but they can help contribute toward something they might deem a worthy project, and earn something along the way.<\/li>\n<\/ol>\n In a few previous essays, I\u2019ve argued that \u201cartificial intelligence\u201d is a flawed term, because while it is<\/em> artificial, it\u2019s not<\/em> intelligent \u2014 and furthermore, the fear porn surrounding artificial general intelligence (AGI) has been completely unfounded because there is literally no risk of this thing becoming spontaneously sentient and killing us all. A few months on and I am even more convinced of this.<\/p>\nA More \u2018Based\u2019 Bitcoin Language Model<\/h2>\n
![](\"https:\/\/bitcoinmagazine.com\/.image\/c_fit%2Ccs_srgb%2Cfl_progressive%2Cq_auto:good%2Cw_620\/MTk5OTM5MTgwOTI5ODIwMjg4\/language-models-all-the-way-down.jpg\")
<\/p>\n<\/figure>\n\n
Probability Programs, Not AI<\/h2>\n
![](\"https:\/\/bitcoinmagazine.com\/.image\/c_fit%2Ccs_srgb%2Cfl_progressive%2Cq_auto:good%2Cw_620\/MTk5OTM5MTgwOTI5NzU0NzUy\/awake-yet.jpg\")
<\/p>\n<\/figure>\n