I’ve been tinkering a bit with large language models small enough to run locally on my M1 MacBook Pro without having to reach out across the internets. Local LLMs means not having to send anything across that might be collected, and thus losing control on whatever I type in (not that whatever I type in is worth collecting…).

You can go look at GPT4All here: https://gpt4all.io/index.html

The application, when first installed, can’t do anything. You need to begin to add LLMs via a curated selection it presents. These are the eleven currently listed.

1. Wizard V1.1
2. GPT4All Falcon (the one I’m currently using)
3. Hermes
4. ChatGPT-3.5 Turbo
5. ChatGPT-4
6. Snoozy
7. Mini Orca
8. Mini Orca (small)
9. Mini Orca (large)
10. Wizard uncensorted
11. Llama-2-7B Chat

I’ve barely scratched the surface, having just installed the application and then selected a given model. My criterial right now for selecting a model is that there is no link back to the cloud and that it can run in the “limited” 16 GiB of memory on my MacBook. GPT4All Falcon, for example, will run in 8 GiB.

If you stop and think about the resources required by Falcon it suddenly becomes apparent that it could run on an Apple iPhone if the iPhone had enough memory. Today’s contemporary iPhones certainly have the processor horsepower and more than enough local storage to hold such a model. The problem is that the iPhone 15, Apple’s latest device, doesn’t. Perhaps the iPad Pros running with either the M1 or M2 has enough, but I know the iPhones do not.

Why would I want that much memory? To run one of the LLMs locally. And why do that? So that the LLM could “learn” about me from all the data that passes through my iPhone from me, and help the iPhone to be an even better aid. Smartphones, especially the iPhone, has long since passed the point of “good enough” to perform such tasks. The big question is how much power will that require, because there are always trade offs with portable computers. The more hardware you have on an iPhone, such as more memory, the more power the device will consume. The more processing the CPUs on the iPhone needs to perform, the more power the device will consume. As the saying goes, be careful what you ask for.

And there’s another reason for looking at this for local processing, and that’s in support of the intelligent home. My wife and I grow progressively older. We’re both in our early 70s. I would like a home with sensors and microphones and speakers to be able to assist our living here, rather than have to go live in a classical assisted living facility. And I do NOT want devices that require cloud connectivity and a monthly fee. Right now I have an intelligent [sic] doorbell which is absolute shit. I want far better, and I want local autonomy.

I haven’t even scratched the surface with any of this software. But it’s now approachable and I can run it locally on my MacBook, which I find absolutely amazing. I hope I can do more with it going forward.