Rummaging through some boxes of old stuff, I found one of my old smart phones. It's an Asus Zenfone 4 Max (ZC554KL, Snapdragon 430, 8x Cortex-A53, 3 GB RAM, 32 GB storage) that I bought back in 2017 and so it's been sitting in a box for ~8 or 9 years. Still works! So I thought I'd give it a chance to avoid the garbage by turning it into a tiny AI chatbot server that can connect to any wifi network and immediately make itself available to chat via a web UI on the local network.
The phone now boots into postmarketOS, with a small touch screen UI for picking a Wi-Fi network. Once connected, the web UI is started and a chat URL provided (the app sets its hostname to ittybitty so the standard chat URL will be http://ittybitty.local or just the IP address) and a QR code that will take you to the chat UI.
llama.cpp was compiled on the phone itself (the first full build took over an hour). The model is Qwen2.5-0.5B-Instruct at 4-bit (403 MB).
It can't really do a whole lot other than chat and answer questions using it's training data, which is tiny so not super-intelligent on its own, but I added a few helpers to the web page (a calculator, the date/time, Wikipedia lookups). This project was mostly just to see if it would work at all, the results were relatively impressive.
What I'm seeing:
- ~4.5 tokens/sec generating (generally faster than you can read the output being generated), ~13 tokens/sec reading the prompt
- llama-server uses about 570 MB of RAM with the model loaded
- two people can chat at once, users wait their turn
- I tried SmolLM2-360M first. It was faster (6 tok/s) but it looped and refused to answer things. The 0.5B model is noticeably better at holding a conversation.
The coolest thing is it's portability. Anywhere you go that has a wifi to connect to you can just turn the phone on, connect it to the wifi, and now that network has a little AI chat bot to talk to. If this old crappy smart phone does this well I can only imagine how newer, better, faster, bigger phones would do.
If anybody has this same smart phone and is interested in doing the same thing just let me know and I can give you the custom OS and llamacpp builds to start from.
Not a whole lot to 'show' since it runs inside my local network but here are some screenshots of the mobile app/server and the web UI:
The phone now boots into postmarketOS, with a small touch screen UI for picking a Wi-Fi network. Once connected, the web UI is started and a chat URL provided (the app sets its hostname to ittybitty so the standard chat URL will be http://ittybitty.local or just the IP address) and a QR code that will take you to the chat UI.
llama.cpp was compiled on the phone itself (the first full build took over an hour). The model is Qwen2.5-0.5B-Instruct at 4-bit (403 MB).
It can't really do a whole lot other than chat and answer questions using it's training data, which is tiny so not super-intelligent on its own, but I added a few helpers to the web page (a calculator, the date/time, Wikipedia lookups). This project was mostly just to see if it would work at all, the results were relatively impressive.
What I'm seeing:
- ~4.5 tokens/sec generating (generally faster than you can read the output being generated), ~13 tokens/sec reading the prompt - llama-server uses about 570 MB of RAM with the model loaded - two people can chat at once, users wait their turn - I tried SmolLM2-360M first. It was faster (6 tok/s) but it looped and refused to answer things. The 0.5B model is noticeably better at holding a conversation.
The coolest thing is it's portability. Anywhere you go that has a wifi to connect to you can just turn the phone on, connect it to the wifi, and now that network has a little AI chat bot to talk to. If this old crappy smart phone does this well I can only imagine how newer, better, faster, bigger phones would do.
If anybody has this same smart phone and is interested in doing the same thing just let me know and I can give you the custom OS and llamacpp builds to start from.
Not a whole lot to 'show' since it runs inside my local network but here are some screenshots of the mobile app/server and the web UI:
https://spottedmarley.com/ittybitty-home.png
https://spottedmarley.com/ittybitty-ui.jpg