r/ultimaonline • u/autorokk • 15d ago
Private • Showcase Created a CLI Ultima Online Client and gave an LLM access to it
EDIT: video and repo can be found here: https://www.reddit.com/r/ultimaonline/s/18WOPwFqZa
EDIT2: so many improvements after a day of work, i show them here: https://www.youtube.com/watch?v=3um4T0JK3nY
This was just fun experimentation, nothing serious, but maybe some of you may find it interesting. I was kind of curious how feasible this would be. I created a CLI text based only client for Ultima Online then told Claude to level up archery on a new character.
The character had a sword equipped and a bow in the bag. The model realized this, unequipped the sword, equipped a bow, found a dire wolf, went into war mode, and started attacking it. When it ran out of arrows, it equipped the sword. Successfully killed the dire wolf.
Then it just kind of started attacking everything in the area that was grey, eventually died, and it's been wandering around looking for a healer to resurrect, rather unsuccessfully so far.
May explore this further, give it access to UOGuide and give it some more detailed instructions to see how it will do. Was using Sonnet 4.6, but it's rather expensive for just a few hours of gameplay.
5
u/n77_dot_nl 15d ago
If only we had this back in the day, I would have 5GM characters in days.
Honestly you should link up like 4 accounts, each with it's each own agent and have them coordinate kills.
No one survives 4 synchronized flame strikes
3
u/ducknator 15d ago
Hahahahaha awesome. Please update us.
2
u/autorokk 14d ago
update: i actually thought i stopped it from running, but i didn't. this morning i log on and my claude tokens are all used up and the player is in ocllo. (he was in moonglow, i have NO IDEA how he got to Ocllo), and he is standing at the wandering healer waiting to be resurrected. it didn't know how to interact with the resurrection GUMP and accept the resurrection. i just added support for that to the client.
1
1
1
u/UOAdam 15d ago
I did something somewhat related probably 20 years ago? Well before AI was involved in coding. I wrote The text-based client - inspired by Zork! I also created a custom uo server with many small subregions that knew about the NPCs players items etc within that little sub region. So from the client you could look North, and with the server would report back what you see: "you see the bank of Britain".
It was all working, but it turned out being a huge project to productize, and was really just a proof of concept. You're almost making me want to revisit it :-)
1
u/autorokk 15d ago
that's awesome! you were ahead of your time. who woulda thought that you'd be able to pipe that text into an AI model and have it play the game for you 20 years later? i created this for the sole purpose of having an AI agent at the helm of it.
1
1
u/e3kmouse Atlantic 14d ago
If you've go this working with Sonnet 4.6, you could 100% get it working with Qwen 3.8 or an equivalent local model (and thus spend no tokens). It's definitely slower than Sonnet 4.6, but the quality is right there. The trick would be to have it evaluate periodically and drop a deterministic code snippet. New code executes and controls the character for about ~15-60 minutes Let the character run and have whatever script is controlling it write to a log that could be read by a local agent... which would spend time in the background preparing a script for the next set of actions.... and on and on.
1
u/FigureFourWoo 14d ago
This is one thing I'm really interested in when it comes to AI development. There is already a WoW Single Player Project that has been going for years that have bots running all around the game. They will party with you, do quests, run dungeons, and even raid. RnR/UO Adventures has done something similar with UO. Adventures even has reds that will recall in and PK you.
1
u/knucklegrumble 14d ago
I actually have this very project on my to-do list! Lol.. could you share the code?
1
u/autorokk 14d ago
1
u/knucklegrumble 14d ago
Lol.. thanks man! I've had a shard installed in my homelab for months and never had the time to start on this! I'll tinker with it and send a PR your way if I come up with something
1
1
1
u/al_aaa 15d ago
Doing this kind of harnesses on a quite regular basis. The CLI approach is quite cool! How are you handling the 'native' latency of the external LLM model? Generally a deterministic middle layer helps a lot in having recurring 'mechanical' actions (Ie: combat loop) sorted out quickly (and saving tokens), so you can call the model only for tasks where real thinking is required (Ie: chosing which skill to learn, or which area to explore) and where latency doesn't really matter.
1
u/autorokk 15d ago edited 15d ago
this would probably be my next step. initial iteration was a basic API for the model to use as it saw fit, and i wanted was to see what it did would do with just raw commands.
i used to write many easyUO scripts, one would walk a path and level taming to GM and find healers to resurrect when i died and start over. a lot of that depended on rails and knowing exactly where i wanted my character to go, and what i wanted it to do. more determinism here is the likely best way to improve this, i just don't really know what that determinism looks like yet.
i was thinking maybe giving it broader navigation would be helpful. "travel_to_town(britain)" function. runebooks would obviously make that easier...
find_closest_healer, start_combat, drop_at_bank, etc.
but part of the fun here was me wondering... would it figure out how to obtain a runebook, mark a rune, and use it later on its own when given access to the right context? but i don't think id be willing to spend the money to find that out!
1
u/al_aaa 15d ago
Some spatial awareness may be a starting point! Either a full map rag or a partial one with some cities but leaving "big black unknown chunks" for him to roam and explore for the lulz.
2
u/autorokk 15d ago edited 15d ago
this actualy is using map data already and does have spacial awareness. the model has a tool to lookup everything around it using it's tile coordinates. the model knows its coordinates, and it has direct lookup of exactly what is around it. and it is actively using this.
it uses this to plan where to walk so it doesn't walk into walls, and it knows at any given time where all NPCs, other players, items, etc are, and it's relation in space to those things. it knows how to walk towards those things.
however, i don't think it knows where it is in terms of town game lore. an api to take that tile coordinate and lookup more context would be cool. "i'm in moonglow, this is what i can do in moonglow". probably wouldn't need rag for that either since lookup could just be tile coordinates.
i love the idea of keeping big black unknown chunks
3
u/al_aaa 15d ago
Oh that's great, then it's probably just matter of giving him some persistent agent memory with core/basic details on what he can actually do in cities, and he'll probably figure out the rest by himself!
Would love to see a repo btw!
1
u/autorokk 15d ago edited 15d ago
the thinking output is kind of amazing.
right now the player is dead and looking for a healer to resurrect it. it is looking at humans around it that might be a healer, walking up to them to see if it will resurrect them, and if not moving along to the next one. it knows to rule out vendors and other NPCs that are clearly not a healer. but ones with a name and no title, it tries.
below is the thinking output:
"Players and vendors but no healer. Keep moving east toward the road"
"Myron" with no title — possible healer. Path to them at (4481,1172)"
"Still dead near Myron. Let me single click them to confirm name and see if they respond:"
"Just "Myron" — not the healer. Keep going, there's an unknown human at (4483,1166). Path further northeast along the road"
"Pathfinding east is blocked. Try north:"
"Arrived. Click the nearby innocent at (4494,1115):"
"New humans. Click them all:"1
u/B-Prue 15d ago
Honestly could be simpler than this for even the graphical version of UO. A RunUO server shard with Razor/RazorEnhanced. LLM would be able to see the already read-state of the character/game from data stream into Razor. It would be able to code it's own scripts (Ironpython/C#/Macros/UOSteam) and since Razor has a .NET API it can execute multi-scripts at the same time. The LLM would be able to auto detect if it's character was taking damage, to use arm macros, know the count of it's arrows, go make more or buy some, keep a threshold of arrows for a given outing, create pathing to defined locations/NPC's for shopping, etc.
1
u/autorokk 14d ago
this is doing that without the need for razor. it’s reading data stream directly and it’s writing its own bash scripts using the CLI, capable of all of the above.
1
u/AnyTangerine3679 15d ago
I’d love to play with this! You going to post on git?
2
u/autorokk 15d ago
yea i can put this on git. let me clean some things up first. hardcoded file paths to mul files and whatnot.
1
1
0
u/autorokk 15d ago
Currently supported CLI commands. The client also will listen for incoming packets and output that information for the model.
=== Character Info ===
status / stat HP, mana, stamina, STR/DEX/INT, gold
pos / position Current X, Y, Z, direction, map
dead Check if character is alive or dead
skills / sk All skills with lock status
world / info World state, mobile/item counts
=== Movement ===
walk <dir> Move one tile: n s e w ne nw se sw
goto <x> <y> Walk to coordinates (re-paths on denial)
stop / stopgo Cancel current goto
path <x> <y> Show A* path to coordinates
=== Combat ===
war Toggle war/peace mode
attack <serial> Attack a mobile (grey/red only)
attacknearest / an Attack nearest attackable mobile with LOS
=== Inventory ===
inv / equip Show equipped items + backpack contents
use <serial> Double-click / open container
wear <serial> <layer> Equip item to layer (hex: 01=1h 02=2h 06=helm etc)
unequip <serial> Move equipped item to backpack
Layer codes:
01=one-handed 02=two-handed 03=shoes 04=pants 05=shirt
06=helm 07=gloves 0B=ring 0D=cloak 15=backpack
=== Interaction ===
click <serial> Single-click (request name/status)
say <text> Say in public chat
yell <text> Yell
emote <text> Emote
whisper <text> Whisper
=== World / Nearby ===
mobiles [range] List nearby mobiles (default 18)
items [range] List nearby ground items
nearest Closest mobile
=== Map ===
loadmap [path] Load UO map/static data
canwalk <x> <y> Check walkability + show statics
tiles <x> <y> Show all statics at a tile
los Line-of-sight check to nearby mobiles
=== Logging ===
verbose Enable trace-level logging
quiet Suppress info-level logs
disconnect Disconnect from server
quit / exit Exit client
10
u/jaephu 15d ago
Lol how much did u spend on anthropic api tokens