This post discusses a collection of new tools that I recently wrote and released on OSRSBox (website and GitHub). The OSRS Item Search tool is a web-based application to search the item ID numbers and item names that are available in OSRS. Check out the image below to get an idea of what the tool looks like:
My OSRS item search tool fills a much-needed void. When starting doing some OSRS software development, I found that there were no up-to-date item ID databases or search tools. There were some solutions floating around the Internet, but none of them had game items that were recently added. This raised a big question in my brain… Why are there no up-to-date item databases? Potential answer laziness of web devs? More probable answer: bad workflow for dumping the OSRS cache and updating databases. This leads me on a path to find a decent solution to providing a consistently up-to-date item search tool.
I split the project into two primary categories:
In addition, the tools and general workflow were designed to intergrate (and populate) my existing OSRSBox Database project.
Is it just me, or does RuneLite seem to be the current answer for any OSRS problem?! After some serious reading and research about OSRS cache tools, I went full circle and ended up back with RuneLite. I had previously written a blog post about using Runelite to extract NPC and item definitions. This post culminated with a solution to extract the
ItemDefinitions structure from the OSRS cache but had no support for image extraction. Also, my build process was ridiculously complex and the compiled code,
.jar files were messy to run. My ideal solution: a RuneLite plugin to extract metadata about every OSRS item!
I always start a project (even an informal project) with a list of requirements for what the plugin needed to do! This lead to the following functional requirements:
Before doing anything, I started looking for a JSON library that could be integrated easily into the RuneLite build process. I found lots of solutions that involved importing
.jar files - which is fine, but not exactly scalable to different development environments (read: different computers that I, or anyone else, wants to compile RuneLite on). It seems like a Maven dependency was the best solution.
I found a decent JSON library called json-simple. However, finding the best solution to use as a Maven import was difficult. There were license issues with one version I initially tried (so I was told in a blog post). Then I found a fork of the original project supported by Maven, named JSON.simple. After finding the right library, the import was relatively easy. I simply added the dependency to the
pom.xml file for the
runelite-client. To be specific, this involved editing the following file:
In this file, I added the XML dependency object into the
dependencies XML tag. The code listed below documents my additions.
Sometimes explaining where exactly to edit code can be difficult. If you want more information, you can view my modified
pom.xml file in my forked RuneLite repository (under the
osrsbox-plugins branch). This file has the additions that I have made to support JSON generation.
After the addition to the
pom.xml file, you can import the required components of the library into your plugin file. I only used two imports, one for creating JSON objects and one for creating JSON arrays.
This was the last step to modify the RuneLite client to support JSON generation. Later in the article, I briefly discuss how to actually use the JSON Simple library to create JSON objects containing item metadata.
As with any RuneLite plugin, I started by picking a name: Item Scraper. This seemed descriptive! As I have discussed in previous blog posts, I then made a package in the
plugins folder named:
itemscraper. Then I made two files to start plugin development:
You can view all the code and the structure I used by looking at my
orsrsbox-plugins branch of my forked version of the official RuneLite repository on GitHub. The base files for the itemscraper plugin has the two files used to create the plugin.
I started by setting up a base plugin structure with the mandatory imports (writing this I think someone, maybe me, should author a base plugin for starting plugin development - let me know in the comments if you are interested in this). Anyway, the first thing to do is determine when and how to start the Item Scraper plugin. All plugins usually have a
startUp() function that is run when a plugin is started. The code to extract item metadata could be included here, but I thought that it might be better to have an action that can be triggered to run the plugin. The idea of using a command seemed like a good idea, which I ended up going with.
Even the OSRS vanilla client supports a variety of commands. For example, the
::renderself command can be used to remove visibility of your OSRS character in-game. There are a bunch of other commands and even chat effect commands. RuneLite itself uses commands in a variety of plugins. For example, check out the Chat Commands Plugin which can use commands such as
According to the DevToolsPlugin class, there is an
onCommand() function that listens to entries in the chat box. Any chat that is entered and contains two colons (
::) can be caught and handled in a RuneLite plugin. I wrote a simple function based on other plugins to listen to chat commands, specifically, my implementation listens for a
::dump command and then executes my custom
dumpItems() function. An example of my
onCommand() method is provided below.
Notice how the
case statement (similar to switch for non-Java people!) looks for the
dump command, then runs a function called
dumpItems() … this is the function where the magic happens! Well, it is not very magical, it is more logical!
After adding a new
dumpItems() function, I started to write the code to dump the actual items. In OSRS, every item has a unique ID number, so the first thing is to loop every item ID and then process each item. However, I could not find any RuneLite API methods to determine the total number of item ID numbers or to provide a list of valid item ID numbers. Furthermore, any list would be dynamic and would be extended every game update. I went with a dirty hack and wrote a simple
for loop to start from
0 and iterate through to the highest item ID number. To find the highest item ID value I looked at the ItemID.java file from RuneLite which contains a list of all items. I decided to create a loop from 0 to 25,000 to ensure complete coverage.
Fetching the item information is pretty easy thanks to RuneLite’s API. There is an
ItemComposition interface which populates a template with metadata about any item.
In addition to performing a check to see if the
null, I also checked the item name for
"" (an empty string). This was to remove any item ID numbers that were not valid. The RuneLite API documents the ItemComposition methods, which can return a variety of useful information. Check the API documentation for a complete list - but some of the methods I used are listed below:
getID(): Gets the item’s ID (we already have this)
getName(): Gets the items name
getNote(): Gets a value specifying whether the item is noted
getPrice(): Gets the store price of the item
isMembers(): Checks whether the item is members only
isStackable(): Checks whether the item is able to stack in a players inventory
isTradeable(): Returns whether or not the item can be traded to other players
getInventoryActions(): Gets an array of possibilities of right-click menu actions the item has in a player inventory (useful to determine if an item is equipable)
From here, it was as simple as creating a JSON object and populating it with item metadata. The code snippet below displays the process of creating a new
JSONObject then populating it with the item ID and name.
I used the methods listed earlier to populate a variety of different metadata properties about each item. Basically trying to populate as much useful information as I could get from the ItemDefinition methods provided by the API. The next logical thing is to save the actual JSON file. The plugin saves all JSON files to a folder named
items-json-runelite in the root
runelite folder which has to actually exist (yes… probably should have added support to create this folder on run-time - hopefully in the future). It will save the output file to
<itemIDnumber>.json, for example:
1243.json. The code snippet below documents how I saved the file, but it is pretty much the same for any other Java application.
The end result from this process is approximately 21,000 individual JSON files. One for every item in OSRS! Below is an example entry for the
Cannonball item with the ID number of
In addition to the individual JSON files (one for each item), I also added support to generate a
summary.json file that contains every item name and item ID number. This is the file that I use for the actual item search web application. For interest sake, this file is available from:
The final plugin requirement was to save the item icon image in PNG format. I found a method in the
ItemManager class that had a
getImage() function to fetch the icon image and return it as a
BufferedImage type. Again, the plugin saves all PNG files to a folder named
items-icon-runelite in the root
runelite folder (which has to exist).
Saving the image was the final part of the RuneLite plugin design. The full source code of the ItemScraperPlugin.java is provided on my GitHub repository. It is much easier to read the code in completed format - so make sure to have a look.
After getting an up-to-date list of item ID numbers and names, it was possible for me to make an item ID and item name search tool using the summary.json file. Aside from an up-to-date item list, it was important that the item ID and item search was fast and responsive. I am never a fan of re-inventing the wheel, so the first objective was to find a decent library for building a table solution to handle the following requirements:
The only issue I faced was that when loading the initial table using DataTables, the item icon images crashed the web browser. I was, after all, making over 20,000 GET requests for PNG images. I discovered a simple solution on the DataTables forums (sorry, I have lost the link) to fake lazy loading of the images. Problem solved! Have a look at the
item-seach.js file if you are interested in the code.
I hope you found this an entertaining and informative blog post. Looking back, this project was somewhat different from regular RuneLite plugin development like creating overlays - but I thought it was an interesting journey to document. Hopefully, you have found some of the techniques and thought process interesting!
As always please leave any feedback or comments below. Always happy to hear from other avid OSRS players and developers. See you all next time!