TUTORIAL: Using the Splinterlands API and KNIME to determine what cards you require to complete your untamed collection

avatar
(Edited)

image.png

About me


I've been playing Splinterlands since 2018.

I love data. I love crunching it. I love looking at it, storing it, and making decisions based on it. In my day job, I use a tool called KNIME which is an analytics tool that can transform and crunch data in any way shape or form. KNIME is freely downloadable, and available on all major operating systems.

About this tutorial

This tutorial is lengthy and advanced in nature. If you're a developer, you likely won't gain anything from this. If, however, you are an Excel junkie and have not yet graduated into the realm of data science and analytics, KNIME is a tool that can certainly take you to the next level. You'll learn about KNIME basics and apply it to something fun - Splinterlands cards.

Using Knime

Let's begin by downloading knime.

Once it is downloaded, install the software.

Run the software, set a directory to use as Knime's "scratch", and you're ready to go.

image.png

Knime has a variety of Windows. The top left is your explorer, where different KNIME flows live in your working directory. Here is also where you will find example analytical models from the community to learn how to do various things in KNIME.

The next window down is a workflow coach which helps you out with what sort of nodes can be linked together.

What are nodes? Nodes are blocks that perform operations on your data. They may modify it, search for something, filter, or do any of the other common operations you may want to do when working with large, small, or unfamiliar data sets. You can also perform a number of standard SQL functions like joins, which are incredibly useful.

The Node Library has hundreds of plug ins, and you can even drop R, Python, or JavaScript snippets into KNIME to perform operations for which there may not be a node. You've then got the convas in the middle of the screen, which is where you will do most of the work in KNIME.

KNIME sees you link a series of nodes to perform operations or analysis on data, and the processed result is then accessible at the end of the flow. For the rest of this tutorial, I will explain what I have created which can be built upon further in the future. The creation I've produced is indeed a minimum viable product, and just one of the many applications you could perform on Splinterlands card data.

Building the analytical flow

In order to determine the answer to the query: "How many regular foil cards do I require to max out my Untamed deck?" we need to know a number things.

  1. What cards do I as a player have?
  2. What cards exist in Untamed?
  3. What is the number of cards required to max out my deck?
  4. What is the difference between what I have and the max level cards?

It is important to know everything you need before starting any sort of analysis work - or you may have to start from scratch all over again. In my day job I make sure that parameters and scope is set before I start my research in order to avoid complexities in future iterations of reporting or analysis which means prior data sets are invalidated by iterative improvement (or changing) business requirements.

Here, we've got a fairly flat set of requirements - so let's get started.

Thanks to @kiokizz's excellent Splinterlands API documentation, a lot of the information can be extracted direct from the game's API.

What cards do I have as a player?

This is the first part of the puzzle. The Splinterlands API responds using a data format called JSON. Thankfully, Knime can import JSON data structures as well as CSV, Excel files, or even connect directly to a SQL database!

I create a JSON Reader Node by searching for JSON reader in the node repository window, and dragging it onto the canvas. I then right click to configure the node.

image.png

Right clicking the node gets me:

image.png

The next step is decoding and tabulating the JSON data to individual rows that can then be analysed. Otherwise, the response is a lengthy single table cell which lists all the individual cards I have:

image.png

Now, I don't need all the information about the card in order to answer my query - so I create a list of JSON paths (ie the variable structures) that I want to focus on. Think of this as a filter to only grab the variables and information that are relevant to the question I'm asking - which is "How many cards do I need to max out my Untamed regular foil collection?"

image.png

I need to know the following:

  • uid

This is the unique ID for a card.

  • card_detail_ids

This is the "serial number" for the card - it will tell me what number card it is - more on this later.

  • golds

a boolean value that identifies if the card is a regular, or gold foil

  • xps

The XP (how many cards) have been combined under the uid (or, as we know it, bcx)

  • players

Who owns the card

  • levels

What level the card is at currently

  • editions

What edition the card is - Alpha, Beta, Untamed, Reward, Dice, Orbs, etc

The next operation in Knime is to "Ungroup" the JSON data of my card collection, so I get a list of each and every card that I have.

image.png

This gives me a table structure with each and every card I have. The UID is the card's unique "serial" - or the underlying NFT number. The card_detail_ids tells us what card it is - then if its gold, how much xp, who owns it, what level it is, and from which edition the card came...

image.png

But - it isn't very human readable. For that, we're going to need to use a really cool Knime feature called the Rule engine. Because I can go build a reference table to find out what edition a card is, I can make that "Editions" Column something that is very human readable.

image.png

This results in the previous table having a new column added:

image.png

However, we still don't know what card is what just by looking. This is where the card_id field will come in handy - but I certainly don't want to go and build a massive table listing each and every card's name, splinter, and other variables, when that already exists in the Splinterlands game API - I just need to pull it using a different part of the API documentation.

This is everything we've achieved so far:

image.png

Now - we'll look at the second part - getting the details of the cards from the API, which is the second line of the screenshot above.

What are the card details?

To get the card details, we need to make a call to the Splinterlands API.

Using the JSON Reader node as we did to get a user's card details, we simply make a different call to the game API:

image.png

The subsequent nodes transform the data into a table that looks a bit more useful...

image.png

image.png

For some reason, some of the number based variables came through as Strings - so I need to convert them to a number in order to further categorise the output, and make the table a bit more human readable.

Determining Rarity:
image.png

Determining the Splinter:

image.png

Next is where the major magic happens - a JOIN.

Joining is a function of SQL and data science / analytics where you take two disparate sets of data with a field that allows an association. In a list of people - this might be their full name. So, you might have a list of full names, and a list of drivers licences and full names, along with a date of birth. You might have another table that has the driver's licence number and the person's traffic infringement history.

image.png

By joining the driver's licence, and full name from the people table to the "fines" table, using the driver's licence number - you can enrich that data to learn about Bob's speeding tickets from the past.

Except here, we're wanting to take all the lovely details we have about Goblin Shamans, Hydras and Tyrus into our own card collection.

The top table is my card collection, as pulled from the API. The bottom table is the card details, as pulled from the API. By joining, or matching on the card details, I get an enriched data set of the cards I have.

image.png

You can see here that there's now card names, edition details, and a whole bunch more. However, this only works for the cards that I DO HAVE. If I'm missing anything from Untamed, it won't appear in this list. In order to figure this out, I need to calculate the difference - or, the cards that DIDNT match - where there's a card I don't have in my collection at all.

This doesn't really have a name for it, but I like how KNIME calls it an "Anti-Join" - this is the data from the secondary table that wasn't matched. I sort of build a cache to store this infomration in temporarily.

image.png

I then split out any gold / non gold cards,after only looking for untamed cards in the collection. That is the "Rule based Row Filter node"

The next step is to COUNT the individual cards I have. This is to determine how many of EACH Untamed card I have, and, if they're combined, how much XP (BCX) they have - to determine further calculations which will enable us to work out how many cards I need to buy to max out my particular card.

This is achieved by the GROUP BY node, which works much like a Pivot Table in Microsoft Excel, except it has dozens of powerful functions and features, and has the added advantage of processing in a fraction of the time.

image.png

I want to group by the individual card.

image.png

Then, I want to injet some meaning into the card...

I want a SUM of the xps - and the rarity and splinter details, so that I can sort it all later, that's the next tab in the Group By configuration.

image.png

The resulting table is as such:

image.png

A list of cards, and the XP, which splinter they belong to, and their rarity. Though these may be a combination of individual (1 BCX) and some combined cards - that doesn't matter, as the only thing I'm interested in is how many more BCX do I need to complete a max instance of an individual card.

The next step is to bring in some figures about what it takes to get a card to max level in the Untamed Edition.

I do this by using a subsequent Rule Engine, which explains the max levels:

image.png

Now of course, as I indiated earlier, this will only work for cards that are in my collection. For those that are not (those in the so called Anti-Join) - I can make the assumption that my owned quantity is zero, and that I therefore need the maximum amount to get them to the max level.

To find what cards I'm missing, I then concatenate the joined cards and the "Anti-Joined" cards (ie, the missing ones, into one tabulated list)

image.png

The output is a bit messy, so I use a few nodes to clean up the data set prior to completing my final calculations...

image.png

You can also see at the bottom of this output exactly which UNTAMED cards I am missing in my collection!

The final clean up is as follows - and I've documented it in my KNIME flow itself, as these are rather simple, explanatory nodes with basic configuration - up until the Math Formula Node. \

image.png

The Math Formula node is where this all comes together, to generate the final tables and calculations of what will be required to complete the collection at Max Level.

image.png

Here's the result of that formula:

image.png

I then tidy up that data to make it even more human readable....

image.png

From here, I have a list of what I need to complete a Max Level Untamed Collection.

Card NameBCX OwnedRaritySplinterCards for MaxCards to Buy / Sell
Albatross485CommonWater400-85
Azmare Harpoonist3EpicWater4643
Beatrix Ironhand47EpicEarth46-1
Biceratops430CommonEarth400-30
Bone Golem115RareDeath1150
Bortus135RareWater115-20
Byzantine Kitty1LegendaryDragon1110
Camila Sungazer12LegendaryDragon11-1
Cave Slug731CommonLife400-331
Chain Golem1LegendaryNeutral1110
Chanseus the Great11LegendaryLife110
Child of the Forest123RareEarth115-8
Contessa L'ament115RareDeath1150
Coral Wraith48EpicWater46-2
Cornealus2LegendaryNeutral119
Crystal Jaguar401CommonLife400-1
Cursed Slimeball49CommonDeath400351
Dark Astronomer130RareDeath115-15
Dark Ha'on3LegendaryDeath118
Darkest Mage46EpicDeath460
Death Elemental115RareDeath1150
Diamond Dragon3LegendaryDragon118
Dragon Jumper11LegendaryDragon110
Drake of Arnak115RareDragon1150
Elven Defender444CommonNeutral400-44
Elven Mystic18RareNeutral11597
Failed Summoner49CommonEarth400351
Feasting Seaweed680CommonWater400-280
Ferexia General7EpicFire4639
Fire Elemental23RareFire11592
Giant Scorpion527CommonDeath400-127
Giant Squid115RareWater1150
Goblin Chariot23RareNeutral11592
Goblin Fireballer34CommonFire400366
Goblin Firemage18EpicFire4628
Goblin Thief632CommonEarth400-232
Herbalist402CommonLife400-2
High Priest Darius11LegendaryLife110
Horny Toad275CommonNeutral400125
Ice Pixie7RareWater115108
Khmer Princess160RareEarth115-45
Kobold Bruiser47CommonFire400353
Kron the Undying2LegendaryEarth119
Light Elemental4EpicLife4642
Lir Deepswimmer2LegendaryWater119
Living Lava20RareFire11595
Lobstradamus46EpicWater460
Lone Boatman400CommonLife4000
Luminous Eagle115RareLife1150
Maggots605CommonDeath400-205
Magma Troll63CommonFire400337
Mantoid47CommonNeutral400353
Mimosa Nightshade2LegendaryDeath119
Mitica Headhunter1EpicEarth4645
Mother Khala115RareLife1150
Orc Sergeant626CommonEarth400-226
Parasitic Growth1042CommonNeutral400-642
Phantom of the Abyss2LegendaryWater119
Pyre115RareFire1150
Pyromaniac13EpicFire4633
Scale Doctor1LegendaryDragon1110
Scarred Llama Mage2LegendaryEarth119
Serpent of Eld16RareWater11599
Serpentine Spy42CommonFire400358
Shieldbearer1RareLife115114
Sniping Narwhal30CommonWater400370
Soulstorm5EpicDeath4641
Spark Pixies133RareFire115-18
Sporcerer3EpicEarth4643
The Vigilator7LegendaryDragon114
Thunderbird48EpicLife46-2
Tortisian Chief23EpicNeutral4623
Tortisian Fighter403CommonWater400-3
Tower Griffin18RareNeutral11597
Truthspeaker127RareLife115-12
Undead Badger477CommonDeath400-77
Unicorn Mustang122RareEarth115-7
War Chaang22EpicNeutral4624
Wizard of Eastwood115RareEarth1150
Yodin Zaku3LegendaryFire118
Magnor0LegendaryFire1111
Gloridax Magus0LegendaryDragon1111
Gloridax Guardian0LegendaryDragon1111

And I'm all done!

If you want me to calculate how many UNTAMED cards you need to MAX OUT your collection, please upvote and leave your username below. When I get some time, I'll generate a report for you!

Future improvements to the above flow can be made by incorporating current market prices, or looking at historical card versions in much the same way - I just need to build the logic. I've got all the cards I (probably) want from prior versions and editions, so I am completely unmotivated to do the work on that currently...

I hope to see you all on the battlefield!

TLDR: Want me to generate a list of cards you need to max out your Untamed REGULAR FOIL collection?

Drop your in-game username below, and please feel free to upvote or leave me a tip!



0
0
0.000
8 comments
avatar

Awesome mate, this is the same app you were showing me last month?

0
0
0.000
avatar

Wow, this is so awesome! I wish I knew how to code! Great post! I miss the old Steem Monster days when battles first started! I'll never forget you telling me.... you just heal, heal, heal! LOL!! I did! I used to put 2 or 3 healers on a tank if I could haha! Have a great weekend! ~@clove71

0
0
0.000
avatar

Cheers @clove71 - I don't have the stones to play in the Champion Leagues any more... but I stick around in diamond these days. :D Now you can put repairman on healers and be even more annoying :D

0
0
0.000
avatar

I'm really interested in this. I'm a retired data scientist/wrangler and this looks like a very cool tool! I'm trying to replicate your workflow (mostly as a learning experience - my own use case is different) and I have some questions. Not sure the best way to proceed.

0
0
0.000
avatar

Send me a message on discord holoz0r#7605

0
0
0.000