Building a Poker Bot: String and Number Recognition
This is the second part of Building a Poker Bot series where I describe my experience developing bot software to play in online poker rooms. I’m building the bot with .NET framework and F# language which makes the task relatively easy and very enjoyable.
Here is the first part: Building a Poker Bot: Card Recognition
Why string recognition
Reading cards and other fixed images was the first step. The bot should also be able to read different text-based information from the screen, e.g.
- Current blind levels
- Current pot size
- The size of bets made by each player
- Player names
- Stack sizes
- Chat messages (for advanced scenarios)
We need this vital information to make proper decisions, so let’s look at how to parse the textual data.
String recognition has some specific difficulties when compared to fixed images like cards:
- The size of a string is not predefined. Obviously, the longer the string, the more space it takes on the screen
- The position of a string is not fixed either. Some strings are aligned to the center, others may diverge based on other variable parts like stakes or blinds
- Different strings might be rendered in different font size
Here is what needs to be done to overcome these complications:
- Pick the layout which makes your life easier
- Adjust fonts and positions if possible
- Make sure that all important strings are always visible and not overlapping to other information
- For each string define a region where it belongs to in 100% cases. The background of this region should be more or less evenly filled with a color in contrast to the font color.
String recognition steps
We start with a screenshot of a poker table again:
We know our fixed regions where our labels are located, so we take those regions for processing:
For each region we trim away the blank margins around the text (i.e. left, top, right and bottom padding):
We find dark lines between bright symbols and we consider them as gaps between characters:
The final step is to compare each symbol to the known patterns and find the best match (in case of my layout the match for symbols is always 100% perfect). Let’s look how these steps are implemented.
Removing padding around the text
Because the padding is removed from all 4 sides of the region, I decided to use
Array2D data type to be able to iterate in different order. The whole algorithm operates
with black or white points defined as a helper type:
type BW = B | W
removePadding function has type of
BW[,] -> BW[,] and looks
let removePadding pixels = let allBlack s = Seq.exists ((=) W) s let maxWidth = Array2D.length1 pixels - 1 let maxHeight = Array2D.length2 pixels - 1 let firstX = [0..maxWidth] |> Seq.tryFindIndex (fun y -> allBlack pixels.[y, 0..maxHeight]) let lastX = [0..maxWidth] |> Seq.tryFindIndexBack (fun y -> allBlack pixels.[y, 0..maxHeight]) let firstY = [0..maxHeight] |> Seq.tryFindIndex (fun x -> allBlack pixels.[0..maxWidth, x]) let lastY = [0..maxHeight] |> Seq.tryFindIndexBack (fun x -> allBlack pixels.[0..maxWidth, x]) match (firstX, lastX, firstY, lastY) with | (Some fx, Some lx, Some fy, Some ly) -> pixels.[fx..lx, fy..ly] | _ -> Array2D.init 0 0 (fun _ _ -> B)
The first part finds the amount of fully-black columns and rows in the array. Then, if white points are found, the second part returns a sub array based on the indices, otherwise empty array is returned.
Split the text into characters
First, we convert our 2D array into the list of lists, where each item in the top-level list represents a single column of pixels:
let pixelColumns = [0..Array2D.length1 pixels - 1] |> Seq.map (fun x -> pixels.[x, 0..Array2D.length2 pixels - 1] |> List.ofArray)
Then we can fold this list of columns into the symbols, where each symbol itself is the list of columns:
let splitIntoSymbols (e : BW list) (state: BW list list list) = match state with | cur::rest -> if isSeparator e then match cur with | _::_ -> ::state // add new list | _ -> state // skip if we already have empty item else (e::cur)::rest // add e to current list | _ -> [[e]] Seq.foldBack splitIntoSymbols pixelColumns 
The type of
state is a bit of brain teaser, I guess it could be improved
by introducing some intermediate type with descriptive name, but I decided
to leave that part for now. Read it as list of symbols, which are lists of
columns, which are lists of pixels.
Match the symbols vs the known patterns
This part was already described in my first article. Basically we compare the list of black or white points to the patterns of the known symbols:
let getChar patterns bws = let samePatterns h p = Seq.zip h p |> Seq.forall (fun (v1, v2) -> v1 = v2) let matchingPattern = patterns |> Array.filter (fun p -> List.length p.Pattern = List.length bws) |> Array.filter (fun p -> samePatterns bws p.Pattern) |> Array.tryHead defaultArg (Option.map (fun p -> p.Char) matchingPattern) '?'
Putting it all together
recognizeString function accepts lower-order functions to match
symbols and get pixels together with width and height of the region:
recognizeString: (BW list list -> char) -> (int -> int -> color) -> int -> int -> string
It builds an array of pixels, removes padding and folds with recognition.
let recognizeString matchSymbol getPixel width height = let pixels = Array2D.init width height (fun x y -> isWhite (getPixel x y)) |> removePadding let pixelColumns = [0..Array2D.length1 pixels - 1] |> Seq.map (fun x -> pixels.[x, 0..Array2D.length2 pixels - 1] |> List.ofArray) Seq.foldBack splitIntoSymbols pixelColumns  |> List.map matchSymbol |> Array.ofSeq |> String.Concat
Then we use it with a specific recognition patterns, e.g. known digits in case of numbers recognition:
let recognizeNumber x = recognizeString (getChar numberPatterns) x
A way to produce these patterns is discussed in the previous part.
String recognition takes a bit more steps to execute comparing to the recognition of fixed objects. Nevertheless it’s pretty straightforward to implement once we split it into small and well-understood conversion steps. The full code for card recognition can be found in my github repo.
Proceed to Part 3 of Building a Poker Bot: Mouse Movements.