Maverick Wins CCRL Division 7 (and Gets a CCRL Rating)

Over the past month Graham Banks has been running the Division 7 competition. I was delighted when Maverick managed to win with a score of 27.5 out of 44 games. After nine round Maverick languished in the bottom half of the table. It managed to fight back and win! During the tournament I logged onto Graham’s site quite a few time and it was nice to chat with Graham and Erik. There were many nail-biting games – not good for the blood pressure!

Graham then ran a gauntlet competition for Maverick to get enough games for a rating. It managed a respectable rating of 2317 ELO on the CCRL scale. You can see the details here:

Maverick’s CCRL Rating

As I mentioned on a previous post, Maverick doesn’t do so well at slow time controls, so I have happy it came out above 2300 ELO on CCRL.

Many thanks to Graham for taking the time to test Maverick!

Maverick’s Rating at Fast & Slow Time Controls…

I do most of my test at fast time controls. Sometimes game in 10 seconds or 5 seconds plus a small increment. This enables me to evaluate and tune changes using oodles of games.

Based on these super-fast test Maverick 0.51 rating seems to be about 2375 ELO on the CCRL scale. For example, I pitted Maverick against TJ-Chess rated 2367 ELO at 10 seconds per game. After 1000 games Maverick wins by 518 wins 347 loses and 1356 draws. This is +60 ELO.  Maverick seems to get similar results against other engines close to TJ Chess’ strength e.g. OliThink and Dragon 4.6.

So when Graham Banks tested Maverick in Division 6 I thought it would do quite well. I was wrong! Maverick ended up in 11th place out of twelve participants (TJ Chess came in 4th):

Division 6 Results

I thought I’d investigate and run a match at slower time controls. I used the same time controls as the Division 6 tournament (40 moves in 25 minutes) on my a 2.8 GHz i7. The results were not what I expected (or hoped for). Maverick lost with 20 wins 40 draws and 40 loses! This results shows TJ Chess is about 100 ELO better at slower time controls. This is a swing of 160 ELO between slow and fast time controls – far more than I thought.

As a result I’ve revised my estimate of Maverick’s strength based on the time controls.  At super fast time controls it’s about 2400 ELO while at longer time controls it’s 2250 ELO! 

Why the Difference?

I suspected the branching factor in Maverick is quite high. I ran some tests and indeed it seems to be about +4.5. This is high by today’s standard. I think Stockfish is about +2.0. This means every ply takes 4.5 longer to complete than the previous ply. At super-fast time controls Maverick does quite well it’s a relatively fast searcher (4.2 million nps on my i7). As the  time controls get longer the branching factor takes its toll and the more selective engines out-search Maverick.

The high branching factor is almost certainly due to lack of selectivity in Maverick’s search. It does use Null Move but it doesn’t use any form of Late-Move-Reduction, which seems to be the cost common, and most effective, form of selectivity. This is by design. I have a hunch that if I can create a reasonably good evaluation function, I’ll be able to add a more effective form of LMR selectivity, guided by the evaluation function. My aim is to take Maverick up to 2500 ELO before I add this selectivity. It looks like I have 200 ELO to go!

How strong is Maverick 0.2?

Maverick 0.2 has been out a couple of days now.  I’ve run some test games and I really quite surprised at how strong it seems to be.

Fruit 1.0:

The first test was against Fruit 1.0.  I played this match using Shredder’s GUI. As I expected Fruit won.  I really was dazzles by Fruit’s silky smooth search as it carved it ways through each ply (especially in the endgame).

  • CCRL Rating = ???? (between 2300 and 2400)
  • Time = 1 minute plus 1 second per move
  • Result for Maverick: Wins = 24, Loses = 63, Draws = 13
  • Percentage = 30.5%
  • ELO Difference = -146

I was pleased with this result.  However, it would seem Fruit’s strength is not well established.  As far as I can tell it us somewhere between 2300 and 2400 ELO on the CCRL scale.  This would put Maverick 0.2 between 2150 and 2250 ELO.

Phalanx XXIII:

The second opponent was Phalanx XXIII.  Another classic engine.  As I watched them play it is clear Phalanx has much more knowledge than Maverick.  Yet Maverick did well to win some game based on its speed.

  • CCRL Rating = 2387
  • Results for Maverick: Wins = 36, Loses = 49, Draws = 15,
  • Percentage = 43.5%
  • ELO Difference = -45

Once again I was pleased.  Based on this result Maverick is over 2300 ELO, which is probably too high but let’s see.

And remember, at this stage Maverick is lacking some standard features:

  • No mobility in the evaluation
  • No passed pawn code
  • No king safety
  • No pawn structure code
  • No late move reduction
  • No Internal Iterative Deepening
  • Lots of endgame evaluation missing

So I’m encouraged by Maverick’s strength.  If you run any sort of ratings test I’d be interested in the results.

I will not release another version until it has gained 100 ELO.  This may be some time.