Creating an Opening Book for Maverick

Maverick is the first chess engine I’ve written which can use its own opening book.  I decided at an early stage Maverick would read Polyglot opening books.  It makes so much sense.  Polyglot seems to be a great tool create opening books and its format seems to be emerging as the de-facto opening book standard.

So now I need to actually create an opening book.  I did some research on the internet and it really is quite straightforward. Here are the steps I took:

  1. There are some really good free tools which you’ll need to download:
  2. You will also need a database of high quality game.  I used Ed Schroeder’s Million Base. Make sure your database is in PGN format.
  3. Open the MillionBase using SCID.
  4. We’ll be filtering the games using SCID’s excellent filtering system – select “Search / Header…” (Ctrl-Shift-H)
  5. I created a database consisting of only White wins by players over 2400 ELO with opponents over 2200 ELO (see diagram)Filter Options
  6. I then saved the white win database as “white2400.pgn” using SCID’s “Tools / Export All Filter Games” command
  7. Repeat for Black
  8. Now put both PGN files in the same folder as Polyglot.exe, PGN-Extract.exe, the batch file.
  9. The batch file contains the commands to create the  opening book

  1. The first line runs PGN-Extract and simply cleans the database, removing and games which don’t start from the normal starting position.
  2. The second line creates a opening book of white moves, where the positions have occurred at least 50 times.
  3. The third line creates a broader book which give more responses to a variety of openings.
  4. This is repeated for Black
  5. Finally all of the books are merged

The final opening book is a little over 1 Mb (it’s available on the download page)- quite small by most standards.  I’ll need to run some test to see how good it is compared to other Polyglot books.

If there are any opening book specialists out there I’d appreciate any feedback. Can the process of creating the book be improved?

  • cd

    Very well written tutorial . Thank you very much.

    I recommend ” Scid vs. PC” rather than SCID

  • Steve Maughan

    Hi CD – Thanks for the comment! What’s the difference between SCID and “SCID vs. PC”? It’s a strange name!?

  • cd

    I was going to say this: “SCID vs. PC is a fork bug fix of scid (since 2009-2010) , which latter does not seem to be maintained.
    For comparison , I do not know to tell you the differences between them.”

    But I realize now that this is false, because it seems in fact that scid is somewhat active (last release May 2013 – last code commited July 2013). But IMO Scid vs PC is much more active.

    Regards

  • Derrick Wright

    I loved the topic you wrote about making the bin book, your explanations were very clear. I have been interested in making bin books for a good while now. I had downloaded a free polyglot book maker, which is extremely easy to use, but it uses a lot of memory. It even allows you to combine bin books but I have not figured out how to do that yet. I can send you the link to it if you want it. Anyway, I have been trying to duplicate the process you used to create maverick’s bin book using a much larger game data base but have been unsuccessful so far. After running maverick’s batch exec, I keep getting the messages pgn extract is not recognized as an internal or external command, operable program or batch file also can’t open file “clean-white2400.pgn”: No such file or directory and the same goes for black. Another message reads can’t open file”b1.bin”, can’t open file “w12.bin”: No such file or directory, and and so forth. It’s like as if the pgn files are not there although I have everything inthe same folder. What am I doing wrong please advise, thank you. I am trying to make the book on a windows 8 intel i5 lenovo ideacentre k450 desktop

    • Steve Maughan

      Hi Derrick,

      I’ve added a video version of the post. Take a look here http://www.chessprogramming.net/computerchess/create-a-polyglot-chess-opening-book/. Hopefully you’ll find it useful.

      Steve

      • Derrick Wright

        Hello and thanks for replying, I beat you to it, already saw the video and still can’t see what I am doing wrong. I just can’t figure out what it is that I am not doing or doing wrong that is preventing me from getting it right, even after watching the video twice, it’s really not that complicated at all, I get the same response as before. I guess I will just have to keep trying to figure it out. I think your book making tool is better than the other one I have as it gives a broader range of responses and you have more control over the outcome. Thanks again!

  • MvK

    I do almost exactly the opposite. I book the games where the repertoire player loses, and then make sure I provide an alternative move along the path.

    • Steve Maughan

      Hi MvK – so you’re plugging the gaps. Is it possible to automate this approach?

      • MvK

        I automate it. Book updating is a full time effort.

        • Steve Maughan

          Cool – it doesn’t sound like an easy task

  • Nicolas

    Hi Steve,
    Why have you using ply 16 ?
    May be ply 20, can be more interesting ?

    • Steve Maughan

      Hi Nicolas – I’m not sure 20 ply would make much difference but you’re certainly free to give it a try!

  • Just a FWI 16 to 20 ply make a huge difference….

  • molicab

    I used your batch file, but it stops when it reaches Poliglot after 120000 games. Lexical error. Is there a way to skip problematic games?

    • molicab

      My bad, after checking that video you posted on YouTube (pure gold), I see I didn’t check the correct null move exporting option.

  • faizan

    i ran the batch file but it shows can’t open clean white or black and no such file or directory plzz help me

    • Steve Maughan

      I’m not sure what the problem is. Do you have all the applications and database in the same folder?