A Belated Correction for the 8x8 Opening Book

[hhfhhfhfh1]

I should have submitted this issue and the related file half a year ago, but due to being busy and a bit lazy, I've only gotten around to it now (it is recommended to open the file with Lizzieyzy):

8x8.txt

This file contains the lines (paths) where the KataGo analysis significantly conflicts with the content in the web opening book. Most of these are positions where the web book shows a win rate of 40%–70%, but in reality they are winning for Black or White. A small portion are positions with a 70%+ win rate that are actually draws.

Before this, I did a rough traversal of the 8x8 opening book and picked out positions where the score lead/win rate didn't match, where the win rate deviated significantly from 0/50/100, or where the complexity was high. I analyzed them separately, kept the results that were ultimately proven correct and differed from the opening book, and these are what remain:

https://katagobooks.org/book8jp/1E/15CEBA6C1E958292C6F40CF7EDD31F1E818EA2F8C77A6DD9A332DBA37EDD0CF0.html?symmetry=3
https://katagobooks.org/book8jp/E5/049AE508E591EE1518B58D2C142D03743003416DCEA3888987F10C800512A31B.html?symmetry=5
https://katagobooks.org/book8jp/CB/18680C56CBF34B648314CC95B8189CC257AE02ABD46DFD7A546DAC6A185253F8.html?symmetry=0
https://katagobooks.org/book8jp/F3/24228C03F39A32852E4493984C6AA65F4A99244E044513825A0EF510A990B5E3.html?symmetry=6
https://katagobooks.org/book8jp/08/2312287108F48706663E05B91B4F6A0DF7570D46CA0D626E19C9399A1B451421.html?symmetry=2
https://katagobooks.org/book8jp/0C/178B004C0C7FF7B2DEFDDCF6D1A45B79384FA1309F041BB727C42DC06A514809.html?symmetry=1
https://katagobooks.org/book8jp/05/02D206B4050C64F805F671E9B669772A2840D33C4F6D7A5D32751CE4F1CE3C32.html?symmetry=1
https://katagobooks.org/book8jp/F9/1684CA76F9BC8BEDB1E30FFAA4EEBEC7D0E9B10516F02AB9AD079C5E7D379FD9.html?symmetry=1
https://katagobooks.org/book8jp/F6/406226A8F6E3EE014E3B75CBB195FCDC315AA31E0EBEFB95080CDEFD517882E7.html?symmetry=2
https://katagobooks.org/book8jp/06/0F4242E606B7695B0A1F7DDF7B6C76D6F7D26AE0A7D42F166F31958418969E3D.html?symmetry=2
https://katagobooks.org/book8jp/3B/0C48486D3B64F92797A112DE56E83204C91C79201BD50AD2E92EAD5832B36DD8.html?symmetry=4
https://katagobooks.org/book8jp/EA/3B64D8B7EA5551173DE5F85092822ACC7D0E2C2BB10C407B655242455D2C7F34.html?symmetry=1
https://katagobooks.org/book8jp/75/3FAFEBD675A3D87DC7F0AE01A4C315AEA817DF5B8BCB82F7AE123545F779E6D6.html?symmetry=3
https://katagobooks.org/book8jp/95/352D46DA95B8E3207E6097E260A9475358296AFA5C5FD8D7570377F12A78F965.html?symmetry=3
https://katagobooks.org/book8jp/91/02F40061912F95F1252649C9F24BB370F16381A368591DA4F40DA7B990DE0F19.html?symmetry=4
https://katagobooks.org/book8jp/A6/0A7DA73FA648D9221CB8D23DE57CC0A0A0E3E0D5B36B26008FA34D2EBABBF955.html?symmetry=1
https://katagobooks.org/book8jp/DC/011FF089DCF003AA0045F7AA201E9C8E52C6F31FE58976BEB10B2967A5534FC5.html?symmetry=6
https://katagobooks.org/book8jp/8C/073578F58CB8F2D7502E15B877E8DA5A7AFBC594B2EDFB4540138DC4ACAEF72A.html?symmetry=1
https://katagobooks.org/book8jp/DE/4F5E6299DEF474CB0FB75DB83F1709D7907021A9AAA1F8F4F78905BC51CDD72F.html?symmetry=6
https://katagobooks.org/book8jp/75/059B4EED75CA6EAA686C8230304DAD723E3C20601995588BE9BC9D3DBEED2AE6.html?symmetry=3
https://katagobooks.org/book8jp/FB/1679BE3FFBAD63E038DA603816F8AADC2D9B0A73B7B0062C7941F51F96A50C56.html?symmetry=4
https://katagobooks.org/book8jp/98/65E2834B98C948346A50962468EA80D9062BCB1ABD41BF90AEF16D1500D05FB8.html?symmetry=6
https://katagobooks.org/book8jp/2B/0DF97EE82B08B948C6ED3B0B14D3ED8D19D7D78351661A399420350470D41332.html?symmetry=3
https://katagobooks.org/book8jp/12/03AC0F9B12DB96DE6C72CA5F277902A4D0D082A812DBED27BD434D777879859B.html?symmetry=1
https://katagobooks.org/book8jp/F7/064005C6F79D94A7D9C665C5D3DC5EF45A7233E5111E5493099CF9AE693ABB8B.html?symmetry=2
https://katagobooks.org/book8jp/40/0267F8F040A41E642197F2F04E014D86493C911BD2392CBFF9B72602B96F5E27.html?symmetry=6
https://katagobooks.org/book8jp/13/0399AEE113AAFEF7C3C695189B8DDB640B28201ECDE918845889C5197B9C253D.html?symmetry=6

All of these can be found in the corresponding positions within the SGF file I provided. I can't be bothered to introduce them one by one—if you're interested, feel free to download and verify them yourself.

中文版本：

我本该在半年前就提交这个问题和文件，但是由于我比较忙且比较懒，所以才拖到现在提交（建议使用lizzieyzy打开）：

8x8.txt

这个文件是kata分析与网页开局书上的内容有较大冲突的路径（line），其中大部分为开局书上胜率为70-40，但实际上是黑胜/白胜的局面，小部分是70+胜率但其实是和棋局面。

我在此之前对8x8开局书进行了粗略的遍历，把其中目差胜率不符/胜率离0/50/100差别较大/复杂度较高的局面单独提出来分析，保留分析结果最终被证明正确且与开局书不同的局面，最终剩下了这些：

（同上，略）

这些部分都能在我提供的sgf文件中找到相应局面，我懒得一一介绍了，如果感兴趣可以自行下载并研究验证。

[HackYardo]

You confuse Position and Result.
WinRate or ScoreLead is a Position Value,
Not Result Value.

You can't write down the wr/sl of a position,
play best moves to the game end,
then find out the final result wr/sl is not
the same value as that position.

Go/Baduk/Weiqi AI is also a Picture AI,
input is a picture(position),
output is an evaluation of the picture.
It don't play moves to game end exactly.

As far as I know,
KataGo play only 8 moves from a position.
So moves over 8 may get different evaluation.

[lightvector]

@hhfhhfhfh1 Thank you for the analysis, this looks great! It might be some time before I have the free time to work on updating the book(s) again given the many other things that need attention (releasing the next version, training new models, reviewing some PRs for rocm support, updating the training server, etc), but when I get to this seems like a very useful set of positions to expand the book and correct!

Have you tried running the same on 9x9? I think 9x9 will probably have vastly many more major errors due to the much greater challenge of the board compared to 8x8, and because the book itself is much larger (so there are many more possible nodes that could be proven to have clear misevaluations). And 9x9 also tends to be of greater interest to players than 8x8, so it would potentially be very useful.

Also even if such a report reveals thousands of nodes that need correction, I'm happy to download such a list and apply it once I get to updating the books. Alternatively, one could also find a way of pruning an error report down to just the more important lines, but practically speaking, if you have a reliable process to find misevaluated positions, I'm happy with a larger dump that isn't as perfectly filtered so long as most of the positions are genuine enough. I would handle the book expansion based on them programmatically anyways so it's no harder if it's 50 or 500 or 5000 positions, as long as enough of the positions are valid that to make the compute well-spent in correcting major errors.

[hhfhhfhfh1]

Have you tried running the same on 9x9? I think 9x9 will probably have vastly many more major errors due to the much greater challenge of the board compared to 8x8, and because the book itself is much larger (so there are many more possible nodes that could be proven to have clear misevaluations). And 9x9 also tends to be of greater interest to players than 8x8, so it would potentially be very useful.

Also even if such a report reveals thousands of nodes that need correction, I'm happy to download such a list and apply it once I get to updating the books. Alternatively, one could also find a way of pruning an error report down to just the more important lines, but practically speaking, if you have a reliable process to find misevaluated positions, I'm happy with a larger dump that isn't as perfectly filtered so long as most of the positions are genuine enough. I would handle the book expansion based on them programmatically anyways so it's no harder if it's 50 or 500 or 5000 positions, as long as enough of the positions are valid that to make the compute well-spent in correcting major errors.

@lightvector I'm really glad you like the idea. Automatically checking the nodes is very appealing, but the program only does some initial steps for me. Finding wrong evaluations is mostly done by myself.

I did try to run it on the 9x9 book, but the result wasn't ideal. I concluded two reasons.

The main reason is that my PC isn't powerful enough to analyze the full book, and the networks—whether the latest b28net (at that time) or the 9x9b18net—don't fit the 9x9 board well enough to converge to the best line quickly; sometimes they just can't converge to win/draw.

The second reason is that the program is aimed at finding positions where KataGo is uncertain, not where KataGo might be wrong. The problem isn't the number of positions but the depth. In the 8x8 book, usually 10–14 moves can clarify the outline of the black and white areas, but in the 9x9 book it can take up to 16–20 moves to reach similar complexity. In the 9x9 book, KataGo's analyzed win rate and score can deviate from draw/win/loss before reaching 16–20 moves in depth. Putting a very early position into KataGo usually makes it very hard for KataGo to get a clear win/draw/loss result, because there are too many lines to consider and it can easily miss a small trick 20 moves later.

But with a more powerful computer, there's no doubt that this method can detect many more wrong evaluations to correct.

I deleted the selection script after finishing the selection, so I can only explain how to generate it from memory. I asked DeepSeek to write a script to do a traversal check of nodes on katagobooks.org/book8jp/root/root.html and:
1.Prune all nodes with <20% or >90% win rate.
2.Prune all nodes with more than 14 depth.
3.Prune all nodes with less than 10 depth.
4.Select nodes with abnormal features—here I used >60% or <40% win rate, or < -0.5 or >0.7 score lead.
5.If a child node has its parent node selected, delete it.

Then I wrote a script to analyze the nodes with the 9x9b18net, old b28net, and latest b28net (now they can be replaced by the latest official b28net, 9x9b18net, and zhizi b60net) up to 500k visits twice, and compared the 7 results (including the book node's result). If the results were similar enough, I deleted the node; if not, I preserved it for manual checking. If the book's node evaluation was proven wrong, I would check the parent node and sibling nodes.

If you want the script, I can ask the AI to write another one.