added the word “Appeal” to the appeal button, because like all buttons except maybe save and undo, nobody knows what your icon stands for:

please add the word “Appeal” to the Appeal button, wherever it appears.

wrote this blog and added it to the top header:

Please add a “Blog” link to the top header, and link it to blog.evalgeni.us (it’s hosted elsewhere, I’ve taken care of it)

(finally) bought pro because:

• i wanted to get that share with a friend feature working
• I ran out of credits
• i wanted to directly edit the code on occasion (sooo much easier to just capitalize that T)

Implemented the share with a friend feature, added it to the top nav and to the results page (“challenge a friend”)

To do on Day 10+:

• Genius Circles, (think private leagues you can start and/or join)
• A daily reminder feature “remind me if I haven’t played and it’s four hours before the game ends” that’s toggleable in the profile. disabled it for now since I wanted to make sure opt-outs were working properly.
• Tournaments (3-day and 7-day to start)
• Lots more scoring algo tweaks
  • I mean, if you know me, you know how much I love reciprocal ranking, that is definitely coming soon.
• Some anti-cheat work
• Suggest a Puzzle
• Delete my data

since Claude is actually the judge in this competition, not me, I decided to add an “Appeal” button to the final matches:

on the scoring page, in the section where you display each of the matches, I’d like to install an Appeal button. it should be in the top right corner of each of the boxes that hold one of the matched items.

When the user clicks the Appeal button, a modal should appear. The modal has room for two fields, a dropdown and a small text box, but only the dropdown appears first. The “title” of the dropdown is “Reason:” and the default choice is “select a reason for appeal”. The actual choices, one of which is required, are: “this answer is a subset of the answer”, “this answer is not accurate”, “this answer is not real/contrived”, “general bullshittery”, and “Other”. in the bottom right corner of the modal is a submit button, it should not be clickable until the user has made their selection (And followup if applicable – see below)

When the user selects either “general bullshittery” or “Other”, the text box below the drop down and the user is required to enter text in the box before clicking the submit button to proceed. If the user selects any of the first three choices, no text box appears and the submit button becomes activated.

when submitted, the appeal is stored in the new “Appeals” section of the admin area that you’ll need to create for me. When I am an admin, I can click on the appeals tab and see all appeals. each row in the appeals admin area has username, criteria used, answer being appealed, reason from the dropdown, and the text from the comments box, if used. Eventually I’ll be able to take action from here as an admin, but for version 1, let’s just get those appeals into a list in the admin area for them to look at.

The night Started off with a few design tweaks:

Three changes please:

1. please put the logo on the right side of the Title instead of the left, and also put it on every page, wherever the title currently appears. Also, the Eval Genius wording gets too opaque at the end, the gradient is too much… maybe just all-same-color now that there’s a logo decorating it?
2. Where it says Final score, add “Lowest is Best”
3. On the profile page, compute each user’s Average Daily Percentile , and Average Daily Score, and give those numbers a place to be seen on the profile page with the other aggregate data

then, immediately:

you know what, i was wrong, the logo looks better on the left of the logo everywhere. Please swap it from the right side of the title to the left side of the title everywhere it appears.

after that, I tried to build a share with a friend feature, but found that feature was only available in pro, so i deprioritized that feature:

I’d like you to add a “Share with a Friend” feature. It’s a button on the scoring page, where the logged in user can share today’s puzzle (and a score challenge, optionally, with a checkbox.) when they click the button, a modal should pop up, where the user can enter a friend’s email address, and check box to include their own score in the email as a challenge. then there’s a “share” button that submits the form and closes the modal after the modal swithces to a “success!” feedback for a few moments. WHen the user invites a friend, the system (admin@evalgeni.us) needs to send an email on behalf of the user that says:

“Your friend [their username] (their email address) has invited you to play EvalGeni.us, a Daily Puzzle Game that’s a fun way to get better at thinking and writing evals. [Click here to play today’s puzzle.](link to EvalGeni.us)

-Your Friends at Eval Genius”

let me know if you need an SMTP server or if you can build one for me to use.

Today was about finishing the on-page guidance text, after another round of feedback.

Please make the following text and layout changes:

Change 1:

on the home page, below “Eval Genius”,

Change:

“Train your Mind to Think in Evals”

to:

“A Daily Puzzle Game”

and

change:

‘Welcome to the daily puzzle game that teaches you the “eval mindset” – the art of thinking precisely and systematically.’

to:

A fun way to learn the “eval mindset” – a precise, systematic way of thinking that AI/ML teams use everytday.

change 2:

add the attached Eval Genius logo (evalgeniuslogo.gif) to the right of the Eval Genius Header Title

change 3:

add a section below “How it works” section

separated by another line like you did after Continue with Google

with the bold header “Sample Puzzle (screenshot)”

and the attached screenshot “Sample.png” displayed below that. It should be clickable which would bring it to full size.

change 3: on the /game page, below the word of the day, change:

“Create evaluation criteria that uniquely identify this word”

to:

“Score low and win by writing the minimum criteria that uniquely identify today’s object”

change 4: also on the /game page, replace “Lowest Score wins” with

“Score low and win by writing the minimum criteria that uniquely identify today’s object.”

I got some great feedback from Volker that he wanted to see the scores of the people that were beating him. That’s totally in line with our mission – learn from the best! In a combination of fairness and credits optimization, I decided to expose everyone’s criteria on the leaderboard, but only visible once you’ve submitted their scores.

please add the words chosen (like you’ve done in the play history page) to the leaderboard, (so users can see what others have entered but ONLY allow them to be visible once the player has played that day. If the user who hasn’t played yet today visits the leaderboard, they should still see the gamertags and rankings, but they should see neither the scores, nor the words chosen by the leaders, until they’ve made their day’s guess.

Also, to protect against spam. The game is naturally spam resistant – to get to the top of the leaderboard, you’ve got to score well, and since the criteria are the score, any wasted words on off-topic rants push troll scores down naturally.

Anything that [meets my criteria for redaction] should not appear publicly and should say -REDACTED- instead. This is a family app, after all!

Balancing elimination of off-topic and allowing for lots of creativity will be an ongoing challenge , so expect further tunings to the REDACTED phrasing, and please do report inappropriate submissions!

Today’s goals were around usability, and adding a My Profile section. I had not written instructions nor given any scoring tips yet:

Leave the How Scoring Works Header, but replace everything else in that box with the following markdown:

Lowest Score Wins!

Your score has 2 components:

Your Brevity score is calculated by the number of criterion you enter, and the number of words in each. Scoring punishes putting multiple criterion in one line – it’s better to break them up, one thought per line.

Your Clarity score is (an exponent of) the number of objects Claude can find that meet all the criteria you entered. One is the target here.

These are multiplied together to achieve your Total Score.

Scoring Tips:

The scores are weighted such that it’s always better to add an extra criterion to reduce the Clarity score to 1, as the clarity score gets worse exponentially and criteria worsen linearly.

Articles and lead-in words should be stricken from scoring, so “Is a flower” should score the same as “flower”, but since you’ve read the directions, you can just say “flower”

One exception: “has” – if your eval “has” something, you’ll need to say that word. We assume “is”, but never “has”

And I also wanted to add a profile so people could have anonymous usernames, check their history, and eventually, do other “Account stuff” like join Genius Circles or Change their Usernames.

We need to create a new section – a My Profile page – it should have a link up in the top row with archive and leaderboard. When you go to the My Profile page, you’ll be able to see your GeniusTag (username), and a list of all of your previous plays, where each day’s “row” (entry) has information about where you ended on the day’s leaderboard, what the puzzle was, and a list of the criteria you entered.

Heres the prompt I used to generate your Genius Tags:

When the user first signs up, ask them to create their Genius Tag. There should be a default Genius Tag in the box. It should be two words concatenated together. The first word will come from the following list:

Gloopy, Radiant, Zigzag, Misty, Bouncy, Velvet, Crinkly, Golden, Humble, Spunky, Dotted, Frosty, Mellow, Zesty, Placid, Breezy, Shiny, Wobbly, Silky, Clever, Polished, Grassy, Cheerful, Sturdy, Dapper, Jovial, Crisp, Fluffy, Teal, Noble, Static, Braided, Gentle, Puffy, Kinetic, Rustic, Snappy, Eager, Balanced, Nimble

the second word will come from the following list:

Flabberjax, Glipglop, Zoorp, Quibber, Blonket, Snazzle, Murp, Voomp, Plincher, Skritch, Troodle, Glimp, Wozzel, Drimble, Splatnet, Floofle, Krink, Zandle, Blorp, Snarkle, Ploof, Gribble, Twizzle, Vronk, Jibber, Skrazz, Mip, Blidgit, Thrumble, Quirp, Glonk, Sprazzle, Flimm, Droozle, Plonk, Skree, Vazzle, Bromp, Tinket, Zibble

in that modal interstitial first screen, the user should be allowed to edit the default Genius Tag – it’s just a suggestion after all.

Two topics today – first, I wanted to add an Archive, so people could still play puzzles they missed, or catch up.

this is great! Now, can you add a section to this site for logged in users called “puzzle archive?” In that zone, there’d b a calendar view, and you could click on a day and see that day’s puzzle. You could try it, and get a score, but you wouldn’t be added to the leaderboard – the leaderboard is only for people who played the puzzle on the day that it was a daily puzzle.

Second, the leaderboard didn’t actually work, so I used the rest of the day’s free credits to get that working:

…the leaderboard still doesn’t work. when I look at the leaderboard after completing my game, it says no submissions yet today, but… I just played, my score should be there!

Five minutes later… 

add a link to the leaderboard to the top nav area where archive and sign out currently live, but also, it still isn’t working, i’m still seeing a blank leaderboard

Day 2 was about three major updates. First, the connection to claude was failing, so I had to debug that. I described the errors, the problem got found and fixed 3 credits later.

it says failed to submit, please try again when i try to submit. I tried from two different accounts

Test submitting criteria from a regular user account to verify the fix works end-to-end

The second was returning the list of items that matched to the player’s results page. Not only did early user testers want to see it, but I needed it for debugging purposes:

On the Your Results Page:

1. Please add a list of the items that matched. The results page says I have 5 matches, but I don’t know what they are.
2. No need to have the Matches:5 for each criteria. The Matches should be at the top, with the count, and the list of each match in the list. Then belwo that, a list of eval criteria with word counts.

The Third was adjusting the scoring algorithm. It would undergo lots of further adjustment, and is still being optimized.

This all started with a word salad on LinkedIn the week before…

EVALS.

You’re seeing them more and more as LinkedIn’s algorithm feeds your interest in AI.

You probably know “Eval” comes from “Evaluate”. You know evals are important, but… are they important for you? where do you start?

If your job is tied in any way to improving the answer or task completion quality of users interacting with your models, The art of writing good Evals starts with knowing what they are. Like all good guidelines, these flex to the situation, but Evals are generally :

• Yes/No Questions
• Concise and Precise (and generally pronoun-free) with no supersets.
  • You wouldn’t need both “Is an animal” and “is a mammal” to describe a cat. All mammals are animals, so “is an animal” is unnecessary.
• Atomic (“One Thought per Line”)
• When combined, they uniquely describe your object
  • (in the real world, they’d be uniquely combining your project success criteria, but for the purposes of this game, we’re doing evals on objects).

But I didn’t need words to inspire me. I needed to git gud. And I wanted my teammates to git gud. So I fired up lovable and started with this:

Project Goals:

The overarching goal of this site is, through habitual daily use, to train product managers and AI engineers to “adopt the eval mindset”, which might also be expressed as “thinking in terms of evals” or could also be expressed as “training yourself to think eval-natively”.

But to be habitual, you need to be fun!

So to make learning this mindset fun, we’re going to use gamification techniques… nay, an actual game! To help get people hooked.

I’ve invented such a game, called Eval Genius, and you’re going to help me build it.

Overview

This is a daily puzzle app.

Step 1: Login/Create Account

Standard stuff here, nothing exciting, do whatever’s easiest including requiring login w/ google.

A user must be able to login, change their password, delete their account, and track their progress. More details below.

Step 2: See Today’s Object & Enter Evals

After login, the user is directed to that day’s “Game”

The game screen consists of a large single word/small phrase (3 words max) at the top, and then a series of text boxes, aligned on both edges in a vertical row, with a button to add more rows, and a button to submit, both at the bottom.  The user adds as many evals to the list as they think they need (remember the goal of evals is as few as possible to do the job), and then clicks submit and moves to step 3.  There are five lines visible by default.  The user must be instructed to phrase their question so that the object in question is matched to a “YES” answer for all questions.

Step 3: Submit & Score

In step 3, we score their work. we gather each of the entered evals and build this prompt:

[here’s where I added my scoring algorithm. Happy to share it if you message me, but not publicly posting this part]