r/MachineLearning
[D] We audited LoCoMo: 6.4% of the answer key...
[Projects are still submitting new scores on LoCoMo as of March 2026.](https://github.com/snap-research/locomo/issues/34) We audited it and found 6.4% of the answer key is wrong, and the LLM judge accepts up to 63% of intentionally wrong answers. LongMemEval-S is often raised as an alternative, but each question's corpus fits entirely in modern context windows, making it more of a context window test than a memory test. Here's what we found. ## LoCoMo LoCoMo ([Maharana et al., ACL 2024](https://
Read original on reddit.com41
4Related
Hacker News
If you don't opt out by Apr 24 GitHub will train o
This is where you can opt out. It's absurd that they are automatically opting users into this.https://github.com/settings/copilot/features
news.ycombinator.com274
115Hacker News
Moving from GitHub to Codeberg, for lazy people
Discussed on Hacker News with 602 points and 310 comments.
unterwaditzer.net602
310Hacker News
AI got the blame for the Iran school bombing. The
Discussed on Hacker News with 286 points and 248 comments.
theguardian.com286
248Liked this? Start your own feed.
Your own feed is waiting.
2 comments
This is a really insightful piece. The data backs up what I've been seeing in the industry.
Agreed. Would love to see a follow-up with more recent numbers.
I'm not sure the conclusion holds for smaller teams. Would be interesting to see this broken down by company size.