Predict the future with LLMs (?)

The Subnet's Mission Statement

Incentivizing the prediction of future events.

Product / usecases from the subnet

I couldn't see any product or usecases right now :(

Incentive mechanism

Miners are scored on their ability to predict future events. They are asked to submit probabilities, and get a score based on how close their probabilities were to the true prediction.

There is a good article by the subnet owners already on this here which saves me a lot of typing.

The key point here for me to note is that the incentive mechanism does not encourage miners to use LLMs. This seems quite contrary to the goal of the subnet.

It's fairly simple, but well thought out. 7/10

Code quality

A lot of old unused template code kicking around. Lots of heavily commented code, and side effects. An example is the setting of weights that happens with a 'sync' function. But the sync function is called on init of the validator. But don't worry as the should set weights function has a check for whether we are on the init or not.

All of that is a bit convoluted, and likely remains from trying to fit to the subnet template. There are also some imports which don't resolve easily with IDEs - which if anything just makes me unable to control+click and scan the code :angry:

Miners have the ability to set weights, which should never be used.

We have lots of concurrent forward functions happening, which all mutate the state of the validator at the same time. Since they are all async, we could get potential side effects there.

The code quality is very rough and template heavy. I would strongly recommend a refactor to make it easier for all parties to work with. It's quite hard to dig through the code and understand what's going on.

1/10

Miner competition

The incentive curve is good. There is a high right-hand tail for the top performers.

It's hard to assess the actual performance without any utilities or dashboards. Though I must make this point:

If we can't easily assess how good miners are performing, how can we be sure they are performing as expected? What is even a good performance in real-world terms?

I think it should always be obvious what a good miner is in real-world terms (not just on emission values), I fail to see that here.

4/10

Potential

LLMs will be smarter than us - I think LLMs acting as prediction models is a niche area but still an area with potential. I like the direction of this subnet, and it's very unique.

8/10

Contributes to decentralised AI

Yes - but this could be improved by being able to have the models of a miner - or at least being able to use the subnet better.

7.5/10

Subnet utilities (e.g. dashboards, insight for miners and validators, etc)

One public Grafana here and a lovely website here.

The Grafana is OK but doesn't really provide much insight in real-world terms.

3/10

Decentralisation

No Subnet owned centralized APIs, which is great. Miners are decentralized and can compete fairly. There are centralized APIs needed, but they are not run by the subnet.

Since the subnet relies on centralized APIs, it is not truly decentralized, but it is very good.

8/10

Innovation

Very innovative, unique and good idea! It's hard to give this subnet more score until we see what that innovation actually means in real terms though.

5/10

Community Engagement

The subnet owner is constantly engaging with the community, and pushing new features. They also responded to my queries quickly. There is not much engagement from the community though, and almost no miner questions. I wonder if that is because there are limited unique miners.

7/10

Advice to improve

• More products and tangible usecases
• Think how we can further decentralise AI - perhaps with miner submitted models