Please login or register.

XMR Pool Infrastructure Overhaul

Update 2018/6/12: this FFS was paid to hyc nVidia solo miner project

This is a proposal that I'm going to do in parts, due to the nature of it - I won't know how much work there is to do, or what milestones to set, until the first step is done. So, first, let me describe the problem XMR has.

All PoW mined cryptocurrencies today face the possibility of 50%+ attacks. While they're called "50%+ attacks," or "51% attacks" - this is a misnomer. Having greater than 50% only ENSURES your attack will succeed. Success can be likely with even 30% of the network hashrate, especially if reversing few confirmations. The attacker has infinite tries, and only needs to get lucky once. Now, a single entity amassing this much hashpower becomes more and more unlikely as the currency grows, but we face more of a threat from the pools the miners use.

XMR currently has what is referred to as "stratum," but is nothing of the sort. It is effectively Getwork over TCP. Now, the scalability issues aren't much of an issue here as they were with Bitcoin - as XMR miners tend to mine in KH/s at the most, miners will hardly run out of nonces to try. But the greater threat here is that the miner knows NOTHING about what they're hashing. They will blindly solve hashes for the pool, even if the block being mined is for attacking the network. Additionally, block creation is not decentralized in any way - the pool chooses the transactions which go into the blocks, as well - this is a disaster for decentralization. On top of all of this, the current mining infrastructure code is so badly put together that it will break in the future - exactly when, I've not calculated, but XMR's block header size, and the location of the nonce, is not constant. When the timestamp grows by a byte, and it will, most miners will cease to mine valid blocks.

Clearly something needs to be done, but what? I propose a heavy modification of Getblocktemplate (GBT) from Bitcoin. The only thing that can be taken from it is the basic ideas, because of how different XMR and BTC are, but the goals are such that miners who run daemons/wallets will check the pool's work against the network, as well as create their very own custom blocks. The pool will still be accepting shares, the same as before, but instead of checking with a block header they sent you, they will check your block proposal and ensure it pays the pool. This means miners may include their own transactions, their own coinbase signatures, and examine every detail of the pool's work that it's handing out. If even a small portion of miners run daemons/wallets, allowing the miner to police the pool, then they can warn everyone else if a pool becomes malicious. Since attacks take time and sustained hashrate, this guts the effectiveness of abusing innocent miners' hash for a malicious attack.

Now, in order to determine how to proceed, I need to do a LOT of research. I need to know the exact layout of blocks, transactions, and how blocks are built from the ground up. This is because block creation will have to be implemented in the miner - I'll also need to consider pool side verification of shares, and the best way to implement communication between the miner and pools. I'll have to bring that all together to create a technical specification for the new protocol - for everyone to be able to implement. After this is complete, I probably will post another thread proposing the relevant development.

So, is this something the community would like to pursue?

EDIT:

As a footnote, here's a comment I posted with proof of the mining code breaking:

For you to look, here's the relevant details:

Here's where the miner hardcodes that the nonce is 39 bytes into the block header: https://github.com/lucasjones/cpuminer-multi/blob/master/cpu-miner.c#L1071

And here's the block header format, showing that before the nonce, there are three variable-sized integers: https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_basic.h#L289

Really, all the current infrastructure has had corners cut at every possible opportunity.


I propose 750 XMR in total over two milestones here. The first milestone will be supporting documentation, and will help XMR in more than one way: more of the data structures and the internal workings of Monero will be documented for people wishing to study it for other purposes than mining. This will include the format of blocks, which requires the format of Monero's variable-sized integers to be documented as well. Also needed for this milestone is documentation on difficulty (difficulty to target and back, calculating average network hashrate from difficulty, this sort of thing), and detail on creating AT LEAST the coinbase transaction (the one that pays the miner.) Full documentation on transactions is not necessary, but would be nice. These entries are to be posted on monero.wikia.com.

The second milestone I propose is the details of the protocol between the pool and the miner. This will include the format of the information and what sorts of information need to be passed between the miner and pool. It will also detail rules for communication between them, and detail how to construct block proposals for the pool, etc.

450 XMR for the first milestone, as it requires the most research and time put in, 300 XMR for the second.

Replies: 29
Gingeropolous edited 7 years ago Weight: 0 | Link [ - ]

WARNING

The lack of activity on this project has put these funds in danger of being re-appropriated.

If any funders significantly disagree, please voice your concern. Funds will probably be moved to pay for hycs work on the solo nvidia miner (which, imo, is well deserved and sort of also gives a nod to the fact that he's, in general, doing awesome monero developing and stuff)

This comment has been stickied at the top of the thread! Go to post

nagrom1981 posted 8 years ago Weight: -61 | Link [ - ]

Donated.

Drhiggins posted 8 years ago Weight: -62 | Link [ - ]

Donation sent.

Reply to: Dufkin
Dufkin edited 8 years ago Weight: -65 | Link [ - ]

glad to see this is going live. sent some xmr as promised :)

Reply to: smooth Gingeropolous smooth Wolf0 fluffypony
Gingeropolous posted 8 years ago Weight: -111 | Link [ - ]

> That's not what happened and what did happen is relevant.

thanks for the clarification. us 3rd wave crypto folks just dont know some things.

rocco posted 8 years ago Weight: -111 | Link [ - ]

thank you wolf0 for working with monero. most of us who are here for some time have respect for your work and know you are a real expert in this.

thats why i support your effort. mining infrastructure has to increase, we should treat this as a security issue.

supporting decentralization where possible is something we always have to do!

Dufkin posted 8 years ago Replies: 1 | Weight: -111 | Link [ - ]

Thank you Wolf0! I'll support this!

Reply to: Drhiggins Wolf0
Wolf0 posted 8 years ago Weight: -114 | Link [ - ]

I believe that's actually because it's estimated based on shares, but they might fuck the divide there, too. Not sure.

Reply to: Wolf0
Drhiggins posted 8 years ago Replies: 1 | Weight: -117 | Link [ - ]

I've always wondered about this because my reported hash rate on the mining pools is always lower than my actual hash I see on screen. Good info to know.

Reply to: Gingeropolous smooth Wolf0 fluffypony
smooth edited 8 years ago Replies: 1 | Weight: -117 | Link [ - ]

> FFS, look at bitcoin - the pool infrastructure could have been designed such that individual miners are validating their own blockchain, but what got hacked together with probably very little foresight was the current infrastructure

That's not what happened and what did happen is relevant. Getblocktemplate was designed around the same time as stratum, maybe even a bit earlier. Miners and pools did not adopt it because they didn't care and just wanted a simpler and lower overhead solution, with no concern whatsoever for validating anything.

The more recent developments with SPV mining shows miners and pools will go beyond even that, and cut any corner to save costs even if it means greatly compromising the security of the chain.

And all that is in the case of Bitcoin, which doesn't have the additional issue of alts where miners frequently just move on to the next coin, giving them even less reason to care about the health of any one coin.

You really can't expect buy in from miners on making the chain secure. They will do the minimum that allows them to get paid, and will search for ways to do even less.

Drhiggins posted 8 years ago Weight: -117 | Link [ - ]

If this was set as a bounty/project for funding I'd donate. Since half the XMR I hold have been mined I'd like to make the network as healthy as possible.

Wolf0 edited 8 years ago Replies: 1 | Weight: -117 | Link [ - ]

Fun fact: In my poking around for the solo miner, I discovered that the pools all show a slightly wrong nethash due to dividing by 1024 instead of 1000. Threw me off for a bit until I realized it was wrong by ~0.5MH/s.

Reply to: smooth Wolf0 fluffypony
Lloydimiller4 posted 8 years ago Weight: -118 | Link [ - ]

My beliefs coincide with Smooth on this one. This overhaul may very well be needed, but its importance may fall behind some other necessary projects that we should not want to deviate funds from.

I would suggest that if this project gets going, that in the research portion, it is looked into as to the impact of the overhaul, and as to whether or not the importance exceeds other projects.

I believe pool mining is necessary, but like others have stated, it isn't perfect. I would rather p2pool or something similar be used more than ordinary pools as well.

I think Wolf0 is a great developer and I am glad he is working with Monero. If he and the rest of the community deem this project to be a necessity, and that it should be worked on sooner than later, then I will happily contribute.

Reply to: smooth Wolf0 fluffypony
Gingeropolous edited 8 years ago Replies: 1 | Weight: -119 | Link [ - ]

For what its worth, I agree with the sentiment of funder burnout, but I don't know when that will occur, and when that occurs depends on how large the funding efforts are. If this one rolls through, we would currently have 2 large fundraising efforts. 18k XMR to Moneromoo, and a currently unknown amount to Wolf.

I think, in general, part of the situation is a lot of Monerians really want to see things develop, so we're excited to fund development of anything, and we're excited to fund developers to spend time on the Monero code. For reasons unknown, Monero isn't attracting talent left and right, and even previous contributors presumably don't have the time to become community-funded developers (with the exception of moneromooo and tewinget). Thus, to brush aside interest in developing because it doesn't align with the research and development goals doesn't seem right.

And what is particularly relevant for this effort is inertia / momentum. What I mean by that is that if, for whatever reason, this effort is tabled for now and the current system keeps chugging along, its possible it will get to the point where it doesn't change simply because the way things are working is "fine". FFS, look at bitcoin - the pool infrastructure could have been designed such that individual miners are validating their own blockchain, but what got hacked together with probably very little foresight was the current infrastructure, and now its the norm that "pooling" means 1 node defines a block, and participants of the pool submit solutions to that block. Thus, IMO, the definition of a pool as a mining cooperative is wrong. In a true cooperative, individuals would have a say.

Thus, yes, we do have pools, and they are not perfect. But while the community / network / infrastructure is small, we have the chance to make things better before it becomes unwieldy to effect change.

And I concur... I also am in the camp that pool resistance should be a priority, but Wolf repeatedly reminds me that this approach would be more of a true cooperative mining architecture than the current "pooling" architecture. I mean, in essence, if we could have a p2pool type thing, that would be best. I think keeping a centralized pool server might somehow fix the problem encountered in p2pool with latency or whatever, but hopefully you get my drift.

but, as always, take this all with a grain of salt because I just came into this space 1 year ago, so there's that.

Reply to: smooth Wolf0 fluffypony
Myagui posted 8 years ago Weight: -119 | Link [ - ]

Would like to express that I am one of these folks (highlight mine):

> Finally, some within the community don't even support pooling at all and view it as inherently harmful (and suggest that a development priority should be pool resistance measures).