Bincode dependency now unmaintained

[ratmice]

In the last day or so, the bincode crate has been marked as unmaintained.
The situation is described in the crates.io readme https://crates.io/crates/bincode

Since the code generated by grmtools uses bincode, I'm struggling to see any
way to resolve this without affecting semver (short of new maintainers for the bincode crate).
That seems very unfortunate, since it wasn't that long ago we did a semver breaking release (and bincode reached 2.0).

I don't know of any issues, bugs or otherwise with the current bincode library
that affect grmtools or would otherwise make this an urgent issue other than the maintenance status.
I am not well informed but I believe the current maintenance status is the result of a doxxing that was preceded by the mistaken suspicion of a supply chain attack.

[ltratt]

Because grmtools recompiles things when its own version changes, I think it wouldn't be a breaking change to move to something incompatible. That said, since wincode is apparently compatible, maybe that's the easy short-term fix?

[ratmice]

I guess I was worried that the version of grmtools used as a build-dependency and the version of grmtools used as a runtime dependency not actually matching. Thus we would be encoding with bincode and decoding with wincode.

But for some reason I was misremembering that we actually use bincode in the generated code, but that is not the case. The generated code just contains byte slices, while the generator uses bincode/wincode, and the library calls the decode function within __reconstitute__.

So I do think you're right if wincode can manage to decode bincode encoded slices.

[ratmice]

I filed anza-xyz/wincode#59 asking for clarification to which version of bincode it's compatible with, as it is compatible with the default, but the default changed between 1.0 and 2.0 versions.

grmtools current code generators use config::standard()/variable int encoding

[ratmice]

One (rabbit hole) that is churning through my head a little is serde/syn optionality.
In the code base we still had the dependency on the pretty-printer prettyplease which brought in the syn dependency for all builds of lrpar and lrlex, so much of the compile time savings of bincode 2.x wasn't actually realized.

It appears that wincode-derive at least does depend on syn.

One thought is that CTParserBuilder, and CTLexerBuilder are largely only used when compiling as a build-dependency. If one generated their lexer/parser in the src/ directory (commiting the generated code to git), we could have a feature where the compile-time builders as a whole are optional. I don't remember if this came up when we discussed pretty-printer optionality.

If we wanted to go in that direction, it could be better to find/work towards a fork/continuation of the bincode 2.x code base.

Edit: Let me know if you want me to experiment with seeing how much build-time savings such a project structure could save before a decision is made?

[ratmice]

One last thing which occurred to me is that I think there is going to be some minor semver breakage any way we handle it technically, because the trait implementations of Encode and Decode will change regardless. Seems like wincode has it's own SchemaRead and SchemaWrite traits, and a fork/continuation of bincode codebase would have a different copy of it's traits. Generally this won't actually affect anything but the code generator/reconstitute process though as it seems unlikely any user code relies on the encode/decode traits of grmtools types.

[ltratt]

One thought is that CTParserBuilder, and CTLexerBuilder are largely only used when compiling as a build-dependency. If one generated their lexer/parser in the src/ directory (commiting the generated code to git), we could have a feature where the compile-time builders as a whole are optional. I don't remember if this came up when we discussed pretty-printer optionality.

Just to check we're on the same page: this would mean that consumers of grmtools would have to commit (or at least .gitignore) the compile-time output of lrlex/lrpar? If I've got that right, I must admit I'm not sure imposing that conceptual burden on users is worth the compilation time saving.

Generally this won't actually affect anything but the code generator/reconstitute process though as it seems unlikely any user code relies on the encode/decode traits of grmtools types.

I think we're OK here, for two reasons. First, that stuff is all deliberately opaque, so we've made no commitments to users about the contents. Second, because we're using VERGEN_BUILD_TIMESTAMP, if someone upgrades grmtools, we'll regenerate the compile-time output naturally. So I think it will "just work (TM)" (famous last words sometimes, I know!).

[ratmice]

Just to check we're on the same page: this would mean that consumers of grmtools would have to commit (or at least .gitignore) the compile-time output of lrlex/lrpar? If I've got that right, I must admit I'm not sure imposing that conceptual burden on users is worth the compilation time saving.

Somewhat, in that what you say is what I imagined, but I imagined it would just be an option.
Generally it should continue to work exactly as everything currently does, so the normal process would be unchanged and continue generating into $OUT_DIR.

I'd just be an option, where if you want to build a parser with reduced dependencies you would pre-generate the parser in the source dir and commit it. Then when the end user builds the pre-generated parser that can be done without the syn dependency, because syn is now only needed for the code-formatter during pre-generation.

It'd probably only be worthwhile if you have a very popular library/crate with a parser, the conceptual burden could be worth it if it shaves a little time off each build. Where the syn dependency only needs to be built by the person generating/committing the parser.

The grmtools changes would be something like:

1. Add a default compile-time feature
2. Some option to enable output of generated sources to the src/ dir.
3. The syn dependencies become optional and

The downstream user's burden

1. run the CTBuilder such that it pre-generates the parser in the src/ dir.
2. commit the generated parser/lexer.
3. disable the compile-time features for the runtime lrpar/lrlex dependency.

[ratmice]

I think we're OK here, for two reasons. First, that stuff is all deliberately opaque, so we've made no commitments to users about the contents. Second, because we're using VERGEN_BUILD_TIMESTAMP, if someone upgrades grmtools, we'll regenerate the compile-time output naturally. So I think it will "just work (TM)" (famous last words sometimes, I know!).

I'm not sure if what I meant to say came across, let me try again. (I really don't think it is a big concern, I just wanted to note it).

Currently types like PIDX and TIDX in the cfgrammar crate implement traits like the bincode::Encode and bincode::Decode trait, these are public in cfgrammar, while the actual code-gen is in lrpar & lrlex. Thus in theory (albeit unlikely) since it's pub people can be using the traits outside of the code-gen process.

Switching out the bincode dependency for another is going to swap out those trait implementations another trait like the wincode one.

Sorry I was half asleep when I wrote the other attempt to explain this.

[ltratt]

Currently types like PIDX and TIDX in the cfgrammar crate implement traits like the bincode::Encode and bincode::Decode trait, these are public in cfgrammar, while the actual code-gen is in lrpar & lrlex. Thus in theory (albeit unlikely) since it's pub people can be using the traits outside of the code-gen process.

Right, I'm with you. You are right that this is technically a breaking change... but probably not one that breaks anything in practise. I think let's just go ahead on the assumption that this isn't a breaking change in practise.

Somewhat, in that what you say is what I imagined, but I imagined it would just be an option.
Generally it should continue to work exactly as everything currently does, so the normal process would be unchanged and continue generating into $OUT_DIR.

I'd be fine with this as an option for the most committed of power users; I think it can be done later, separate from upgrading the dependency.

[ratmice]

I'd be fine with this as an option for the most committed of power users; I think it can be done later, separate from upgrading the dependency.

Okay, so the thing and the reason I bring it up is that it appears to affect the choice of dependency.
I.e. wincode appears to depend on serde via the solana-short-vec dependency, and it's derive macro depends on syn.

So this would be something that would be most easily done by using a codebase based on the current bincode codebase. However, that likely means finding a project continuting it, or passively maintaining it (myself?).

I just bring it up because it's something easier to do with our current code, than it would be if we switched to wincode. So it seemed worth thinking about whether it's something we want to try and do at all.

[ltratt]

Ouch, I don't think we want to maintain this ourselves, unless you really want to! I wonder if anyone else might be taking over bincode?

[ratmice]

I certainly not super enthusastic about taking on maintainership.
Also I'm not actually certain how much of a compile-time speed increase can be achieved by getting rid of the syn dependency. It may not even be worth the effort. But I also feel like if it is worth the effort. It is probably worth the burden of maintaining the crate. So I guess I'm torn.

I reached out to the only other user of it I know, they seem like they might be just maintaining their own fork of the current crate. In that thread there is a message from the former maintainer:
where they explain the situation a bit more, they cannot pass the torch to a new maintainer,
because they deleted the github accounts linked to crates.io logins tied to the crate.

jneem/imbl#146 (comment)

[ltratt]

Since we've now realised that we can (and are willing) to change the dependency without breaking change issues, we're not even tied to wincode. If another bincode-ish crate is better, for whatever reason, we can move to that. I'd suggest we do whatever is best for grmtools in the longer term.

[ratmice]

Sounds good, I feel like maybe it'd be good to wait until the holidays are over and see if anyone works or wants to collaborate on a continuation of the bincode codebase, or whether some of the other options could be used to the same benefit?

[ltratt]

Works for me!