Improve generic EOF header

[chfast]

Current situation

In the current spec the structure of the EVMv1 header is not very generic. In particular we have:

• mandatory and optional sections (therefore also header entries)
• single and array sections; there is no generic rule which section is which
• mandatory sections of size zero (data) and non-empty optional sections (containers)

I believe, two historically separated decisions contributed to this:

• we decided that it is easier to define the data section as mandatory with the potential zero size; back then all sections were mandatory,
• later we added optional subcontainers section; because it is optional we don't allow it to be of size zero.

Also it is important to note that in the future we are going to mostly extend EOFv1 by adding optional sections.

Potential improvements

Data section is optional

1. Forbid sections of size 0. For array sections, the count and all sizes must not be 0.
2. Having 1, the data section must be optional.

Indicate single/array section kind

3. Introduce a bit in the section id indicating a section of this kind is single. E.g. limit the ids range to 0–0x7f. The 0x80 bit indicates the section is single. Sections are ordered by id & 0x7f. Example new ids in order:
   • type: 0x81
   • code: 0x02
   • containers: 0x03
   • data: 0xff

Implementation considerations

The changes allow improving the header parser implementation. You can have a generic EOF version agnostic header parser. The parser should all sections as optional and build internal representation of the EOF sections. In the second phase of validation we can check if the mandatory sections required by specific EOF version are present. Similarly, we can check any other version-specific requirements (e.g. type section matching the number of code sections).

[gumb0]

• we decided that it is easier to define the data section as mandatory with the potential zero size; back then all sections were mandatory,

It's not just historical legacy, main reason why data section is mandatory is aux_data appending operation during RETURNCONTRACT. It may increase data size, and increasing from zero to non-zero size with optional data section would mean we need to insert data section into the header, which would make RETURNCONTRACT an O(n) operation.

Maybe this could be worked around by requiring deploy containers to always have non-empty data section.

Also maybe insertion wouldn't be that big of a deal, given that successful deployment pays 200 gas per byte already anyway. This doesn't help because this charge is applied after the appending.

[shemnon]

We also need to keep the possibility of "large" containers larger than 2 byte size.

[pdobacz]

I have a vague feeling of a self-contradiction in the future-proofing argument being made here:

Also it is important to note that in the future we are going to mostly extend EOFv1 by adding optional sections.

If we're adding optional sections to EOFv1, having a generic format/parser is not an advantage - we only have one version to handle.

Once EOFv2 materializes, it may be that (if the major bump is driven by a container format change, not sth like opcode removal) it will not be handled by the generic parser together with EOFv1. Take for example variable length encodings for EOFv2 - would the generic format/parser be generic enough?

Can we speculate on what an EOFv2 would look like, which would parse generically with EOFv1, but not be possible to still be EOFv1-compatible?

Say, if EOFv2 removes a section kind, I find it likely that any new section kind to replace it would not reuse a section kind, so there will still be a global kind => single vs array association. Say, we want an array of type sections in EOFv2 (whatever...), not a single one, we can assign it a new section kind and require it moving forward. Heck, we can even do that in EOFv1 and make old and new type section kind just mutually exclusive in a single container... I guess...