OS and protobuf version
go1.18.1 linux/amd64, v1.5.2
Introduction
I am trying to use recursive proto definitions.
.proto file
message AsyncConsensus { int32 sender = 1; int32 receiver = 2; string unique_id = 3; // to specify the fall back block id to which the vote asyn is for int32 type = 4; // 1-propose, 2-vote, 3-timeout, 4-propose-async, 5-vote-async, 6-timeout-internal, 7-consensus-external-request, 8-consensus-external-response, 9-fallback-complete string note = 5; int32 v = 6 ; // view number int32 r = 7;// round number message Block { string id = 1; int32 v = 2 ; // view number int32 r = 3;// round number Block parent = 4; repeated int32 commands = 5; int32 level = 6; // for the fallback mode } Block blockHigh = 8; Block blockNew = 9; Block blockCommit = 10;
}The following is how I Marshal and Un-Marshal
func (t *AsyncConsensus) Marshal(wire io.Writer) error { data, err := proto.Marshal(t) if err != nil { return err } lengthWritten := len(data) var b [8]byte bs := b[:8] binary.LittleEndian.PutUint64(bs, uint64(lengthWritten)) _, err = wire.Write(bs) if err != nil { return err } _, err = wire.Write(data) if err != nil { return err } return nil
}
func (t *AsyncConsensus) Unmarshal(wire io.Reader) error { var b [8]byte bs := b[:8] _, err := io.ReadFull(wire, bs) if err != nil { return err } numBytes := binary.LittleEndian.Uint64(bs) data := make([]byte, numBytes) length, err := io.ReadFull(wire, data) if err != nil { return err } err = proto.Unmarshal(data[:length], t) if err != nil { return err } return nil
}
func (t *AsyncConsensus) New() Serializable { return new(AsyncConsensus)
}My expected outcome
When marshaled and sent to the same process via TCP, it should correctly unmarshal and produce correct data structures.
Resulting error
error "cannot parse invalid wire-format data"
Additional information
I tried with non-recursive .proto definitions, and never had this issue before.
2 Answers
The stupidest error I can think about is that the wire.Write(bs) don’t write as many bytes as the io.ReadFull(wire, bs) read - so I’d just make sure that their return value is actually 8 in both cases.
Then I don’t know the golang/protobuf very well, but I guess it should be able to do this. Shouldn’t you create the go-code and then call out to it? I’m not sure how to call it.
If you think that it’s actually a problem in the protobuf implementation, there are some online protobuf-decoders, which can help. But they sometimes interpret the stream incorrectly, which could be the case here with a recursive pattern, so you have to be careful. But at least they helped me to debug the dedis/protobuf package more than once.
As a last resort you can make a minimal example with recursive data, check if it works, and then slowly add fields until it breaks…
This is not a bug with Protobuf, but its a mater of how you marshal and unmarshal protobuf structs.
As a concrete guideline, never concurrently marshal and unmarshal protobuf structs as it my lead to race conditions.
In the specific example you have provided, I see recursive data structs, so even if you use a separate struct for each invocation of marshal and unmarshal, it's likely that the pointers in the parent can lead to shared pointers.
Use a deep copy technique to remove any dependency so that you do not run in to race conditions.
func CloneMyStruct(orig *proto.AsyncConsensus_Block) (*proto.AsyncConsensus_Block, error) { origJSON, err := json.Marshal(orig) if err != nil { return nil, err } clone := proto.AsyncConsensus_Block{} if err = json.Unmarshal(origJSON, &clone); err != nil { return nil, err } return &clone, nil
}
