WIP: p2panda Migration script#239
Draft
adzialocha wants to merge 16 commits into
Draft
Conversation
This patch integrates the new `p2panda` crate into the "backend" of
Reflection.
`p2panda` is the new all-in-one high-level Node API which should make
usage of p2panda features easier, removing a bunch of code in
`reflection-node` which is now handled in p2panda directly, such as:
* Encoding / Decoding of operations and their message payloads
* Make sure changes to the append-only log are strictly serialized, to
avoid breaking consistency / integrity of data due to race conditions
* Re-playing all operations on demand, this is required by Reflection to
re-materialize all state for a document
* Manage header extensions to include log ids and prune flags
* Process local and remotely incoming operations ("ingest", "pruning")
* SQLite storage handling for logs and operations
* Mapping between logs and topics
* De-duplicate messages in ephemeral streams by adding timestamp
* Add public key / signatures to give provenance to ephemeral messages
Some features implemented in `reflection-node` stay as they're adding
functionality _on top_ of `p2panda`. Some of them might be moved there
as well in the future, such as:
* Author Presence: Indicate which authors have contributed to which
topic and if they're currently online.
Related issue: p2panda/p2panda#1002
* Persisted Topics: Store to persist all previously used topics
Unfortunately we need to remove one feature for now which is to change
"connection modes", causing the network to be disabled / enabled.
`reflection-node` handled that well in the past by only disabling the
networking layer while keeping the event processing layer intact. We
will have to do something similar in `p2panda` to offer this feature as
well since the new API surface will not allow such low-level access
anymore.
See related issue: p2panda/p2panda#1093
BREAKING CHANGES
These changes introduce breaking changes to the core operation format
(`previous` field was removed) and the header extensions changed which
means that all previously created, persisted operations will _not_ be
compatible anymore.
THIS RESULTS IN LOOSING ALL PREVIOUSLY CREATED DOCUMENTS.
Since `p2panda` is not stable yet this might happen again in the near
future, though this was the most important and largest update and we're
much closer to a stable API and protocol now.
Future SQLite files will be written to a new path, the network id has
also changed now. This prevents "old" nodes talking to "new" ones,
similarily we allow users to retreive old documents from the database.
Final remarks:
Some minor changes in `reflection-doc` where also necessary, I
specifically decided to make `p2panda-core` a concrete dependency there
since too much functionality is used to justify re-exports from
`reflection-node`.
This will assure that users have a chance to recover documents from past Reflection versions and move them into this new version with a migration script, etc.
This ensures that we're not able to sync with Reflection users of an older version.
The new p2panda Node API brings timestamp out-of-the-box from ephemeral streams, we don't need to provide them ourselves anymore.
There's more stream processing related errors we want to expose to -docs from p2panda aka -node. To avoid confusion around two SubscriptionError types I've renamed one of them which was anyhow more "publishing" related. I'm considering renaming more types in a subsequent patch to make the PubSub relationship clearer.
Topic is used already all over the place in p2panda and indicates the
identifier of the topic itself (32 byte string). A stream on top of a
topic is a "topic stream", we can subscribe to it ("topic subscription")
and publish into it, following a PubSub pattern.
Reflection "enhances" these types with "tracking", that is, we know
which topics have been used and which authors are connected to it and
when they are online.
This hopefully clarifies the types a bit better and stays consistent
with naming in p2panda to avoid confusion:
Topic -> TrackedTopic
Author -> TrackedAuthor
Subscription -> TopicStream
SubscribableTopic -> TopicSubscription
adzialocha
force-pushed
the
adz/p2panda-high-level-api
branch
2 times, most recently
from
4d52d54 to
abfd6e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gist is: