Graph projections

A graph projection projects events onto a property graph: nodes, edges and their properties. It works like a projection onto a document read model. It uses the same observer, the same event routing, the same key and property expressions, the same replay and the same failed partition handling. The difference is the target. Instead of folding events into one document per key, it turns each event into a few idempotent operations on nodes and edges.

The graph is what the events leave behind. Nothing writes to it directly.

Nodes and edges

A node has a label, a unique key and properties. The label is the name of the type. A property is a primitive or an array of primitives. An object property is an error: model the object as a node of its own and connect it with an edge.

An edge connects two nodes. It has a type, such as MEMBER_OF, a source node, a target node and properties. By default an edge is identified by its type, source and target, so there is at most one edge of a type between two nodes. An edge type can name an identifying property, which allows several edges of the same type between the same two nodes.

A graph is named. Put [Graph("Organization")] on the types or on the assembly. Without it, the graph is named after the assembly’s root namespace.

Model-bound

Mark the types with [Node] and [Edge] and use the model-bound projection attributes on them. [Key] on a node marks its unique key. On an edge, [Source<>] and [Target<>] mark the members that hold the keys of the end nodes. Those members are not properties of the edge.

C#
Elixir

[Graph("Organization")]
[Node]
[FromEvent<PersonRegistered>]
[RemovedWith<PersonRemoved>(detach: true)]
public record Person([Key] Guid PersonId, string Name);

[Graph("Organization")]
[Node]
[FromEvent<GroupCreated>]
public record Group([Key] Guid GroupId, string Name);

[Graph("Organization")]
[Edge("MEMBER_OF")]
[EdgeFrom<MemberAdded>(sourceKey: nameof(MemberAdded.PersonId), targetKey: nameof(MemberAdded.GroupId))]
[EdgeRemovedWith<MemberRemoved>(sourceKey: nameof(MemberRemoved.PersonId), targetKey: nameof(MemberRemoved.GroupId))]
public record MemberOf(
    [Source<Person>] Guid PersonId,
    [Target<Group>] Guid GroupId,
    [SetFromContext<MemberAdded>(nameof(EventContext.Occurred))] DateTimeOffset Since);

View C# snippet source on GitHub

defmodule MyApp.Events.PersonRegistered do
  use Chronicle.Events.EventType, id: "graph-person-registered"
  defstruct person_id: "", name: ""
end

defmodule MyApp.Events.PersonRemoved do
  use Chronicle.Events.EventType, id: "graph-person-removed"
  defstruct person_id: ""
end

defmodule MyApp.Events.GroupCreated do
  use Chronicle.Events.EventType, id: "graph-group-created"
  defstruct group_id: "", name: ""
end

defmodule MyApp.Events.MemberAdded do
  use Chronicle.Events.EventType, id: "graph-member-added"
  defstruct person_id: "", group_id: ""
end

defmodule MyApp.Events.MemberRemoved do
  use Chronicle.Events.EventType, id: "graph-member-removed"
  defstruct person_id: "", group_id: ""
end

defmodule MyApp.Graphs.Person do
  use Chronicle.Graphs.Node
  defstruct person_id: "", name: ""
end

defmodule MyApp.Graphs.Group do
  use Chronicle.Graphs.Node
  defstruct group_id: "", name: ""
end

defmodule MyApp.Graphs.MemberOf do
  use Chronicle.Graphs.Edge, type: "MEMBER_OF"
  defstruct since: ""
end

defmodule MyApp.Graphs.Organization do
  use Chronicle.Graphs.Graph, id: "Organization"

  alias MyApp.Events.{GroupCreated, MemberAdded, MemberRemoved, PersonRegistered, PersonRemoved}
  alias MyApp.Graphs.{Group, MemberOf, Person}

  node Person, key: :person_id do
    from PersonRegistered, key: :person_id, set: [name: :name]
    removed_with PersonRemoved, key: :person_id, detach: true
  end

  node Group, key: :group_id do
    from GroupCreated, key: :group_id
  end

  edge MemberOf, source: Person, target: Group do
    from MemberAdded,
      source_key: :person_id,
      target_key: :group_id,
      set: [since: {:event_context, :occurred}]

    removed_with MemberRemoved, source_key: :person_id, target_key: :group_id
  end
end

View Elixir snippet source on GitHub

Properties are mapped by name from an event that has a property of the same name, or explicitly with the attributes you already know: [SetFrom<>], [SetFromContext<>], [Increment<>], [Decrement<>], [Count<>], [AddFrom<>] and [SubtractFrom<>]. The node key is the event source id unless [FromEvent<>] names an event property.

Fluent

The same graph can be declared with a fluent builder. It produces exactly the same definition as the attributes.

C#
Elixir

[Graph("Organization")]
public class OrganizationGraph : IGraphProjection
{
    public void Define(IGraphProjectionBuilder builder) => builder
        .Node<Person>(n => n
            .IdentifiedBy(p => p.PersonId)
            .From<PersonRegistered>()
            .RemovedWith<PersonRemoved>(b => b.Detach()))
        .Node<Group>(n => n
            .IdentifiedBy(g => g.GroupId)
            .From<GroupCreated>())
        .Edge<MemberOf, Person, Group>(e => e
            .From<MemberAdded>(b => b
                .UsingSourceKey(x => x.PersonId)
                .UsingTargetKey(x => x.GroupId)
                .Set(m => m.Since).ToEventContextProperty(c => c.Occurred))
            .RemovedWith<MemberRemoved>(b => b
                .UsingSourceKey(x => x.PersonId)
                .UsingTargetKey(x => x.GroupId)));
}

View C# snippet source on GitHub

Elixir does not support this workflow yet.

View Elixir snippet source on GitHub

Counters

Counters work on node and edge properties.

C#
Elixir

[Node]
[FromEvent<GroupCreated>]
public record Group(
    [Key] Guid GroupId,
    string Name,
    [Increment<MemberAdded>] [Decrement<MemberRemoved>] int MemberCount);

View C# snippet source on GitHub

defmodule MyApp.Events.TeamFormed do
  use Chronicle.Events.EventType, id: "graph-team-formed"
  defstruct team_id: "", name: ""
end

defmodule MyApp.Events.TeamMemberJoined do
  use Chronicle.Events.EventType, id: "graph-team-member-joined"
  defstruct team_id: ""
end

defmodule MyApp.Events.TeamMemberLeft do
  use Chronicle.Events.EventType, id: "graph-team-member-left"
  defstruct team_id: ""
end

defmodule MyApp.Graphs.Team do
  use Chronicle.Graphs.Node
  defstruct team_id: "", name: "", member_count: 0
end

defmodule MyApp.Graphs.Teams do
  use Chronicle.Graphs.Graph, id: "Teams"

  alias MyApp.Events.{TeamFormed, TeamMemberJoined, TeamMemberLeft}
  alias MyApp.Graphs.Team

  node Team, key: :team_id do
    from TeamFormed, key: :team_id
    from TeamMemberJoined, key: :team_id, increment: :member_count
    from TeamMemberLeft, key: :team_id, decrement: :member_count
  end
end

View Elixir snippet source on GitHub

Edges with an identity

Give an edge a [Key] member to identify it among edges of the same type between the same two nodes. Every mapping of the edge must then say which event property holds the identity, with the identity argument or UsingIdentity. A sink that cannot hold several edges between the same pair of nodes rejects such an edge type.

C#
Elixir

[Edge("VISITED")]
[EdgeFrom<PlaceVisited>(
    sourceKey: nameof(PlaceVisited.PersonId),
    targetKey: nameof(PlaceVisited.PlaceId),
    identity: nameof(PlaceVisited.VisitId))]
public record Visited(
    [Source<Person>] Guid PersonId,
    [Target<Place>] Guid PlaceId,
    [Key] Guid VisitId,
    [Count<PlaceVisited>] int Times);

View C# snippet source on GitHub

defmodule MyApp.Events.CustomerRegistered do
  use Chronicle.Events.EventType, id: "graph-customer-registered"
  defstruct customer_id: ""
end

defmodule MyApp.Events.ShopRegistered do
  use Chronicle.Events.EventType, id: "graph-shop-registered"
  defstruct shop_id: ""
end

defmodule MyApp.Events.ShopVisited do
  use Chronicle.Events.EventType, id: "graph-shop-visited"
  defstruct customer_id: "", shop_id: "", visit_id: ""
end

defmodule MyApp.Graphs.Customer do
  use Chronicle.Graphs.Node
  defstruct customer_id: ""
end

defmodule MyApp.Graphs.Shop do
  use Chronicle.Graphs.Node
  defstruct shop_id: ""
end

defmodule MyApp.Graphs.Visited do
  use Chronicle.Graphs.Edge, type: "VISITED"
  defstruct visit_id: "", times: 0
end

defmodule MyApp.Graphs.Visits do
  use Chronicle.Graphs.Graph, id: "Visits"

  alias MyApp.Events.{CustomerRegistered, ShopRegistered, ShopVisited}
  alias MyApp.Graphs.{Customer, Shop, Visited}

  node Customer, key: :customer_id do
    from CustomerRegistered, key: :customer_id
  end

  node Shop, key: :shop_id do
    from ShopRegistered, key: :shop_id
  end

  # The visit identifies the edge, so a customer can have several VISITED edges to one shop.
  # Every event of the edge reads the visit from the event property of the same name, or from
  # the property `identity:` names.
  edge Visited, source: Customer, target: Shop, identified_by: :visit_id do
    from ShopVisited, source_key: :customer_id, target_key: :shop_id, count: :times
  end
end

View Elixir snippet source on GitHub

The operations

A graph projection does not produce state diffs. For every event it produces a GraphChangeset, a list of five idempotent operations:

Operation	What it does
`UpsertNode`	Creates the node when it does not exist and sets its properties.
`SetProperties`	Sets properties on a node or an edge that exists. Nothing is created.
`DeleteNode`	Deletes a node. With `detach`, the edges attached to it are deleted too; without, a node that still has edges cannot be deleted.
`UpsertEdge`	Creates the edge when it does not exist and sets its properties.
`DeleteEdge`	Deletes an edge. The nodes at both ends stay.

Some rules follow from the operations:

Edges ensure their nodes. An edge operation creates a bare end node, without properties, when it does not exist. A node mapping completes it later.
A projection owns what it declares. It owns the edges and the node properties it maps, and it never deletes a node it did not declare.
Counters carry absolute values. The engine reads the current value of the node or edge from the sink, applies the counter, and the operation carries the result.
Removing a node needs an explicit decision. RemovedWith deletes the node. Set detach to delete its edges with it.

Redelivery and replay

Every node and edge stores the watermark: the sequence number of the event that last wrote it. A write whose event sequence number is at or below the watermark is skipped. Redelivered and out-of-order events therefore do not apply twice. A bare node that only an edge has created has no watermark yet, so its node mapping can still write it.

Replay is mark and sweep. A replay starts a new generation for the replaying projection. An element of the projection that the replay has not written yet reads as absent, so counters are rebuilt from zero instead of counted on top of the previous result, and the first write replaces the element. When the replay ends, every element of the projection that was not written is deleted. Elements that other projections wrote, and nodes that only edges created, are never touched. A stale node that edges of other projections still point at stays as a bare node.

Choosing the sink

A graph is stored by a graph sink. The sink type is set per client with DefaultGraphSinkTypeId, which defaults to Neo4j.

C#
Elixir

services.Configure<ChronicleOptions>(options =>
{
    // Neo4j is the default. The InMemoryGraph sink keeps the graph in the kernel's memory.
    options.DefaultGraphSinkTypeId = WellKnownSinkTypes.InMemoryGraph;
});

View C# snippet source on GitHub

{Chronicle.Client,
  connection_string: "chronicle://localhost:35000",
  event_store: "my-store",
  # Neo4j is the default. The in-memory graph sink keeps the graph in the kernel's memory.
  default_graph_sink_type_id: :in_memory_graph,
  graphs: [MyApp.Graphs.Organization]}

View Elixir snippet source on GitHub

Read the sinks overview for the available sink types, and Neo4j graph sink for the default.

Writing a graph sink

A sink implements IGraphSink, separate from the ISink of document read models, because the fold semantics of ISink do not apply. A graph sink:

Applies a GraphChangeset with ApplyGraphChanges, subject to the watermark rule, and applies the operations of one changeset atomically.
Stamps every element it writes with the owning projection and its replay generation, and implements BeginReplay, ResumeReplay, EndReplay and LeaveReplay as mark and sweep.
Reports an element of the replaying owner that has not been rewritten as absent from FindNode and FindEdge.
Creates its constraints and indexes from the graph definition in EnsureSchema.
Reads instances for the Workbench with GetNodes and GetEdges.

The in-memory graph sink is the executable reference for these semantics, and every behavior has a specification.

Testing

GraphScenario runs a graph projection against the in-memory graph sink, so a specification can append events and assert the nodes and edges.

C#
Elixir

using var scenario = new GraphScenario();

await scenario.Append(personId, new PersonRegistered(personId, "Ada"));
await scenario.Append(personId, new MemberAdded(personId, groupId));

var person = await scenario.Node<Person>(personId);
var membership = await scenario.Edge<MemberOf>(personId, groupId);
var memberships = await scenario.Edges<MemberOf>();

View C# snippet source on GitHub

Elixir does not support this workflow yet.

View Elixir snippet source on GitHub

Analyzers

The code analysis package reports a graph declaration that cannot work:

CHR0066: a node or an edge has an object property.
CHR0067: a node does not have exactly one key, or an edge has more than one.
CHR0068: a type is both a node or an edge and a read model.