docs(auditlogs): describe guarantee of delivery in appendix

Author: mlenkeit

projects/audit-logging.md

@@ -68,4 +68,42 @@ TODO: add link
 
 ## SIG Meetings and Other Info
 
-TODO: add information
+TODO: add information
+
+## Appendix
+
+### Appendix A: Guarantee of Delivery
+
+In the context of this document, guarantee of delivery describes the ability of delivering audit logs from source to destination through OTel means while ensuring that all such signals arrive at the destination and/or providing the source with a means to handle failed delivery.
+
+Messaging protocols that support different levels of delivery guarantees may refer to this behavior as _at least once_ or _exactly once_, as opposed to _at most once_.
+
+We assume that every component that is involved in the delivery of audit logs from source to destination must support guarantee of delivery individually, rather than assuming that this ability can be provided by e.g. only the collector or SDK.
+
+The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel collector, and a durable storage. The workload acts as the source and produces audit logs via the OTel SDK/API. It writes the data via OTLP to the OTel collector. The OTel collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket.
+
+The following implications would apply:
+
+- workload produces an audit-relevant event:
+
+  The workload emits the event via the OTel SDK/API. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc.
+
+- OTel collector receives the event:
+
+  To ensure that the event is not lost even if the collector process is terminated or crashes, the collector may need to persist the event before acknowledging receipt to the workload or SDK. If the event cannot be persisted, receipt must be rejected.
+
+- OTel collector exports the event:
+
+  Once the event is exported and the target (i.e. S3) acknowledges receipt, the event can dropped from the collector's persistence.
+
+- S3 receives the event:
+
+  Acknowledges receipt after persisting the event.
+
+  Note that this is outside the scope of the OTel. More general, when using OTel for audit logging purposes, it's the users (e.g. Ops) responsibility to configure a suitable export target.
+
+Note that this example may contain implementation details for illustration purposes. The actual implementation may differ as long as the requirements are met.
+
+The example is kept simple for illustration purposes. Many edge cases need to be discussed by the SIG, such as batch-sending of signals or handling of multiple export targets.
+
+It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes.