From de5881111f344ff8760b749f73cf4d16ea2be348 Mon Sep 17 00:00:00 2001 From: Hugo Osvaldo Barrera Date: Mon, 31 Mar 2025 17:13:46 +0200 Subject: [PATCH 1/4] vdir: import specification Verbatim copy from the vdirsyncer documentation, adapted from ReStructuredText to Markdown. See: https://github.com/pimutils/vdirsyncer/blob/41b48857ebd7ad6d7b818295b86677fc84c13dae/docs/vdir.rst --- content/specs/vdir.md | 107 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 content/specs/vdir.md diff --git a/content/specs/vdir.md b/content/specs/vdir.md new file mode 100644 index 0000000..eccb0e8 --- /dev/null +++ b/content/specs/vdir.md @@ -0,0 +1,107 @@ +--- +layout: post +title: The Vdir storage format +date: 2025-03-31 +categories: specs +--- + +This document describes a standard for storing calendars and contacts on a +filesystem, with the main goal of being easy to implement. + +Vdirsyncer synchronizes to vdirs via [`filesystem`]. Each vdir (basically just +a directory with some files in it) represents a calendar or address book. + +[`filesystem`]: https://vdirsyncer.pimutils.org/en/stable/config.html#storage-filesystem + +# Basic Structure + +The main folder (root) contains an arbitrary number of subfolders +(collections), which contain only files (items). Synonyms for "collection" may +be "address book" or "calendar". + +An item is: + +- A [vCard] file, in which case the file extension *must* be `.vcf`, *or* +- An [iCalendar] file, in which case the file extension *must* be `.ics`. + +An item *should* contain a `UID` property as described by the vCard and +iCalendar standards. If it contains more than one `UID` property, the values +of those *must* not differ. + +The file *must* contain exactly one event, task or contact. In most cases this +also implies only one `VEVENT`/`VTODO`/`VCARD` component per file, but +e.g. recurrence exceptions would require multiple `VEVENT` components per +event. + +The filename should have similar properties as the `UID` of the file content. +However, there is no requirement for these two to be the same. Programs may +choose to store additional metadata in that filename, however, at the same time +they *must not* assume that the metadata they included will be preserved by +other programs. + +[vCard]: https://tools.ietf.org/html/rfc6350 +[iCalendar]: https://tools.ietf.org/html/rfc5545 + +# Metadata + +Any of the below metadata files may be absent. None of the files listed below +have any file extensions. + +- A file called `color` inside the vdir indicates the vdir's color, a + property that is only relevant in UI design. + + Its content is an ASCII-encoded hex-RGB value of the form `#RRGGBB`. For + example, a file content of `#FF0000` indicates that the vdir has a red + (user-visible) color. No short forms or informal values such as `red` (as + known from CSS, for example) are allowed. The prefixing `#` must be + present. + +- Files called `displayname` and `description` contain a UTF-8 encoded label/ + description, that may be used to represent the vdir in UIs. + +- A file called `order` inside the vdir includes the relative order + of the calendar, a property that is only relevant in UI design. + +# Writing to vdirs + +Creating and modifying items or metadata files *should* happen [atomically]. + +[atomically]: https://en.wikipedia.org/wiki/Atomicity_%28programming%29 + +Writing to a temporary file on the same physical device, and then moving it to +the appropriate location is usually a very effective solution. For this +purpose, files with the extension `.tmp` may be created inside collections. + +When changing an item, the original filename *must* be used. + +# Reading from vdirs + +- Any file ending with the `.tmp` or no file extension *must not* be treated + as an item. + +- The `ident` part of the filename *should not* be parsed to improve the + speed of item lookup. + +# Considerations + +The primary reason this format was chosen is due to its compatibility with the +[CardDAV] and [CalDAV] standards. + +[CardDAV]: http://tools.ietf.org/html/rfc6352 +[CalDAV]: http://tools.ietf.org/search/rfc4791 + +## Performance + +Currently, vdirs suffer from a rather major performance problem, one which +current implementations try to mitigate by building up indices of the +collections for faster search and lookup. + +The reason items' filenames don't contain any extra information is simple: The +solutions presented induced duplication of data, where one duplicate might +become out of date because of bad implementations. As it stands right now, an +index format could be formalized separately though. + +vdirsyncer doesn't really have to bother about efficient item lookup, because +its synchronization algorithm needs to fetch the whole list of items anyway. +Detecting changes is easily implemented by checking the files' modification +time. From 680342b1c6227830af4e301b3299f8d132b2998c Mon Sep 17 00:00:00 2001 From: Hugo Osvaldo Barrera Date: Mon, 31 Mar 2025 17:23:31 +0200 Subject: [PATCH 2/4] vdir: various clarifications and rewording --- content/specs/vdir.md | 152 ++++++++++++++++++++++++++++-------------- 1 file changed, 102 insertions(+), 50 deletions(-) diff --git a/content/specs/vdir.md b/content/specs/vdir.md index eccb0e8..cd6885a 100644 --- a/content/specs/vdir.md +++ b/content/specs/vdir.md @@ -5,103 +5,155 @@ date: 2025-03-31 categories: specs --- -This document describes a standard for storing calendars and contacts on a -filesystem, with the main goal of being easy to implement. - -Vdirsyncer synchronizes to vdirs via [`filesystem`]. Each vdir (basically just -a directory with some files in it) represents a calendar or address book. - -[`filesystem`]: https://vdirsyncer.pimutils.org/en/stable/config.html#storage-filesystem +This specification describes a convention for storing calendars and address +books in a filesystem, with the main goal of being ease of implementation, and +compatibility with existing standards. # Basic Structure -The main folder (root) contains an arbitrary number of subfolders -(collections), which contain only files (items). Synonyms for "collection" may -be "address book" or "calendar". +A main (root) directory contains an arbitrary number of subdirectories. Each +subdirectory is a collection (an address book or a calendar). Each of these +collections contains items, which may be contacts or calendar components +(events, todos and/or journal entries) respectively. + +An item may only be one of the following two types: -An item is: +- A [vCard] file, in which case the file extension MUST be `.vcf`. +- An [iCalendar] file, in which case the file extension MUST be `.ics`. -- A [vCard] file, in which case the file extension *must* be `.vcf`, *or* -- An [iCalendar] file, in which case the file extension *must* be `.ics`. +[vCard]: https://tools.ietf.org/html/rfc6350 +[iCalendar]: https://tools.ietf.org/html/rfc5545 -An item *should* contain a `UID` property as described by the vCard and -iCalendar standards. If it contains more than one `UID` property, the values -of those *must* not differ. +An item SHOULD contain a `UID` property as described by the vCard and iCalendar +specifications. If it contains more than one `UID` property, the values of +those MUST NOT differ. -The file *must* contain exactly one event, task or contact. In most cases this +The file MUST contain exactly one event, task or contact. In most cases this also implies only one `VEVENT`/`VTODO`/`VCARD` component per file, but -e.g. recurrence exceptions would require multiple `VEVENT` components per -event. +e.g. recurrence exceptions require multiple `VEVENT` components per event. + +## Filenames The filename should have similar properties as the `UID` of the file content. However, there is no requirement for these two to be the same. Programs may choose to store additional metadata in that filename, however, at the same time -they *must not* assume that the metadata they included will be preserved by -other programs. +they MUST NOT assume that the metadata they included will be preserved by other +programs. -[vCard]: https://tools.ietf.org/html/rfc6350 -[iCalendar]: https://tools.ietf.org/html/rfc5545 +Filenames SHOULD be URL-safe, such that can be retained when synchronising to +WebDAV collections. # Metadata -Any of the below metadata files may be absent. None of the files listed below -have any file extensions. +Any following metadata files may optionally be present. None of the files +listed below have any file extensions. -- A file called `color` inside the vdir indicates the vdir's color, a +- A file called `color` inside the vdir indicates that collection's color, a property that is only relevant in UI design. Its content is an ASCII-encoded hex-RGB value of the form `#RRGGBB`. For example, a file content of `#FF0000` indicates that the vdir has a red (user-visible) color. No short forms or informal values such as `red` (as - known from CSS, for example) are allowed. The prefixing `#` must be + known from CSS, for example) are allowed. The prefixing `#` MUST be present. -- Files called `displayname` and `description` contain a UTF-8 encoded label/ - description, that may be used to represent the vdir in UIs. +- A file called `displayname` contain a UTF-8 encoded label description that + is suitable for presentation to the user. This property is a counterpart to + the `DAV:displayname` WebDAV property defined in [rfc4918 section 15.2]. + +[rfc4918 section 15.2]: https://www.rfc-editor.org/rfc/rfc4918#section-15.2 + +- A `description` contain a UTF-8 encoded description, which provides a + human-readable description of a calendar or an address book. This property is + a counterpart to the [`CALDAV:calendar-description`][ccd] and + [`CARDDAV:addressbook-description`][cad] properties in CalDAV and CardDAV + respectively. + +[ccd]: https://www.rfc-editor.org/rfc/rfc4791#section-5.2.1 +[cad]: https://www.rfc-editor.org/rfc/rfc6352#section-6.2.1 -- A file called `order` inside the vdir includes the relative order - of the calendar, a property that is only relevant in UI design. +- A file called `order` inside the vdir includes the relative order of a + collection. This property is only use for presentational purposes. Its value + shall be a numeric string, and collections shall be sorted relative to this + value. + +These properties and their names were chosen to match existing CalDAV and +CardDAV properties, given that a `vdir` is commonly synchronised using one of +these two protocols. + +Property files with an invalid content SHOULD be ignored. Application software +SHOULD warn the user of the invalidity when appropriate. # Writing to vdirs -Creating and modifying items or metadata files *should* happen [atomically]. +Creating and modifying items or metadata files MUST happen [atomically]. That +is, a file matching one of the extensions mentioned above must never be +partially written; if it exists, it must be complete. [atomically]: https://en.wikipedia.org/wiki/Atomicity_%28programming%29 +This requirement has two purposes: + +- Ensure that one program will never read an item which has only partially been + written by another tool. +- Ensure that a fatal system crash shall never result in a corrupt item. + Writing to a temporary file on the same physical device, and then moving it to -the appropriate location is usually a very effective solution. For this -purpose, files with the extension `.tmp` may be created inside collections. +the appropriate location is usually an effective solution. For this purpose, +files with the extension `.tmp` may be created inside collections. -When changing an item, the original filename *must* be used. +When changing an item, the original filename MUST be used. # Reading from vdirs -- Any file ending with the `.tmp` or no file extension *must not* be treated - as an item. +- Any file ending with the `.tmp` or no file extension MUST NOT be treated as + an item. -- The `ident` part of the filename *should not* be parsed to improve the - speed of item lookup. +- The `ident` part of the filename SHOULD NOT be parsed to improve the speed + of item lookup. # Considerations The primary reason this format was chosen is due to its compatibility with the [CardDAV] and [CalDAV] standards. -[CardDAV]: http://tools.ietf.org/html/rfc6352 -[CalDAV]: http://tools.ietf.org/search/rfc4791 +[CardDav]: https://www.rfc-editor.org/rfc/rfc6352 +[CalDAV]: https://www.rfc-editor.org/rfc/rfc4791 ## Performance -Currently, vdirs suffer from a rather major performance problem, one which +Currently, a vdir suffer from a rather major performance problem, one which current implementations try to mitigate by building up indices of the collections for faster search and lookup. -The reason items' filenames don't contain any extra information is simple: The -solutions presented induced duplication of data, where one duplicate might -become out of date because of bad implementations. As it stands right now, an -index format could be formalized separately though. +Item filenames convey not additional information within the scope of this +specification. Any attempt to encode information in these results in +duplication of data, where one duplicate might become out of date because of +bad implementations. As it stands right now, an index format could be +formalized separately though. + +Synchronisation tools such as [vdirsyncer] or [pimsync] don't really have to +bother about efficient item lookup, because its synchronization algorithm needs +to fetch the whole list of items anyway. Detecting changes is easily +implemented by checking the files' modification time. + +[vdirsyncer]: https://vdirsyncer.pimutils.org/en/stable/ +[pimsync]: https://pimsync.whynothugo.nl/ + +# History + +## Version 1.0.0 + +The original version of this was published as part of [the vdirsyncer +documentation]. However, the conventions described in this document are +intended to be adopted by other command line and desktop tools operating on +calendar and address book collections. + +[the vdirsyncer documentation]: https://vdirsyncer.pimutils.org/en/stable/vdir.html + +## Version 1.1.0 (2025-03-31) -vdirsyncer doesn't really have to bother about efficient item lookup, because -its synchronization algorithm needs to fetch the whole list of items anyway. -Detecting changes is easily implemented by checking the files' modification -time. +- Publish this specification separately. +- Several minor clarifications and rewording. +- Recommend for item filenames to be URL-safe. +- Specify that properties with invalid data should be ignored. From 51fe42b23be55a05d17bcd1606fd8e68182405f7 Mon Sep 17 00:00:00 2001 From: Hugo Osvaldo Barrera Date: Mon, 31 Mar 2025 17:23:45 +0200 Subject: [PATCH 3/4] vdir: link to spec from home page --- content/_index.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/content/_index.md b/content/_index.md index 61ca869..bcf187c 100644 --- a/content/_index.md +++ b/content/_index.md @@ -4,3 +4,11 @@ pimutils is a suite of programs for managing contacts, calendars and tasks on th * [todoman](https://github.com/pimutils/todoman) is a command line interface tasks manager * [vdirsyncer](https://github.com/pimutils/vdirsyncer) can synchronize calendar data, including data from khal and todoman * [pimsync](https://pimsync.whynothugo.nl/) is a re-implementation of vdirsyncer addressing outstanding issue and limitations. + +These tools are build around existing standards formats and protocols. +The `vdir` storage format also builds atop existing standards: + +* [vdir] is a convention for storing calendars and address books in a + filesystem. + +[vdir]: http://localhost:1313/specs/vdir/ From e6e184ca6b0d5169f39e2385f86758a98e3f611e Mon Sep 17 00:00:00 2001 From: Hugo Osvaldo Barrera Date: Mon, 31 Mar 2025 17:24:13 +0200 Subject: [PATCH 4/4] Update copyright year --- layouts/_default/baseof.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/_default/baseof.html b/layouts/_default/baseof.html index 2878a8b..7383c9e 100644 --- a/layouts/_default/baseof.html +++ b/layouts/_default/baseof.html @@ -35,7 +35,7 @@

pimutils

{{ block "main" . }} Content for each page will end up here. {{ end }}