Open full view…

Smarter Entity Synchronization With Scrivener

Fri, 07 Jul 2017 15:14:08 GMT

The goal of synchronising Aeon Timeline with Scrivener is to reduce or eliminate redundant data entry between the two programs. This works quite well for events, but AT2 sort of goes down its own path when it comes to entities. Entities are only supported as meta-data for events. Character development is a huge part of writing, but I think the heavy lifting is largely done in Scrivener. The pain point for me with AT2/Scrivener synchronisation is with simple operations such as character renames. It's fairly common for an author to start out with a placeholder name like "Team Captain" and then decide later that his name is "Wayne Gretzky". If you are at all like me, your characters morph a lot during development. Currently, there's no connection between the characters we create in Scrivener and the character entities in AT2. The EVENT PARTICIPANT meta information has no association with our characters. Our simple character rename example above becomes a manual multi-step update between the two programs. Currently, I think I spend more time doing manual edits because of dupes and rename problems than I save through synch automation for entities. Using the novel template, an empty Scrivener document has a default folder for Characters and Places. This seems to be begging for some sort of automation for AT2 entity synchronisation. I'm not suggesting that we add a metric tonne of metadata or try to handle characters, locations or arcs with extensive schema extensions. What I believe we mostly need is a way to link an entity in AT2 with a corresponding document in the Scrivener binder. If AT2 were able to auto-discover and auto-create a list of entities based on the user designating a folder name, this would be a very nice bonus. I'm also not suggesting that this is trivial as there are definitely complexities here even for simple CRUD operations (Create, Read, Update, Delete). There are going to be quirks and probably some best practices may have to be identified. I thought it would be appropriate to post in the forum so we could perhaps discuss what people would like to see and for us to identify specific challenges. Specific functionality that I'd like to see: * Auto discovery * - The user tells AT2 to look in the "Characters" folder. AT2 descends into that folder and any sub-folders and flags any character documents that don't have associated entities. There should be a right click menu item "Create Entity". AT2 needs to know to create the right type of entity, based on the parent folder in the binder (char vs place vs arc). This should be generic so that the user could create a custom entity type and associate a folder with it in the binder. * Flag Unlinked Entities * - For characters, you probably want to create them in Scrivener so you can apply a template to the document, but if you do create the character as an entity in AT2, it should flag that the new entity is not associated with a document. * Synchronise Scrivener Synopsis with Entity Notes * - Entity notes are hidden, but I think they should be visible in AT2 view headers when you group by entity (requested as a separate feature). To go along with this, the notes should be synched with the object synopsis. * Rename synchronisation * - Changing the name of the entity in AT2 or changing the document (character) name in the Scrivener binder updates the other program. Same rename functionality we have for Events now. * Remove Auto Entity Creation From Meta-Data * - Currently if you add a character to an event's metadata, AT2 checks to see if that entity exists and automatically creates it on sync if it was not present. I believe that if we add the synch mechanism proposed above, this automatic creation is going to cause more grief than benefit. I already find myself getting duplicate entities because I type "Main St." or "Main St" in EVENT LOCATION instead of "Main Street". There's too much onus on the author to remember naming conventions, punctuation, spelling, etc... We're notoriously inconsistent people.

Matthew Tobin
Wed, 12 Jul 2017 00:32:45 GMT

Hi, I think we may have discussed this a little over email already, but for the benefit of the public conversation, I will repeat the limitations that I expressed in that email conversation. We have had a few people asking about increased syncing of "Characters" or "Places" before, but unfortunately, because of the huge flexibility provided by Scrivener, it is very difficult to come up with a model that works for a majority of people. Most of the limitations come from Scrivener itself: - Scrivener does not actually have a concept of "Character" or "Place". - While it is true that one of their starting templates contains folders called "Characters" and "Places", there is really nothing special about these folders, and nothing that sets them apart from any other folder inside Scrivener. - Likewise, the documents placed within the folders are standard free-form text documents. They do not have a fixed structure that would allow me to read or write any information in the documents themselves. -Finally, everything inside a Scrivener project (all folders, all text document s, whether they are inside the Manuscript or Characters or Places or Research folders, and even the folders themselves) shares the same set of custom meta-data fields. So it is not possible for me to create one set of custom metadata fields for characters and another for scenes, for instance. Those limitations make it somewhat difficult to find a generic approach that suits everyone. I am happy to hear other people's ideas here, of course, to see if there is a common ground that suits everyone. Matt

Wed, 12 Jul 2017 02:14:58 GMT

I believe that you may be overthinking the problem and trying to do too much. The power in Aeon Timeline 2 for authors is how it gives us the ability to view the relationships between entities and events. This helps us plan timelines, ensure that characters aren't in two places at the same time, relevant characters move logically between arcs, etc... This is largely a visual process. It does not require knowledge that the butler has red hair, only that he was in the library with Colonel Mustard on the night of the murder. Details that are buried in Scrivener character description documents are irrelevant in the timeline. Nor does the character document need to be aware of references to events or other entities such as places. Scrivener is where we do the heavy lifting with regard to detailed character descriptions in what is essentially free-form data that no two authors are going to agree on a format for. In fact, I modify my character template somewhat between projects. I'm not sure why I would want all that clutter carried into a timeline. We don't nee d Scrivener to be aware of entity types. Entities are purely an AT2 concept. Event associations through the extended meta-data are quite foreign to Scrivener and I would say unwanted. Our demands on Scrivener, with regard to entities, are rudimentary: - The author can designate any folder to contain characters, places or any other potential entity type. AT2 just needs to know which one is the parent folder. - AT2 doesn't need any data from the free-form text documents in the binder other than name and synopsis/description. These fields already exist in both Scrivener and AT2. No schema extensions needed in either program. - Since we're not exchanging additional meta-data, we don't care what the schema for the entity objects looks like. It can be the same as the documents used for events. We're ignoring everything except two fields anyway and those two fields already exist. In fact, we've already extended the schema for those character documents because Scrivener treats them identically to our event documents. Scrivener is equally ignorant of what an event is. So the list of capabilities is actually quite short and not reliant or restricted by any of the limitations you listed. Although your list of restrictions is valid if you wanted to attempt something much more complex (and I would argue, less useful to authors). - Provide a way to quickly identify one parent folder for each entity type so that the "logical character" in the Scrivener binder is bound to an "entity character" in AT2 (ie using the correct AT2 entity type). This is just so adding characters or places is done with the fewest clicks and we're not having to mess around in the entity manager. I suppose this is optional, but it would simply things for the author/user and I don't think the code burden is heavy to provide this convenience. - Ensure that titles and notes/synopsis fields are updated for entities so renames or note changes are reflected across both programs. This likely would use the same logic/mechanism that already exists for event synch. - Identify unlinked entities so the author can see that the characters, locations (or potentially other entities as defined by the author) are in step between Scrivener and AT2. This functionality already exists for events. - Bonus wish, allow entity notes to be viewed outside the entity manager (ie in the header when grouping by entity in either view). So in short, give us very rudimentary and generic linking between documents and entities and let us as authors deal with the devilish details that are buried in our Scrivener docs outside of the timeline.

Matthew Tobin
Wed, 12 Jul 2017 02:21:54 GMT

Yes, I understand that what you are asking for somewhat takes into account the limitations I mentioned above, and that they do not strictly apply to your requests. But since this is a public forum and other users may wish to contribute their own ideas, I thought it was worth making those limitations/restrictions clear from the outset as they may impact on suggestions that others choose to make. We have had users request "character syncing" before in ways that did conflict with the limitations mentioned in my first post. That said, syncing character information in this way has been a somewhat niche request at this point - there are other features that are requested much more frequently. Regardless, it is something on our development list to consider down the track. Matt

Wed, 12 Jul 2017 02:28:56 GMT

I'd really like to hear from other Scrivener users as to how they are managing locations and character sync, or whether they're just doing it all manually. I'm a bit surprised that this is a niche request. I've modified my workflow a couple of times in the last few weeks, first trying to use the automation that was recently added for entity creation and then switching back again to a fully manual process. The latter seems less error prone over the life of a project. Thanks for everything you're doing with AT2 and Scrivener. It's much appreciated.

Wed, 12 Jul 2017 02:52:05 GMT

Before using AT, I had assumed that one could not only sync events to Scrivener documents, but also sync entities to documents. Of course, not the case, so I manually work around the limitation. I'd certainly like to be able to do at least the first three of razyr's features, but a simpler functionality would be fine as well...i.e., I would be delighted if entities (any entity, don't care about entity typing or auto lookups etc.) could be synced to a Scrivener document, with AT telling me if an item was unlinked (razor's feature number 2) and let me see the scrivener synopsis in AT (feature 3).

Thu, 20 Jul 2017 11:43:22 GMT

I've only recently bought AT2 (last two days) and after my first sync with Scrivener was surprised that characters and places weren't synced. After reading this post I now understand some of the complexities. Make no mistake, I think AT2 is a fantastic product and the whole syncing thing is an added bonus. Having said that, it would be great if characters and places could be synced even if only in one direction i.e. from AT2 to Scrivener (and then manually moved and modified) and even if it's only a one off i.e. on create. I agree with @razyr in that Scrivener should do the heavy lifting in this regard. Perhaps start with something simple and add to it later if required. Great software, keep up the good work.

Fri, 04 Aug 2017 14:29:55 GMT

The more I think about this, the more I'm convinced that any sort of auto-discovery or auto-creation of entities is a bad idea. The core functionality which is really needed is the option to assign an arbitrary document or folder to any arbitrary entity of any type and have the Name and Summary/Synopsis synched for essential CRUD operations. Nothing can be safely assumed about hierarchy or inheritance. If we use Scrivener for planning, capturing massive amounts of information and writing then use AT2 for visualisation, we get the best both programs have to offer. The beauty of AT2 and Scrivener as independent tools is their flexibility in how we approach our work. I don't have to learn to think like Matt or read a book called "The Aeon Way" before I use his software. The same can be said for Scrivener, although it's a real work horse and understanding all of the features can be a steeper learning curve. We should expect to give up some of this flexibility as users when we bring two unrelated pieces of software together and expect them to play nicely. I belie ve the problems start when writers use AT2 as an outliner and then expect it to gracefully accept all of the extraneous information that pops into their head. Even if you can shoehorn the extra information into the timeline, AT2 is primarily a timeline program and it's sync focuses on time-based events. This shouldn't be a shock to anyone. Scrivener is ignorant of time, so right off we have two programs that have very different data architectures and back end storage systems. No two writers are likely to organise their binder with the same structure. We have really set the AT2 developers with an impossible task when we start asking for full character information sync. With some analysis, we may discover that it's unnecessary and perhaps unwanted. I think we should start with a very basic model which empowers writers to do work efficiently with minimal shared information, yet preserves the value of sync and AT2 as a visualisation tool. As writers, we need to be using the correct tool for various tasks and not expect AT2 sync to magically mirror details in AT2 which we really only need in Scrivener (or vice versa). We need to adjust our expectations for sync and make sure we're using the best tool for data input in various phases of our projects. We have to accept that there are limitations to what is going to possible (or desirable). One aspect of this forum is that it provides a platform for AT2 users to share ideas. If you are struggling with synchronization, rather than asking for new features from the devs, try asking other users how they would approach it.

Mon, 02 Oct 2017 12:32:52 GMT

I agree! After taking on the rather steep learning curve in Aeon, I am astonished by my new superpowers-- really? my protagonist was 3 years old when that happened? Shocking! And I am also shocked and delighted when notes that I write in the event summary field in Aeon automagically populate the Scrivener synopsis-- a field that I have never used because I find it fiddly-- somewhat helpful but (for me) somewhat redundant and too time-consuming to create. I have already split my single "backstory" document into hundreds of Scrivener cards so that I can sync with Aeon and better manage my backstory events, and would happily do the same thing for characters and even locations. I wouldn't WANT the text of those character/location/whatever documents to sync with Aeon, just the title/summary of the entity and some simple custom metadata that I define in Aeon (birthdate, death) and could view in both programs, either copying and pasting or not. May not be possible, but I cast my vote with @razyr. Speaking of heavy lifting-- Matt, I am really amazed by what you have created, a nd the grace with which you are managing huge onslaught of requests and brainstorms and criticisms that followed. It's like having a baby-- what you really want is champagne and a two week vacation, but what you get is three months of colic and eighteen years of hard work. I can only hope that you are also enjoying the (sometimes hidden) rewards.

Sat, 21 Oct 2017 14:40:47 GMT

Yes! I've actually created a previous Wish List post for precisely this feature. Auto discovery, like razyr says, may cause more grief than it solves. But being able to sync AT2 entities with Scrivener documents would be awesome, and incredibly useful.

Sat, 21 Oct 2017 14:58:03 GMT

How I think it should work: from each entity in AT2, a UI feature to choose the corresponding document in the connected Scrivener project. By default, I think AT2 should offer the documents from the corresponding project folder; so the document selection screen for a character entity should default to the Characters folder. But the UI should allow selection from other docs as well. In fact, to avoid messing with the event sync, perhaps the feature should demand that we select which is the Scrivener Characters folder for Character entities, and limit entity sync to documents from that folder. (Ditto for locations.) Syncing should probably be limited to the Notes field<-> synopsis, and Birth/Death to the Start/End fields in Scrivener. Detailed character sketches can live inside Scrivener.

Sat, 21 Oct 2017 14:59:45 GMT

And finally, let me add my voice to those who've crowed about the excellent, awesome work the AT2 is doing. My first novel really took off only after I'd added AT2 to my toolset. The Scrivener integration turned a murk of tangled timelines into a clear sequence of events, and helped me catch a bunch of chronology errors.

Fri, 05 Jan 2018 19:22:05 GMT

Would like to add a vote for character sync. A primary reason to use AT is the great functionality around mapping characters to scenes across the timeline. (why I bought AT). To echo the general sentiment, the sync needs to be based on a standard. Frankly, for us to speculate about using this custom field isn't useful. The devs need to get together and agree how to build this sync capability so that it is easy and repeatable between their software. Mucking about with manually created custom fields/entities creates a non-standard mess that the devs won't be able to support. Our job as writers and customers is to let them know we need a character sync! Tell your writer friends to weigh in on what an awesome tool this would be to enable writers to see into their stories more fully. :)