Author Archives: adam3smith

Quality Control for CSL Styles

One of the most common questions we get about the CSL project, especially when we’re talking to companies looking to implement CSL into their product, is about quality control. So, how do we ensure quality for now more than 1000 different citation styles?

Travis: The Automated First Wall of Defense

Whenever someone submits a style to the CSL repository on GithHub, the first “person” they interact with is—well it’s not a person but a friendly bot called “Travis.” Travis, or “Travis CI” by full name, automatically runs a series of tests on the repository: It checks whether all styles validate, i.e. don’t contain anything that CSL doesn’t allow. It checks whether there are any macros in the style that are not used or are used but not defined. It also makes sure the style follows the naming convention we use in the repository.

This is how Travis looks when it’s happy:

Travis Pass

An unhappy Travis looks like this:

Travis Fail

By clicking on “Details” you can look at the exact error message(s) that cause Travis to fail. In this case, the submitted style, zeitschrift-fur-theologie-und-kirche, specifies some macros that aren’t actually used—Travis lists the test conditions that are violated.

Travis Details

Several Pairs of Watchful Eyes

There are a handful of people who can commit styles directly to the CSL repository (though the styles still go through a check by Travis before being widely distributed). Everyone else can contribute via so called “pull request,” basically a set of suggested changes. A couple of volunteers, mainly Rintze Zelle and me with occasional help from other people, review these submissions.

For new styles, we perform a very basic review: Does everything in the style make sense? Is it going to be used widely enough to warrant inclusion (we’re pretty liberal here, but aren’t accepting styles used by a dozen people)? Are other requirements like documentation and ISSN (for journals) met?

For existing styles, we take a closer look at the proposed changes. While we mostly take the word of contributors for what citations should look like, we do go over the changes in the CSL code to make sure they will actually and consistently have the desired effect. For widely used styles like APA, Chicago Manual, or Vancouver/NLM, we take extra steps to assure the changes get it right and will consult the respective style manual.

This process takes up a significant amount of volunteer time, especially since CSL attracts a huge amount of submissions, many from people with little or no programming experience. In the last month alone, for example, 10 new people contributed citation styles and/or fixes to CSL. We’re very proud of that! But it does mean that we spend more time guiding people through the process than most other projects likely do.

Code Maintenance—CSL on a Diet

We’ll occasionally identify common inefficiencies, sometimes even outright errors, in CSL code and fix those by scripting—everyone using their favorite tool ranging from simple sed commands to more elaborate Perl or Python scripts— (followed by manual control and fixes). In recent weeks we have removed thousands of line of unused and/or superficial code from CSL styles. While these rarely change the behavior of styles, they make our styles more efficient and easier to modify down the road.

User Feedback: A Thousand Eyes and We Still Need More

Now, you’ll say: “None of this guarantees me that the CSL style actually matches what the journal wants.” And you’ll be right. There is absolutely no way we can do active quality control on 1000 different styles. However, there are hundreds of thousands of people using reference managers that rely on CSL styles. Many of them will notice when something is wrong with a style. And when they let us know, we’ll fix it—quickly, in almost all cases.

Feedback from Reference Managers

The “when they let us know” part is quite important. For a long time, at least 90% of all error reports for styles came from Zotero users, reported via their forums and that’s still probably the single largest source of error reports for styles. But there are many other products using CSL styles now and we want to hear from them. By far the biggest recent improvement for getting error reports has developed out of our talks with Mendeley: they have now set up a simple form through which their users can submit style error reports. Those are (automatically) shared between us and Mendeley’s customer support in a google doc and have proven a very effective way for communicating errors. Unfortunately, with rare exceptions, those two are almost the only source of error reports that reach us (with the occasional pandoc user). Which is too bad, since there are at least another half a dozen reference managers using CSL.

So, if your reference manager uses CSL styles, we want to hear from you. We’d be happy to set up similar mechanisms that we currently have working with Mendeley. (colwiz, Docear, Paperpile, Qiqqa, ReadCube etc.—I’m looking at you.)

Having style errors go through the reference manager’s support system is crucial for two reasons: For one, about a third of error reports are actually errors in the data stored in the reference managers. We’re not in a position to tell users how to put in data in every reference manager.  Secondly, not every reference manager interprets CSL styles the same way. It is not uncommon for error reports to actually point to errors in the way CSL styles are interpreted or fields are mapped in a reference manager—so they need to know about these errors.

Feedback from Editors and Publishers

We are particularly happy when we hear from journals directly about their citation style. This—surprisingly in my opinion—still quite rare. We recently heard from a copyeditor at The Lancet. An Elsevier employee is currently going through their list of journals systematically and submitting corrections (directly as patches, which is extra awesome, but we’d take simple error reports, too). We’ve had some smaller journals contract with me to write CSL styles for their house styles. Since they then go on to check on my work, those are guaranteed to be accurate.

Still, I think there’s a lot of room for improvement here. In particular, I wish more of the bigger publishers like Sage and Oxford University Press would be willing to work with us at least on those styles used in multiple of their journals. There should be, as I’ve said before, more journals covered by generic styles. Publishers should also make it much easier to access lists of their journals with corresponding citations styles, along the lines of what BioMed Central does.

Invisible & Exciting: What’s New in Zotero 4.0.19

Zotero just released version 4.0.18 and 4.0.19, with some major, mainly under the hood improvements. These changes aren’t obvious at first sight, but are a major improvement. There are three major improvements, as well as a couple of minor ones.

Better Metadata from PDFs

Especially for new users of Zotero, the Retrieve Metadata function can be crucial: It allows you to simply drag&drop a couple of PDFs into Zotero and then with a simple right-click add find their metadata online. Up to now, this had two major problems.

      1. The search relies principally on google scholar and users would frequently get locked out as suspected robots after retrieving data for more than 20–25 items, making the feature almost unusable to import large collections of PDFs
      2. The data coming from google scholar is not the best. While it has gotten dramatically better over time, it still doesn’t include DOIs, abstracts, and has gaps and errors at times.

The new Zotero version mostly solves the first of these. Lock-outs are now rarer, and where they happen, users get prompted with the captcha google scholar uses to identify human users. So retrieve all you want! However, the second issue is still relevant, so once you have Zotero up and running, you should use the function sparingly for adding new items. In almost all cases the URL-bar icon is your tool of choice.

Faster Indexing

Zotero doesn’t just let you search through your items’ metadata, but also through the full text of your attachments as long as they are in plain text, html, or PDF format. For that purpose, Zotero indexes the full text of those attachments. In the past this was slow and adding several 100 page PDFs could freeze Zotero/Firefox for minutes. The speed of this has now improved dramatically: “War an Peace in 1.5secs” as Dan Stillman puts it.

It has always been possible to add links in Zotero notes, however, until now you couldn’t actually follow them. This has now been fixed. This is, of course, useful by itself, but let me highlight two non-obvious use cases.

  1. As various users point out in this thread many other organizational tools like Evernote or DevonThink have their own linking protocol (just like Zotero’s zotero://) that allows you to go straight to an item/note. Now you can get to an item in DevonThink directly from a Zotero note.
  2. The indispensable Zotfile has long extracted PDF annotations into notes. Since version 3.1, Zotfile also includes a link to the page containing the annotation with every annotation. Now, with the links functioning, you can click on an annotation and have the PDF open on the right page.

extract-annotations

Some Smaller Improvements

    • Zotero recommends storing titles in sentence case. Right-clicking on an item’s title, allows you to convert that title to sentence case. The new Zotero version contains various improvements of this feature, including better handling of beginning punctuation (like quotation marks or the Spanish ¡ and ¿) as well as capitalization after punctuation within the title. Now the first letter after colon, question mark, and exclamation mark are capitalized, in line with the APA’s requirements for sentence case in titles. Unfortunately, other styles, in particular the Citing Medicine guide do not capitalize after colons, so that we will roll back that part in the next release. This thread explains the logic behind that in more detail.
    • The reference test pane — Zotero’s built in tool that facilitates modifying citation styles—now has a save button. It is no longer necessary to take the detour via a text editor to save CSL styles.
    • Both PMID and PMCID are now properly parsed from the extra field (each should start on a new line) so that citation styles using the PMID and PMCID variables, like the NIH/NLM grant style now work correctly.

Free Software at work

Another remarkable aspect of this release is that it is largely community driven: All major improvements are based on patches submitted by different volunteers: Emiliano Heyns (who I’ve mentioned several times before) submitted the patch for faster indexing; Aurimas Vinckevicius, who together with me does a lot of work on Zotero translators and support (among many other contributions) contributed the retrieve metadata improvements; Joscha, the developer of Zotfile, fixed the links in notes; the improvements to sentence case conversion, the reference test pane, and PMCID parsing came from me.

Zotero is an increasingly mature free software project with a broad base of contributors. That doesn’t mean, of course, that Zotero’s developers are idle: For one, each suggested patch—”pull request” in the language of git— gets reviewed very thoroughly, often with a lot of back and forth. Beyond that, Dan adds a constant stream of small fixes and improvements to both Zotero and the sync server, and, perhaps most importantly, has been working on the complete re-write of the sync infrastructure that will be the cornerstone of Zotero 4.1

Beta Test Zotero

Zotero has just started a regular beta release. The beta version replaces what used to be called the “branch xpi”. It is built regularly from the current release branch. In less technical terms that means that using the beta version you’re testing features that will be in the next minor relase. The current beta, for example contains, code changes that will be in version 4.0.18. The beta version is intended to be usable with minimal risk. It will typically not contain database upgrades, so the risk of data corruption is very low and you can easily revert to the regular version if you’re in a pinch. The beta channel is update a lot more frequently than regular Zotero.
The principal reason for releasing a regular beta is to encourage wider testing of Zotero versions pre-release. If you’re interested enough in Zotero to read this blog, there is a good chance you should run the beta version.

You may want to run Zotero beta If…

      • You want to help Zotero by testing pre-release versions
      • You provide technical support for Zotero and want to be aware of new features before they land (more on that below)
      • You like having new and shiny things before anyone else has them

You (probably) shouldn’t run Zotero beta if…

      • You want maximal stability when using Zotero
      • You panic or get frustrated when something doesn’t work on your computer
      • You have no time or no patience to deal with and report occasional problems on the Zotero forums
      • You’re only using Zotero Standalone (the beta currently is only for the Firefox add-on).

OK, I want to be using the beta version, what now?

Install

First, install the add-on from here. You can simply install it over your existing Zotero, your database will remain untouched. Heed the warning on that page: I’ve never had any trouble running pre-release versions of Zotero and I’m not aware of anyone who has lost data doing so, but it’s beta software and you want to make sure to have regular and automated back-ups.

Check

You can see the currently installed version in the add-ons tab of Firefox or under “About Zotero” in the gears menu. As of this writing the current beta is 4.0.18-beta.r3+fadd486. This means you’re running the 3rd (r3) beta release for the 4.0.18 version of Zotero. The last part after the plus sign corresponds to the last commit to the Zotero source code that’s included in the version, so if you’re following commits (you should be looking at the 4.0 branch) you can easily check whether the version you’re running already contains an addition to the code.

Report Problems

Zotero devs will very much appreciate any error reports from beta users. As for all Zotero errors, you should report them on the forums, and you should provide plenty of details and, if possible, an error report ID. Also mention that you’re running the beta version of Zotero. I had some problems trying out the beta last night, and you can see my report (with quick solution) here.

Talking about shiny things…

The current beta version contains two major improvements in handling PDFs, both coded by community developers. Thanks to Aurimas, “Retrieve Metadata” has improved significantly and you’re now less likely to get locked out by google scholar. Thanks to Emiliano (whose add-ons I praised in my last post), indexing of large (or many) files is now several orders of magnitude faster—a large PDF like War and Peace could take minutes to index and freeze Zotero before, now it takes a couple of seconds.

Tools and Toys

One of the things that make Zotero most useful — and most fun — are the many options it provides for customization and applications via the API or in the form of plugins. Here are some I have come across recently.

Using IFTTT with Zotero

ifttt.com is a neat little service that lets you create little scripts that have different webservices interact without actually doing any scripting. It stands for “If this than that”. It offers about 80 different channels—webservices, email, text messages—that you can set up either as triggers (the “if this” part) or as actions (the “then that” part). There is currently no Zotero channel and no possibility to create channels of your own (though you can request the former and the latter is planned. You can, however use adding an item to Zotero as a trigger by using the “Feed” channel. Here’s a “recipe” I created that will create a WordPress blogpost for every item added to a group, inspired by this job offer. You can easily create feeds not just for an entire library, but also for a collection or a tag. Simply click on the collection (or tag) in the online display of your library and then find your feed address at the bottom left. For private libraries, you will need to click the link to create a private key first.

feed

Autoexporting

Especially for users who employ Zotero as a frontend for bibtex or biblatex, keeping the Zotero database in sync with a data file — a .bib file in the case of bibtex — is a hassle: It requires frequent, manual export. The first plugin to address this issue was Rob Wilson AutoZotBib which exports the Zotero library in bibtex to a designated file on every change. The problem with AutoZotBib is that it slows down Zotero significantly, especially for larger libraries. A more sophisticated solution I’ve come across recently is Zotero Autoexporting by Robert Kühn. Zotero Autoexporting has the same general functionality, but allows for a lot more cutomization

  • Beyond manual export and export on every change, it also offers export in certain time intervals (say every hour)
  •  It lets you export just one (or multiple) collections, creating a .bib file for every collection
  • It allows you to run an automatic post-processing (e.g. bash) script on your file

Setting up these features is pretty straightforward once you’ve found the preferences: For me these are only accessible via the Firefox/Zotero add-ons view:
autoexporting-pref

Zotero Autoexporting can export in any of Zotero’s export formats, so you could also uses it to, e.g., keep a MODS version of your library or to echxchange data with another program regularly.

More Bibtex

If you’re using Bib(La)Tex, you may also want to consider using Emiliano Heyns’s Zotero Better BibTex. It adds a number of convenient features and more flexibility to Zotero’s bibtex handling, including the ability to set custom citekey’s via the extra field and drag&drop citekeys. Some of these features are (by necessity) on the hackish side of things, so you may need to redo some things once they’re properly supported by Zotero, but until then they allow you to use Zotero as a much more sophisticated bib(la)tex front-end.

Customize Reports, Speed up Indexing

Emiliano has been on a roll recently and published several other plugins, two of which strike me as especially useful.
The first one, Report Customizer, takes Jason Priem’s Zotero Report Cleaner and turns it into a plugin. This makes it easier and faster to use and saves your settings. Simply install the plugin and then, in its preferences (which are accessible through the Zotero gears menu), check the box for fields you do not want included in reports, e.g. Date Added and Modified in the example below.

reportcustomizer

The second plugin works mainly behind the scenes: Zotero AutoIndexing implements a different indexing algorithm for Zotero and runs it automatically. The algorithm is significantly faster than the one used by Zotero, so this will be very useful for people adding a lot of large PDFs. (Its downside is that it will not work correctly for several non-Western languages like Chinese). As a bonus feature, the plugin also extracts new annotations from PDFs if you have the indispensable ZotFile installed.

On a personal note, I have written back and forth with Emiliano a couple of times, and he has been incredibly fast in addressing any issues that have come up, so I can recommend installing even early versions of his add-ons.

Mac Only: ZotQuery

As a non-Mac user I’m mainly jealous here, but the new ZotQuery looks too good to not mention it: It is a Zotero workflow for the popular Alfred productivity software that allows quick searches of your Zotero libray and opening of attachments (not unlike Qnotero), and, via the Zotero server API, will even create bibliographies in markdown. (Edit: the above link now points to the developers blog with in-depth documentation).

Writing CSL – Features and Best Practices

With the CSL repository growing steadily, and a very large number of different contributors — more than 250 since its beginning — helping by adding or improving styles, I figured this would be a good time to highlight some features of CSL and what they mean for best practices in coding a CSL style.

Groups

CSL allows you to group items that belong together and set a “delimiter” between them. Think of a common example, the colon between place and publisher: New York, NY: Columbia University Press, 1990. You can code this as

 <text variable="publisher-place" suffix=": "/>
 <text variable="publisher"/>
 <text variable="issued" form="year" date-part="year" prefix=", "/>

The problem with this version is that if your item doesn’t have a publisher, you get a mess like New York, NY: , 1990. Using groups makes this a little longer, but much more robust:

 <group delimiter=", ">
   <group delimiter=": ">
     <text variable="publisher-place"/>
     <text variable="publisher"/>
   </group>
   <date variable="issued" form="text" date-part="year"/>
 </group>

This will produce sensible output like New York, NY, 1990 regardless of which input variable may be missing.

Best Practice 1: Prefer groups and group delimiters to affixes for punctuation and spaces between items

Macros

CSL allows you to use macros in citation style. Simply put, macros allow you to write a relatively long sequence only once. Say you need the style to give the year of publication and if there is none “n.d.”. This is the type of thing that you’re going to use multiple times in a style — e.g. in the citation, the bibliography, and maybe for sorting — each time taking eight lines of code. So instead of writing the whole thing multiple times, define

 <macro name="issued-year">
   <choose>
     <if variable="issued">
       <date variable="issued" form="text" date-part="year"/>
     </if>
     <else>
       <text term="no-date" form="short"
     </else>
   </choose>
 </macro>

and simply “call” the macro using <text macro="issued-year"/> where you would otherwise write the whole eight lines.

Code economy isn’t the only, or even principal advantage of using such macros. Their biggest advantage is that if someone else wants to change your code, e.g. for a new similar citation style, any changes in a macro only need to be made once. A huge time-saver! Finally, looking into the future, CSL was always intended to be a “modular” language: you can take one piece out of a citation style and plug it into another style. Relying extensively on macros in citation styles fosters such modularity. (Frank Bennett has written up some informal thoughts on this).

Best Practice 2: Use Macro’s extensively. Keep the cs:citation and cs:bibliography parts shorts and keep cs:choose elements in them to an absolute minimum

Terms

“Available at,” “no date,” “anonymous,” “pages” — citation styles contain many such words or short phrases. CSL defines many of them as “terms” and provides translations into dozens of language. Strictly speaking, many of these wouldn’t be necessary. To provide, e.g. a URL preceded by “Available at”, <text variable="URL" prefix="Available at "/>, works. But it’s, well, ugly. It also means that someone who wants to adapt the structure of your style but use it in Spanish will have to rewrite all such prefixes. Instead use

 <group delimiter=" ">
   <text term="available at" text-case="capitalize-first"/>
   <text variable="URL"/>
 </group>

and your style can be converted to any language by simply changing the default-locale setting in the first line of the style (or, leaving it blank, will automatically appear in the user’s install language).

But what if the term isn’t quite right? You may need “Available from” instead? You can redefine any term for any language/locale in the beginning of the style, in our example:

 <locale xml:lang="en">
   <terms>
     <term name="available at">available from</term>
   </terms>
 </locale>

Best Practice 3: Use terms and labels. Do not add terms or phrases in affixes or using text value=

Miscellanea

  • If you plan to submit the style to the repository, closely read and follow the requirements

  • Assume that users have titles stored in sentence case. Use text-case="title" as needed, but don’t use text-case="sentence"

  • Some styles have left-over clutter from an earlier version. In particular, you will often see periods (which all abbreviated terms have in CSL) first stripped using strip-periods="true" and then added again using suffix=".". Remove this and similar unnecessary code. Future users/coders will thank you.

Best Practice 4: Don’t use sentence case, and keep styles tidy by removing clutter, following CSL repository guidelines

Examples

Best practices are not ironclad commandments. Of all the “rules” above, only failing to conform to repository guidelines and forcing sentence case on a title will prevent a style from being accepted to the CSL repository. You can look at the styles for American Psychological Association or Vancouver. You’ll see a lot of rule-breaking, but also a general adherence to these recommendations to give you a sense of how a well-coded style looks.

Keyboard Magic

While it is possible to do almost anything you need to do in Zotero with your keyboard, it can be rather cumbersome (think: tab – tab – tab – right arrow – right arrow – tab – return). Personally, I don’t like taking my fingers of the keyboard. For other people this may be a matter of health (e.g. RSIs) or even necessity. Here are a combination of tricks to use Zotero with the keyboard and tools to make it even more keyboard friendly.

Simple Tricks in Zotero

First things first. Zotero already has a number of keyboard shortcuts defined. You can find them in the Zotero preferences under “Shortcuts.” With version 4.0.12, the two base-modifier keys were changed on Linux and Windows to ctrl+shift (they remain cmd+shift on Mac). More excitingly, Zotero has added a keyboard shortcut for clicking the URL bar icon on supported pages — ctrl+shift+s. Adding an article to Zotero has become even more convenient!

Some other small tricks:

  • You can navigate around Zotero using tab and arrow keys, with return typically working like a mouse-click, though depending on what you’re trying to do, this may be a bit tedious.

  • You can expand or collapse all items in both the left hand and the center panel (depending on which you have selected) by pressing the + (expand) and – (collapse) keys.

  • You can start typing to jump to a collection (on the left) or an item (in the center). In the center panel the logic will follow the column you are sorting by.

  • When entering information manually, shift+return allows you to add new items to a list. That’s true for authors (shift+return opens an empty new author) and tags&identifiers (shift+return opens and closes a large field where you can add multiple tags/identifiers separated by return).

  • If you write in Word or LibreOffice, the Quick Format toolbar for the Zotero plugin is designed to work smoothly with keyboard only. Details are in the Zotero documentation

Zutilo

Zutilo by Will S. is an add-on designed to add better keyboard functionality to Zotero. While earlier versions relied on 3rd party tools (like keyconfig or Customizable Shortcuts) to provide shortcut functionality, version 1.2.5 offers its own. Note that as of this writing, 1.2.5 does not appear on mozilla add-on page, as it’s not reviewed yet. You can install it from the version history.

Once installed (no need to restart FirZutilo Preferencesefox/Zotero), the Zutilo Preferences will appear in the action/gears menu. There, you can set shortcuts for a large number of actions: I have set keyboard shortcuts for adding a tab and creating a new web page item — the equivalence of the “Create New Web Page Item from Current Page” button in Zotero for Firefox — here. Zutilo will warn you if a key-combination is already in use by itself or Zotero. It offers some functions that have native Zotero shortcuts, but because you can use any combination of modifier keys, you are more flexible in your assignment of shortcuts. (Take some care in picking your shortcuts, though. It’s easy to create conflicts with existing shortcuts in Firefox or your operating system.)

Zutilo also provides you with some additional functionality, in particular to make it easier to relate items to each other and to copy tags and creators between items. You can access these functions not just via shortcuts, but also via the context menu that comes up when you right-click on an item:

ZutiloContext

These functions as well as Zutilo‘s other main functionality — advanced handling of attachments — are described in more detail in Will’s very thorough documentation.

Using Markdown for Notes

Via Joscha Legewie, the author of the indispensable ZotFile (if you‘re interested enough in Zotero to read this, I’ll assume you’re already using ZotFile) I recently came across Markdown Here, a nifty Firefox extension. Whenever you’re typing in a rich text editor in Firefox, you can write in markdown and the toggle that to rich text with the click of a button or, more appropriately and staying with the theme of this post, a keyboard shortcut. Btw. if you don‘t know about markdown yet, you should. It’s what all the cool kids (and even wannabes like me) are using these days.

Anyway, you ask, what does that have to do with Zotero? Well, Zotero‘s note editor is a rich text window, and if you’re using Zotero for Firefox, Markdown Here will work with it right out of the box. Here are Joscha’s illustrations

Markdown NoteRichtext Note

You won’t have a button, but you can use your keyboard shortcut to toggle. In Zotero Standalone this requires some tweaks, which Joscha has implemented in a fork. Some limitations remain. Happy typing everyone!

Announcing RTF/ODF Scan for Zotero

May 12, 2013: Updated to reflect changes in the 1.0.9 update of the plugin. Changes are backwards compatible, i.e. everything that worked before still works now. See in particular the section on hidden preferences below

Support for Scrivener and google docs is one of the most frequently requested Zotero features. It is with great excitement that Frank Bennett and I announce today RTF/ODF Scan for Zotero, a Zotero plugin that extends Zotero support to any word processor capable of saving/exporting ODF (Open Document Format — .odt). You can find a brief, but comprehensive set of instructions on the project webpage. Here some additional comment and pictures.

The Short version

The plugin will install a “Scannable Cite” translator for Zotero. With that translator you can create citation markers to be inserted in your document (in Scrivener, google docs, etc.). Save/Export that document as an ODF file and convert the markers to Zotero citations using the “RTF/ODF Scan” feature from Zotero’s gear menu. Open the converted document in LibreOffice and set a citation style using Zotero’s LibreOffice plugin. Done!

In a Nutshell

What you need:

  • A word processor that should ideally be able to save/export as ODF (.odt)
  • Zotero for Firefox and/or Standalone on all computers you want to use
  • Install the RTF/ODF Scan for Zotero plugin from the project webpage by clicking on “Download Add-on”
  • You will need LibreOffice and the Zotero LibreOffice plugin installed on at least one computer. You will only need this for the final editing step.

Creating Citation Markers

default-output

Select the Scannable Cite translator as “Default Output”

After installing the plugin (no re-start required), set the Default output format for quick copy in Zotero’s Export preferences to “Scannable Cite.”  (The export format is installed by the add-on). You can now either drag&drop to insert citation markers into a document  or use  the ctrl+alt+c (cmd+shift+c on Mac) to copy them to the clipoboard and then paste them to your document.

A citation marker will look like this:

{ | Smith, (2012) | | |zu:2433:WQVBH98K}

It has five sections for — in that order — prefix, a summary of the cited item, a locator (e.g. a page number), a suffix, and a unique identifier. E.g. the marker

{cf. | Smith, (2012) |ch. 3 |, for the main argument |zu:2433:WQVBH98K}

drag-and-drop

Drag & drop citation markers into google doc

Would be converted into  (Smith, 2012, Chapter 3, for the main argument) when APA style is selected. You can use simple mark-up to use italics and bold print in suffix and affix: a single asterisk for *italics* two  asterisks for **bold**. Citation markers directly adjacent to each other will be turned into a single citation, as in (Smith 1776, 2012). Dragging/copying multiple items from Zotero at once will automatically create such adjacent markers. Locators require a specific set of abbreviations (e.g. p. or pp. for pages). For a complete list see the add-on’s webpage. A sample google docs document with citation markers is here.

Saving and Converting Documents

export-odt-scrivener

ODF export in Scrivener

export-odf-gs

ODF Export in Google Docs

Once you’re done writing, export your document as ODF. Many word processors have that option, including google docs and Scrivener. If your word processor does not export ODF, you can save in a different format (RTF, .doc), open the file with LibreOffice and save as ODF.

select-odf-scanThen go to Zotero and click on the RTF/ODF Scan option in the gears/action menu. Select “ODF (to citations)” as File format, the .odt file you saved as “Input File” and a location for the (converted) “Output File.” Click “Next” to perform the conversion.

odf-scan-window

Selecting a Citation Style

lo-basic-docOpen the converted document — by default it will have the same filename as the input file with (citations) appended — in LibreOffice. The citations will show up without any particular formatting and normally be shaded in dark grey. Click “Set Document Preferences” in Zotero’s LibreOffice plugin and select the citation style you want to use (you can change it later).  All citations will now appear formatted correctly. You can insert a bibliography by clicking “Insert Bibliography” from the same toolbar. The document is now indistinguishable from a document written entirely with the LibreOffice plugin, so you can opt to edit or continue writing it in LIbreOffice and citations will update according to changes in your Zotero database as well as the citation style.

RTF/ODF Scan as a Collaboration Tool

RTF/ODF Scan will work with group libraries, so it is ideally suited for collaborative writing. The only caveat is that, unlike Zotero 3.0+, it will not “store references” in a document, so all people collaborating on a paper need to cite items from a shared library, typically a group.

Reverse Conversion

RTF/ODF Scan can also convert LibreOffice citations in the Reference Mark format back into citation markers. For that purpose choose “ODF (markers)” as the file input format. This may come in handy if you have authored a document in LIbreOffice so far, but want to switch to google docs or Scrivener, you can convert your citations to markers. Another use is the ability to merge adjacent citations as requested by several Zotero users (this is Endnote’s default behavior, so many users are accustomed to it). RTF/ODF scan will do that simply by converting the same document back and forth.

Hidden Preferences

You can modify markers with two hidden preferences. See here for general instructions on accessing hidden preferences. Both of these preferences can be toggled back and forth using right-click –> Toggle in the about:config settings of Firefox/Zotero.

extensions.zotero.translators.ODFScan.includeTitle
will include the title of the item in the marker (default: false, i.e. only author, (date))

extensions.zotero.translators.ODFScan.useZoteroSelect
will use zotero://select links instead of simple item ids. These have the disadvantage of being longer, but can be used to access Zotero items directly (e.g. by pasting them in Firefox’s URL bar). This preference affects both the translator and the markers produced by reverse conversion. (default: false, i.e. citation markers as displayed above). Because of the way library IDs are constructed in Zotero, the group IDs used in the default markers and those in zotero://select differ. Trying to adjust them manually will break ODF scan

Getting Help

Please post questions and problems to the plugin’s thread on the Zotero forums.

{See | Smith, (2012) |p. 45 | for an example |zu:2433:WQVBH98K}

Why you should be excited there are 6000+ CSL styles

This morning, the number of citation styles in the Citation Style Language (CSL) repository went above 6000. This is an amazing success story for a tiny, unfunded open source project. Here are three reasons you should be excited about this.

1. Publishers are seeing the light on citation styles

When Rintze Zelle posted this news on twitter, one response was bewilderment (even anger) about the number of citation styles:

But these folks get it all wrong. The reason there are more CSL styles is that there are fewer citation styles. Take a look at this graph, by Carles Pina, one of Mendeley’s developers (not embedded since it updates daily). See those jumps of the green line in mid-April and mid-March? Those were the dates citation styles for a massive number of journals by Springer and Elsevier were added. But now look at the blue line – the number of “unique” styles, i.e. citation styles that are actually different. It hardly budges on those dates. The reason for that is that both Springer and Elsevier have started to consolidate the citation styles for all of their journals. Once they’re done, (almost) every one of their journals will be based on one of six citation styles. The list with those styles was provided to Papers and Mendeley developers when their companies were bought by Springer and Elsevier respectively. Taylor & Francis is in a similar process of consolidating their journals to five citation styles and they just gave us the list when we asked (and were very nice & helpful in the whole process, I’ll add). All three of these publishers base their styles on popular existing manuals such as APA and Chicago. Sage seems to be doing the same — they have style manuals for a Harvard  and a Vancouver style — but hasn’t replied to my request for a list of journals following them. Sadly, I haven’t seen any similar movements from the major university presses — Cambridge, Oxford, I’m looking at you.

But why do we add those journals to CSL then? And do we keep 500 versions of the same style? As for the first question – our vision is that if you want to publish in a journal you just look it up in the list of styles for you (CSL-using) reference manager and, voilà, you have correctly formatted references. No need to check “Instructions for Authors” on the journal’s webpage or to figure out if the style they provide is “Elsevier Harvard” or not. And of course we don’t keep 500 copies of the same style on the CSL repository. Where one style is used by multiple journals, we create a dependent style, which is typically about 15 lines long and mainly consists of the name of the journal and a link to its “parent” style. You can see an example here if you’re curious.

2. CSL increases competition among reference managers…

…and that improves the quality of the existing products. Seven years ago, when Zotero got started, and CSL was still in its kids shoes, providing a citation function was a major obstacle for a new reference manager. Zotero invested a fair amount of initial resources getting CSL of the ground, writing the first CSL processor (the code that converts data and citation styles into formatted citations) and contributing a fair number of citation styles.

Today, when a new product like Docear or Colwiz enters the same market, things are radically different. They can freely use many thousands of CSL styles (each one under an open CC-BY-SA license). They don’t even have to write their own processor, but can pick from various options: citeproc-js by Frank Bennett (used e.g. in Zotero, Mendeley), citeproc-hs  by Andrea Rosato (used in the wonderful Pandoc) as well as citeproc-ruby  (Sylvester Keil) and citeproc-php, (Ron Jerome) both of which are optimized for web-based implementations. They can direct their users to the visual CSL editor developed by Steve Ridout in a cooperation between Columbia University and Mendeley. Thanks to CSL, implementing a citation function in a reference manager has become relatively easy.

In other words the “barriers to entry” in the reference manager market have been lowered significantly by CSL to the benefit of us, the users. More competition doesn’t just mean more choice, it also means more rapid innovation. Even if you’re still an Endnote user (and you really shouldn’t be) you’ll benefit as Endnote is reacting to the increasingly fierce competition by improving exactly those feature where e.g. Zotero and Mendeley excel (or why did you think there is now a retrieve metadata feature for PDFs and better support for webpages?) and improving theirfree offerings (their #freeEndNote hashtag sounds more like a plea to me, though).

3. Free and Open Source Solutions Work

While much of the work on CSL is performed by a small team of unpaid volunteers (essentially Rintze, me, and the authors of the processors), we have been increasingly successful attracting outside contributions. According to the stats collected by ohloh.net, CSL as a “very large development team.” While I have been critical of the lack of support for CSL by Mendeley et al in the past (and still the vast share of new styles and improvements come from Zotero and its users), both Mendeley and Papers have taken a very welcome more pro-active role in CSL recently. Papers developer Charles Parnot has been instrumental in engineering the push to incorporate more dependent styles (and has developed much of the work-flow we use for that). Mendeley contributed the long awaited visual editor (which is open source) and Carles Pina has helped out with the large Elsevier and Taylor&Francis additions. The brand new CSL logo you see up top was donated by Austrian graphic designer Johannes Krtek. So helping hands all around.

All that said, CSL’s enormous popularity does pose some challenges, as the day-to-day maintenance of the repository at times takes up more time than Rintze and I can reasonably muster. CSL currently has no funding of any kind. We’re working on finding ways to make the project long-term sustainable and assure continuing improvements (and I’m somewhat confident that the large number of stake-holder will help in that process), but that’s a topic for another day.

Zotero Item URIs from Client

While working with Frank Bennett on implementing a Zotero workflow for google docs or Scrivener (almost ready – stay tuned) I wrote two little (tiny, really) tools that I thought I’d share.

The first one allows you to quick copy the URI of an item from your Zotero client. The second one gives you a zotero://select link that opens an item in Zotero.

Zotero Item URI

Say you want to send a friend a link to an item in your public Zotero library or in a group on zotero.org. Right now this is  a cumbersome process: Go online, search for the item, copy the link from the URL bar… With the “Item URI” Translator, you can simply drag and drop a link to the online version of the item. Setting this up takes three steps

  1. Download the file “Item URI.js” from here and place it in the directory “translators” in your Zotero data folder.
  2. Restart Firefox/Zotero
  3. In the Zotero Preferences go to the “Export” and set the Default Output Format to “Item URI” (this will be towards the bottom of the list)quickcopy-itemuris

You can now take advantage of Zotero’s quick copy functionality and drag&drop links from the item in your client right into an email or a blogpost. You can also use the shortcut for “Copy Selected Items –  ctrl+alt+c on Linux/Windows, cmd+shift+c on Mac – to copy the URI to you clipboard. It will look something like this: http://zotero.org/users/76252/items/UUKSSZVK

Personally I think something like this should be included as a button in Zotero’s interface, but until that’s the case, I hope you find it useful.

Zotero Select

Zotero allows you to directly link to an item in your local database. If you paste a valid URI starting with zotero://select into your URL bar in Firefox, it will open Zotero to that item.  To create such a zotero://select link from an item, download the file “Zotero Select Item.js” from here and then follow steps 1-3 above. A link will look like zotero://select/items/0_USN95MJC and work only with your library.

The main use for such links would be to link to Zotero from other software, e.g. from note-taking applications There are several issues involved with that depending on your operating system and I’m not going to go into the details, but see here for a good overview with many more links and here for a Mac/Applescript specific solution. The version of the Zotero select translator linked to above will only create the link – you can find some more elaborate version – including html wrappers and basic item information – linked to from here.

What’s New in Zotero 4.0 – Part 2

In the last blogpost I introduced some of the new features of Zotero that help you to collect and organize your references more easily. Zotero 4.0 also has a number of new features that improve citations and make syincing more flexible.

Automatic Journal Abbreviations

Up to now, Zotero took journal abbreviations from the “Journal Abbr.” field. Many databases don’t populate that field properly or at all, so this required a lot of maintenance by users who needed correctly abbreviated journal titles in citations. Now, Zotero comes with a built-in journal abbreviation list. The list takes the journal title from the “Publication” field and matches it against a list of journal titles and words to create a journal abbreviation. The abbreviation list currently only works when using the word processor plugins. For new documents, the option to “Automatically Abbreviated Journal Titles” is active when using styles requiring abbreviated journal titles. In documents created before the release of Zotero 4.0, you can turn on the option under “Set Document Preferences” in the word processor plugin. It will only be available if your chosen citation style requires abbreviated journal titles.

journal-abbreviations

File Sync on Demand

When you sync Zotero to a new computer with file syncing enabled, all item data and all attachments are downloaded. As attachments can take up multiple gigabytes, this can be a long process, impracticable for users who switch computers often. In Zotero 4.0, users can set file syncing to “Download files as needed.” That way, files on the server will only be downloaded when you try to open them. This option is turned off by default.

sync-on-demand

Relative Linked Attachment Filepaths

By default, Zotero stores all file attachments in a “storage” folder in the data directory. Using file sync via Zotero file storage or a WebDAV server, you can sync these files and comfortably access them on multiple computers. Some users, however, prefer to use their own organizing structure for files in their file system and link to them from Zotero. In the past, this made accessing files on multiple computers difficult. On one computer a file may have had the path ‘/home/workcomputer/Zoterofiles/Important-Paper.pdf’ while on another it was ‘/home/homecomputer/Zoterofiles/Important-Paper.pdf’. The link to this file – what we call an “absolute” link – would only work on one of the two computers. In Zotero 4.0, you can now specify a base directory, and have linked files be relative to that directory. In our example, you would go to “Files and Folders” tab of the “Advanced” tab of the Zotero preferences and on one computer input ‘/home/workcomputer/Zoterofiles’, on the other computer ‘/home/homecomputer/Zoterofiles/’ as the “base directory.” The link to ‘Important-Paper.pdf’ would then work on both computers. This will also work if there is a folder structure within the base directory, as long as it is the same on both computers.

relative-paths

And there is still more…

This is still not everything. Even more new features were introduced in Zotero 4.0. They include multiple performance improvements — for syncing, duplicate detection, pdf metadata retrieval — as well as many little additions that we hope will make Zotero even more useful to you. You can find a full list of all changes here. As always, if you have any questions about Zotero, a dedicated group of users and developers are happy to help you on the Zotero Forums.