Invisible & Exciting: What’s New in Zotero 4.0.19

Zotero just released version 4.0.18 and 4.0.19, with some major, mainly under the hood improvements. These changes aren’t obvious at first sight, but are a major improvement. There are three major improvements, as well as a couple of minor ones.

Better Metadata from PDFs

Especially for new users of Zotero, the Retrieve Metadata function can be crucial: It allows you to simply drag&drop a couple of PDFs into Zotero and then with a simple right-click add find their metadata online. Up to now, this had two major problems.

1. The search relies principally on google scholar and users would frequently get locked out as suspected robots after retrieving data for more than 20–25 items, making the feature almost unusable to import large collections of PDFs
2. The data coming from google scholar is not the best. While it has gotten dramatically better over time, it still doesn’t include DOIs, abstracts, and has gaps and errors at times.

The new Zotero version mostly solves the first of these. Lock-outs are now rarer, and where they happen, users get prompted with the captcha google scholar uses to identify human users. So retrieve all you want! However, the second issue is still relevant, so once you have Zotero up and running, you should use the function sparingly for adding new items. In almost all cases the URL-bar icon is your tool of choice.

Faster Indexing

Zotero doesn’t just let you search through your items’ metadata, but also through the full text of your attachments as long as they are in plain text, html, or PDF format. For that purpose, Zotero indexes the full text of those attachments. In the past this was slow and adding several 100 page PDFs could freeze Zotero/Firefox for minutes. The speed of this has now improved dramatically: “War an Peace in 1.5secs” as Dan Stillman puts it.

Links in Notes

It has always been possible to add links in Zotero notes, however, until now you couldn’t actually follow them. This has now been fixed. This is, of course, useful by itself, but let me highlight two non-obvious use cases.

1. As various users point out in this thread many other organizational tools like Evernote or DevonThink have their own linking protocol (just like Zotero’s zotero://) that allows you to go straight to an item/note. Now you can get to an item in DevonThink directly from a Zotero note.
2. The indispensable Zotfile has long extracted PDF annotations into notes. Since version 3.1, Zotfile also includes a link to the page containing the annotation with every annotation. Now, with the links functioning, you can click on an annotation and have the PDF open on the right page.

Some Smaller Improvements

• Zotero recommends storing titles in sentence case. Right-clicking on an item’s title, allows you to convert that title to sentence case. The new Zotero version contains various improvements of this feature, including better handling of beginning punctuation (like quotation marks or the Spanish ¡ and ¿) as well as capitalization after punctuation within the title. Now the first letter after colon, question mark, and exclamation mark are capitalized, in line with the APA’s requirements for sentence case in titles. Unfortunately, other styles, in particular the Citing Medicine guide do not capitalize after colons, so that we will roll back that part in the next release. This thread explains the logic behind that in more detail.
• The reference test pane — Zotero’s built in tool that facilitates modifying citation styles—now has a save button. It is no longer necessary to take the detour via a text editor to save CSL styles.
• Both PMID and PMCID are now properly parsed from the extra field (each should start on a new line) so that citation styles using the PMID and PMCID variables, like the NIH/NLM grant style now work correctly.

Free Software at work

Another remarkable aspect of this release is that it is largely community driven: All major improvements are based on patches submitted by different volunteers: Emiliano Heyns (who I’ve mentioned several times before) submitted the patch for faster indexing; Aurimas Vinckevicius, who together with me does a lot of work on Zotero translators and support (among many other contributions) contributed the retrieve metadata improvements; Joscha, the developer of Zotfile, fixed the links in notes; the improvements to sentence case conversion, the reference test pane, and PMCID parsing came from me.

Zotero is an increasingly mature free software project with a broad base of contributors. That doesn’t mean, of course, that Zotero’s developers are idle: For one, each suggested patch—”pull request” in the language of git— gets reviewed very thoroughly, often with a lot of back and forth. Beyond that, Dan adds a constant stream of small fixes and improvements to both Zotero and the sync server, and, perhaps most importantly, has been working on the complete re-write of the sync infrastructure that will be the cornerstone of Zotero 4.1