Invisible & Exciting: What’s New in Zotero 4.0.19

Zotero just released version 4.0.18 and 4.0.19, with some major, mainly under the hood improvements. These changes aren’t obvious at first sight, but are a major improvement. There are three major improvements, as well as a couple of minor ones.

Better Metadata from PDFs

Especially for new users of Zotero, the Retrieve Metadata function can be crucial: It allows you to simply drag&drop a couple of PDFs into Zotero and then with a simple right-click add find their metadata online. Up to now, this had two major problems.

1. The search relies principally on google scholar and users would frequently get locked out as suspected robots after retrieving data for more than 20–25 items, making the feature almost unusable to import large collections of PDFs
2. The data coming from google scholar is not the best. While it has gotten dramatically better over time, it still doesn’t include DOIs, abstracts, and has gaps and errors at times.

The new Zotero version mostly solves the first of these. Lock-outs are now rarer, and where they happen, users get prompted with the captcha google scholar uses to identify human users. So retrieve all you want! However, the second issue is still relevant, so once you have Zotero up and running, you should use the function sparingly for adding new items. In almost all cases the URL-bar icon is your tool of choice.

Faster Indexing

Zotero doesn’t just let you search through your items’ metadata, but also through the full text of your attachments as long as they are in plain text, html, or PDF format. For that purpose, Zotero indexes the full text of those attachments. In the past this was slow and adding several 100 page PDFs could freeze Zotero/Firefox for minutes. The speed of this has now improved dramatically: “War an Peace in 1.5secs” as Dan Stillman puts it.

Links in Notes

It has always been possible to add links in Zotero notes, however, until now you couldn’t actually follow them. This has now been fixed. This is, of course, useful by itself, but let me highlight two non-obvious use cases.

1. As various users point out in this thread many other organizational tools like Evernote or DevonThink have their own linking protocol (just like Zotero’s zotero://) that allows you to go straight to an item/note. Now you can get to an item in DevonThink directly from a Zotero note.
2. The indispensable Zotfile has long extracted PDF annotations into notes. Since version 3.1, Zotfile also includes a link to the page containing the annotation with every annotation. Now, with the links functioning, you can click on an annotation and have the PDF open on the right page.

Some Smaller Improvements

• Zotero recommends storing titles in sentence case. Right-clicking on an item’s title, allows you to convert that title to sentence case. The new Zotero version contains various improvements of this feature, including better handling of beginning punctuation (like quotation marks or the Spanish ¡ and ¿) as well as capitalization after punctuation within the title. Now the first letter after colon, question mark, and exclamation mark are capitalized, in line with the APA’s requirements for sentence case in titles. Unfortunately, other styles, in particular the Citing Medicine guide do not capitalize after colons, so that we will roll back that part in the next release. This thread explains the logic behind that in more detail.
• The reference test pane — Zotero’s built in tool that facilitates modifying citation styles—now has a save button. It is no longer necessary to take the detour via a text editor to save CSL styles.
• Both PMID and PMCID are now properly parsed from the extra field (each should start on a new line) so that citation styles using the PMID and PMCID variables, like the NIH/NLM grant style now work correctly.

Free Software at work

Another remarkable aspect of this release is that it is largely community driven: All major improvements are based on patches submitted by different volunteers: Emiliano Heyns (who I’ve mentioned several times before) submitted the patch for faster indexing; Aurimas Vinckevicius, who together with me does a lot of work on Zotero translators and support (among many other contributions) contributed the retrieve metadata improvements; Joscha, the developer of Zotfile, fixed the links in notes; the improvements to sentence case conversion, the reference test pane, and PMCID parsing came from me.

Zotero is an increasingly mature free software project with a broad base of contributors. That doesn’t mean, of course, that Zotero’s developers are idle: For one, each suggested patch—”pull request” in the language of git— gets reviewed very thoroughly, often with a lot of back and forth. Beyond that, Dan adds a constant stream of small fixes and improvements to both Zotero and the sync server, and, perhaps most importantly, has been working on the complete re-write of the sync infrastructure that will be the cornerstone of Zotero 4.1

Beta Test Zotero

Zotero has just started a regular beta release. The beta version replaces what used to be called the “branch xpi”. It is built regularly from the current release branch. In less technical terms that means that using the beta version you’re testing features that will be in the next minor relase. The current beta, for example contains, code changes that will be in version 4.0.18. The beta version is intended to be usable with minimal risk. It will typically not contain database upgrades, so the risk of data corruption is very low and you can easily revert to the regular version if you’re in a pinch. The beta channel is update a lot more frequently than regular Zotero.
The principal reason for releasing a regular beta is to encourage wider testing of Zotero versions pre-release. If you’re interested enough in Zotero to read this blog, there is a good chance you should run the beta version.

You may want to run Zotero beta If…

• You want to help Zotero by testing pre-release versions
• You provide technical support for Zotero and want to be aware of new features before they land (more on that below)
• You like having new and shiny things before anyone else has them

You (probably) shouldn’t run Zotero beta if…

• You want maximal stability when using Zotero
• You panic or get frustrated when something doesn’t work on your computer
• You have no time or no patience to deal with and report occasional problems on the Zotero forums
• You’re only using Zotero Standalone (the beta currently is only for the Firefox add-on).

OK, I want to be using the beta version, what now?

Install

First, install the add-on from here. You can simply install it over your existing Zotero, your database will remain untouched. Heed the warning on that page: I’ve never had any trouble running pre-release versions of Zotero and I’m not aware of anyone who has lost data doing so, but it’s beta software and you want to make sure to have regular and automated back-ups.

Check

You can see the currently installed version in the add-ons tab of Firefox or under “About Zotero” in the gears menu. As of this writing the current beta is 4.0.18-beta.r3+fadd486. This means you’re running the 3rd (r3) beta release for the 4.0.18 version of Zotero. The last part after the plus sign corresponds to the last commit to the Zotero source code that’s included in the version, so if you’re following commits (you should be looking at the 4.0 branch) you can easily check whether the version you’re running already contains an addition to the code.

Report Problems

Zotero devs will very much appreciate any error reports from beta users. As for all Zotero errors, you should report them on the forums, and you should provide plenty of details and, if possible, an error report ID. Also mention that you’re running the beta version of Zotero. I had some problems trying out the beta last night, and you can see my report (with quick solution) here.

Talking about shiny things…

The current beta version contains two major improvements in handling PDFs, both coded by community developers. Thanks to Aurimas, “Retrieve Metadata” has improved significantly and you’re now less likely to get locked out by google scholar. Thanks to Emiliano (whose add-ons I praised in my last post), indexing of large (or many) files is now several orders of magnitude faster—a large PDF like War and Peace could take minutes to index and freeze Zotero before, now it takes a couple of seconds.

What’s New in Zotero 4.0 – Part 2

In the last blogpost I introduced some of the new features of Zotero that help you to collect and organize your references more easily. Zotero 4.0 also has a number of new features that improve citations and make syincing more flexible.

Automatic Journal Abbreviations

Up to now, Zotero took journal abbreviations from the “Journal Abbr.” field. Many databases don’t populate that field properly or at all, so this required a lot of maintenance by users who needed correctly abbreviated journal titles in citations. Now, Zotero comes with a built-in journal abbreviation list. The list takes the journal title from the “Publication” field and matches it against a list of journal titles and words to create a journal abbreviation. The abbreviation list currently only works when using the word processor plugins. For new documents, the option to “Automatically Abbreviated Journal Titles” is active when using styles requiring abbreviated journal titles. In documents created before the release of Zotero 4.0, you can turn on the option under “Set Document Preferences” in the word processor plugin. It will only be available if your chosen citation style requires abbreviated journal titles.

File Sync on Demand

When you sync Zotero to a new computer with file syncing enabled, all item data and all attachments are downloaded. As attachments can take up multiple gigabytes, this can be a long process, impracticable for users who switch computers often. In Zotero 4.0, users can set file syncing to “Download files as needed.” That way, files on the server will only be downloaded when you try to open them. This option is turned off by default.

Relative Linked Attachment Filepaths

By default, Zotero stores all file attachments in a “storage” folder in the data directory. Using file sync via Zotero file storage or a WebDAV server, you can sync these files and comfortably access them on multiple computers. Some users, however, prefer to use their own organizing structure for files in their file system and link to them from Zotero. In the past, this made accessing files on multiple computers difficult. On one computer a file may have had the path ‘/home/workcomputer/Zoterofiles/Important-Paper.pdf’ while on another it was ‘/home/homecomputer/Zoterofiles/Important-Paper.pdf’. The link to this file – what we call an “absolute” link – would only work on one of the two computers. In Zotero 4.0, you can now specify a base directory, and have linked files be relative to that directory. In our example, you would go to “Files and Folders” tab of the “Advanced” tab of the Zotero preferences and on one computer input ‘/home/workcomputer/Zoterofiles’, on the other computer ‘/home/homecomputer/Zoterofiles/’ as the “base directory.” The link to ‘Important-Paper.pdf’ would then work on both computers. This will also work if there is a folder structure within the base directory, as long as it is the same on both computers.

And there is still more…

This is still not everything. Even more new features were introduced in Zotero 4.0. They include multiple performance improvements — for syncing, duplicate detection, pdf metadata retrieval — as well as many little additions that we hope will make Zotero even more useful to you. You can find a full list of all changes here. As always, if you have any questions about Zotero, a dedicated group of users and developers are happy to help you on the Zotero Forums.

What’s New in Zotero – Part 1

This is an extended write-up that I originally prepared for the Zotero blog, but that turned out too extensive for that purpose.

At first glance, the changes in Zotero 4.0 may seem small. It may even have taken you some time to notice that Zotero had updated to a new major version. In reality, though, Zotero 4.0 introduces a wealth of new features — so many, in fact, that it will take us two blogposts to introduce them. In this post I will cover the features that are broadly related to organization and import. The second post will cover features related to syncing, file management, and citations.

Color Coded Tags

Not all tags are created equal. Maybe you have a “to read” tag and you would like to quickly identify items on your reading list. Or you have your favorite articles marked as important. For those and many other cases, Zotero now allows you to assign colors to up to six tags. You can assign a color to a tag by right-clicking on it in the tag selector in the left hand panel. The item will then appear with a colored square before its title in the middle panel. Obviously one item can have multiple color coded tags.

Color coded tags can also be assigned particularly quickly. When you assign a color, you also assign a number to a tag. You can then automatically tag any item(s) selected in the middle panel just by pressing that number on your keyboard.

Adding Multiple Tags and Identifiers at Once

Say you have a list of words that you would like to add to a Zotero item as tags. Or you have a list of DOIs or PubMed IDs that you would like to add using Zotero’s “Magic Wand” add by identifier feature. Up to now, you had to enter these one by one. Now you can paste multiple tags, separated by newlines (return/enter) into the “Add tag” field and similarly input lists of identifiers into the magic wand:

While typically you would copy and paste such a list, e.g. from a text editor or even a spreadsheet, you can also toggle the add tag/add identifier field into multiline mode using Shift+Return.

New Attachment/Note Indicators

You can now add separate columns for Notes and Attachments to the middle panel. The “Notes” column contains the number of child notes (if any) attached to an item. The attachment column – marked by a paper clip – marks items with at least one attachment with a dot. The dot will be colored blue if the attachment is present on the currently used computer and hollow if it is not (e.g. if file syncing is turned off or if files are only synced as needed – a feature that we will introduce in the next blogpost). By the way – we have also added many more options for columns to add to the middle panel.

Display What You are Downloading and Where to

When you import items using the URL bar icon, Zotero now tells you in which collection it is saving and lists all attachments it downloads for an item. That way you will see immediately if an item is saved into the right collection and if e.g. a PDF of an article was successfully downloaded.

In Case you Missed it: IE and Mobile Support with the Bookmarklet

We released the bookmarklet for Zotero several months ago, but in case you missed it: With the bookmarklet you can now take advantage of Zotero’s convenient one-click import from hundreds of sites from mobile devices (saving straight to your zotero.org account) or previously unsupported browsers such as Internet Explorer (saving either to Zotero Standalone or to zotero.org). You can download the bookmarklet and find additional instruction here.