Intelligence you can count on

A reader posed the following question in response to our previous post on enhancing FPinfomart content:

“As a throw-back to the first article in this series, I’m interested in learning more about which content is included in Infomart. For example, Infomart offers the National Post as a source, but not all articles from the National Post are in the database. Can you tell me what the criteria is for whether articles are included? Does it always have to do with copyright? Is there any way that an Infomart user can determine whether articles from a subscribed source are missing?”

This is a great question, and definitely merits its own post.  And, the answer, like many other things is (unfortunately), “it depends.”

There are 4 basic reasons why an article that was published in a newspaper would not be available on FPinfomart:

1. The author is a freelancer (i.e. not employed directly by the newspaper or publication in which his/her work is published), and has not authorized the publisher to distribute the work in any other medium than the original printed format.

2. The story is republished from a 3rd party source (usually a newswire, such as Reuters, Canadian Press, Associated Press, etc.) who has authorized the newspaper to publish their copy in print, but not to distribute the work in any other medium.

3.  The story has been retracted or suppressed (usually for legal reasons) after publication.

4.  The article was actually paid content, such as an ad supplement, an advertorial, “special section,” Appointment Notice, etc.  In the majority of cases, FPinfomart does not carry such paid content, as a 3rd party has paid the publisher to run it, and therefore it does not carry through to a database such as ours.

The part of this answer that is a bit fuzzy is exactly what happens when an article meets one of the criteria above.

For content that is owned or enhanced by Canwest, we actually add ALL content (with the exception of that belonging to type 4, above) to our database.  We keep a “blacklist” of content that is not allowed to be distributed by FPinfomart for the reasons specified above (specific bylines; original sources, articles that have been retracted, etc).  When displaying a list of results to you, we check all articles against this “blacklist,” and if any meet the criteria for suppression, we display this message to you:

The advantage here, of course, is that even though you can’t read the text, you at least know the article appeared, when, and the citation details.

For content that we receive in database form from other publishers (the hundreds of sources we do not enhance ourselves), we are not able to do this, because the publisher removes the articles (also using the criteria stated above) BEFORE they send us the content.   Therefore, we never receive the restricted articles, and cannot provide even citations for them.  We are not provided with a list of suppressed content, and therefore it is as invisible to us as it is to you.

Thank you, Michelle, for submitting your question.  I’m sure there were a lot of other FPinfomart users wondering the same thing.  I hope this explanation provides the answer you were looking for.

More questions about how FPinfomart works?  Let us know in the comments.  You may inspire our next post!

Earlier in the week, I posted a description of how the sources that make up FPinfomart are selected.  Today’s post addresses the technical aspect of “getting content into FPinfomart.”

As I was composing the first post in this series on “how content gets into FPinfomart” I had intended to describe only the licensing issue.  As I wrote, however, it occurred to me that the actual process of moving content from a variety of media into a single, uniform format may also be a bit of a mystery to a lot of our users.  Today’s post attempts to lift the veil and give you a bit of insight into just how we make 4,700+ sources searchable, consistently-formatted, and usable in a database.

How do we get from here...

... to here?

We have many sources for our data.  Some of it comes to us from 3rd parties (publishers, aggregators, etc.) while other content is produced here.  Some of the content comes pre-formatted for us, while other content is run through various automatic conversion programs, and some content is converted through more manual processes.

This post will focus on a process called “Enhancing,” which is how we turn a page formatted for a printed newspaper into separate articles in the FPinfomart database.  We Enhance all the Canwest-owned newspapers, as well as close to 100 additional community papers for whom we provide this service (without which they would have no digital archive of their own newspaper).

Once the paper is sent to the printing press, an electronic copy is sent to us in either PDF or Quark format.  Since this file is laid out for printing, we need to do some work not only to separate out all the individual stories, images, and ads, but also to ensure that the various parts of each story are correctly identified – such as the headline, the byline, the page number, the name of the section in which it appeared, etc – i.e. the fields in our database.

To do this, we use a piece of custom-built software.  We run the formatted pages through this software, which automatically detects upwards of 80% of this information, placing the appropriate information in the correct field.  The data that the software can’t detect, (and some that needs correcting) is handled by our skilled team of Enhancers (individuals trained to get the right data to the right place, very quickly, accurately, and often in the middle of the night).  We have Enhancers working around the clock, because the news never sleeps!

Many of the larger papers that we enhance here arrive in our Enhancers’ inbox shortly after 2:00 a.m. and are online by 4:00 a.m. – which often means you can read a copy of these papers on FPinfomart before they’re available in print.

There are over 70 million documents in the FPinfomart database, and we add between 30,000 and 40,000 new ones every single day.  And, for each of these documents, you can search in standardized fields – such as headline, byline, source, and section.  It’s an amazing undertaking.  I’ll take a moment to thank our Enhancing team and our technical staff for providing us with some order in this potential chaos.  It is because of the routines they’ve created that we are able to find the data we need, without being lost in information overload!

Are there any other mysteries of FPinfomart that you’d like to see described here?  I’d love to hear your suggestions on what other inner workings you’re wondering about.  Use the comments to let me know what interests you! Consider this your backstage pass.