Managing reference checking

Tim, it's not about whether or not members are using FTA to upload their entries - the simple solution I put forward (which had already been suggested by another forum member) works for everyone, whether they are new members or existing members, and whether they are using FTA or not.

If you still disagree please put forward an example of a situation in which it would not work (assuming that the member concerned behaves sensibly). I can't think of one, but maybe you can?

 

Sorry but this is PRECISELY my point. If there is a simple easy to check method for users to confirm they have processed all the records correctly then even someone with 10,000 records entered would be able to see at a glance what they had checked. Thus such a tick box would dramatically ENHANCE the quality of the data, NOT diminish it.

It is because the entire design of the website lends itself to hideously manual and tedious processes that users are actively turned off from entering data and checking what they enter. It is this bad design that threatens the integrity of the database not the entry of data. It is no coincidence that the vast majority of the website users have entered very few of their ancestors and have checked even less.

This is what REALLY grates with me. A trivial change to the website would massively IMPROVE the integrity of the data for ALL users old and new, manual and automated is dismissed as users can do it manually at the time of entry. It is this head in the sand ignoring the BLINDINGLY OBVIOUS fact that users are NOT doing the checks that makes my blood boil. I cannot for the life of me fathom why Peter seems to want his website to be tedious to use and off-putting to users. As Tim says there are millions of records that could be added and could be checked very easily and very quickly. I am more than happy to assist writing the software to do this whether that be on the website end or the FTAnalyzer end. I want the website to succeed as it will help thousands of people but time and time and TIME again we get the same stuck in the past attitude and I'm at my wits end and ready to throw in the towel. As it's just not worth my effort if every little thing I try needs to be done by forcing blood from the stone.

If there was an easy way of seeing what you'd checked then it would be simple for users to upload records then work through checking them. The check would show as unchecked until the user had ticked the box. I've proved time and time again that when users are given a nudge to clean up their tree they lap it up and are keen to make all the errors disappear. I have no reason to think that anything other than a vitreous circle would be created with this feature. We'd have more records entered. They would get checked and more matches would be made thus encouraging more entries etc.

Remember that at present all the user sees is a little arrow icon on their my ancestors page. I would be willing to be a very large sum of money that most users are completely and utterly unaware that they should be clicking this for each and every household and checking. I'd be willing to bet an even larger sum of money that the vast majority of users have checked only a handful of entries if any. Giving users a nudge to check their entries could only improve the quality of the data as my guess would be that 95% plus of the data entered has never been checked - I think this because a large percentage possibly over 50% of the data entered was entered before this feature existed.

 

No, as I explained in the email I sent to everyone who participated in the competition, the standard for the 1881 Census is the LDS transcription, because that is all that was available when LostCousins began, and for some years afterwards. The England & Wales part is now only available online at Ancestry; the Scotland part is now only available online at ScotlandsPeople.

 

My suggestion is that a checkbox shows that the ancestor has been checked if the user ticks the box it could show that status. If there isn't a link to check an entry then the entry would not show as checked. If the user checks via another method then it could show as verified.

At present there is NO WAY to see what has been checked and what hasn't. Since the entire validity of the database depends on whether the records are accurate and Peter is so adamant that users should be checking how does anyone know whether they have checked their records or not. If it's a tedious manual process it puts people off doing that work. Especially since there is a mismatch between the data displayed on the link (doesn't need a FMP subscription) and the data on the my ancestors page (one shows a year one shows an age). To see the same data to make the check easy you need to view the image which requires a subscription. Thus this makes it even harder for people to verify the data another needless hurdle.

 

REALLY??? Sorry Peter that comes across as you being deliberately obtuse. This sort of reply is precisely why I feel I'm wasting my time.

Users are NOT checking their entries whether they enter MANUALLY or via FTAnalyzer. It is NOTHING TO DO with FTAnalyzer whether users check their entries or not this suggestion is about encouraging ALL users to check their entries.

HOW CAN YOU NOT SEE THAT??

I am mad as hell as this head in sand seemingly deliberately obtuse nonsense.

 

Ok, so I have now sorted in date order. How does this help me? Where have I got to? When I started checking and stopped, I was using the list as it was displayed to me, which is in houshold order. I assume then that I'm not behaving sensibly?

I think the point that is trying to be made is that you don't actually know who, or how many households are being checked. You've added a feature which helps users to check, but it doesn't flag to you or to the user that the job has been done.

Having users take pictures or jotting down on paper where they've got up, or some other workaround that they've come up with should be telling you that there is something missing on the website that could help the users.

 

Tim, I haven't just added this feature - it has been in place since 2014. It's likely that most of the members who are ever going to check their entries have already done it.

The main exception will be the members who are using FTA to input new entries, and it's because FTA has only offered this option since December that I suggested that they could sort their My Ancestors page by date.

 

It's a good thing to get feedback from users, especially power users - however I have to consider the interests of the membership as a whole, and sometimes a feature that seems attractive to some can have negative consequences for others.

As I hinted in my earlier post, the time to have introduced the system that is now being proposed would have been 2014. Bringing it in now will disadvantage the members who have already checked many, most, or all of their entries - either they will have to check them all over again, or else I will have to provide members with a means of changing the colour of the arrows for all entries made prior to a certain date.

I know from previous experience that this will cause problems for a proportion of the most loyal members. Some will be demoralised. Some will spend their time unnecessarily double-checking entries rather than making new entries. Faced with the uncertainty some might even decide to close their account.

It really isn't worth taking the risk. The number of incorrect entries is very small as a proportion of the total - and even where there are discrepancies the chance of a match being missed is low, since most matches involve multiple relatives in the same household, and many involve multiple households. The near match flags also pick up a high proportion of matches that might otherwise be missed.

When the arrows were first introduced I described them as being for checking the census references. There were two reasons for this: one was to encourage members entering relatives for the first time, since it was clear that many were concerned they weren't "doing it right" (in most cases they were, so the arrows provided peace of mind).

The other reason for focusing on census references is because when a match is missed it's almost always because the wrong census references have been entered.

Following the introduction of the arrows the proportion of entries where the census references are incorrect has fallen. Where there are still errors they are rarely in the 1881 Census, which is the one that produces nearly 90% of all matches - they are usually in the 1841 Census, typically where Ancestry has been used as the source (as you will know, Ancestry get the folio number wrong about half the time).

If I had to estimate how many cousin matches are currently undiscovered because of incorrect entries I would say it is definitely fewer than 50, and quite possibly fewer than 10. These are very, very small numbers - they're low partly because few errors have been made when entering the 1881 census references, partly because many matches involve multiple households, and partly because some members have corrected errors since the arrows were introduced in 2014.

No matter what changes are made only a proportion of the remaining errors will be fixed by the users - it is very difficult to overcome inertia, especially when we're not asking members to fix specific errors, but to carry out checks in case errors have been made.

There are better ways of eliminating errors from the database. Given the time I could write some software to audit all of the 1841 census references that have been entered so far - we have sufficient entries in the database that this is feaible. In practice this would produce more additional matches than what is being proposed - and with far less disruption, since it would not require any involvement by members who have entered their data correctly (the vast majority). Having written the auditing software there would be the potential to build in a similar check at the time of input.

(It would also be possible to do the same for the 1881 Census, though because the number of incorrect census references is so low it might not produce any additional matches.)

All suggestions are welcome, but when advocating change it is important to consider how other members might be adversely affected.

 

What would be more feasible is to implement a system which applies only to entries uploaded by FTA. This wouldn't affect users who rely on manual input, so would reap most of the benefits without any of the disadvantages (except for the extra processing time when the My Ancestors page is displayed, or the Search button pressed).

 

There are a few differences between the 1881 transcriptions at the two sites, but not enough to worry about, However, if you notice that there is a difference then it's the Ancestry version you should use because (as mentioned in my email of 2nd Feb) they are the only site to have the unaltered LDS transcription.

It doesn't matter which site you use as your source for the 1841 and 1911 Censuses as we use the handwritten information, not the transcription. But if you are tempted to take the 1841 references from Ancestry's transcription (rather than reading them off the census page, as I recommend, bear in mind that the folio number is wrong about half the time.

It's very unusual for an age to be incorrectly transcribed in 1881. But again, if you notice a difference between Ancestry and Findmypast use the age that Ancestry show.

 

Because:

• originally the buttons were for checking census references only, so occasional discrepancies in the transcription didn't matter
• at that time I hadn't worked out how to link to Ancestry, and as I had to link to Findmypast for the 1841 & 1911 censuses it was more straightforward to use the same procedure for all three censuses
• members need a Findmypast subscription to view the 1841 and 1911 images; for them it would be more convenient to link the 1881 census to Findmypast as well
• Findmypast is the only site that can produce a printout of a household showing all the relevant information in a convenient format

However, for me it would be more convenient if the button linked to Ancestry for the 1881 England & Wales census if it can be done - which I would have to check. Ancestry have already changed the way their Scotland 1881 search works so that it no longer works as well as it did when first implemented.

Since 2009 I have had to adapt the site to accommodate changes made by Ancestry, Findmypast, and FamilySearch over which I have no control and which often introduced incompatibilities that didn't exist in 2004-8. All changes have to respect the integrity of data that has already been entered - I cannot expect members to go back and change entries that were correct at the time they were entered.

On the positive side, Findmypast did add the book numbers which were missing for about one-third of the 1841 Census - without this upgrade the 1841 reference check simply wouldn't be possible. Another positive - although it might not sound like it - is that Findmypast completely removed the Scotland census references, which were always mislabelled and often incorrect. But the biggest positive of all was ScotlandsPeople providing free access to the LDS transcription.

 

This isn't usually a problem, because most users input an entire household one after the other. Is it, perhaps, to do with the order in which FTA uploads entries?

If the earliest entries are checked first it's just a matter of noting which was the last household/entry to be checked. (Though ideally one wouldn't to upload more entries than can be checked in one session.)