FTAnalyzer questions.

Part 3: This doesn't work if you have more than one possible match based on name. There may be a minimum amount of information that has to match but past experience shows that a family with "recycled" names, or even cousins of the same name, always tries to match with the earlier birth even if there are lots of data to help the match (and census data never seems to match even if it is the right person - it duplicates).

Phil

 

Pleased it has been of some use, even if not the solution Still, being given the 0 @I..@ INDI reference would make an easier "find" in the editor for the three LC lines to be added, but that still leaves the "CENS" etc duplication problems (unless someone writes a module* to strip out the redundant bits to give a clean Gedcom - it would be trivial** to remove "1 CENS", "1 RESI" entries and all following lines to the next "1 ..." )

I have, from the start, used REFN to provide unique identifiers for both "individual" and "family" records. As you may have spotted, they don't match the FTM @...@ numbering as it renumbers all the records when a Gedcom is loaded; it would be good if FTM used these for matching purposes, but they seem to be ignored

Phil

* I originally wrote "mudule" here

** in BASIC
do
line input #1, z$
if instr(1,z$,"1 CENS") > 0 or instr(1,z$,"1 RESI") > 0 or instr(1,z$,"1 OCCU") > 0 then
do
line input #1, z$
loop until (left$(z$,1) = "0" or left$(z$,1) = "1")
else
print #2,z$
endif
loop until (eof(1) or instr(1,z$,"0 TRLR") > 0)

 

Firstly, there is a mistake in the BASIC above, it should read (I have verified the change):

line input #1, z$
do
if instr(1,z$,"1 CENS") > 0 or instr(1,z$,"1 RESI") > 0 or instr(1,z$,"1 OCCU") > 0 then
do
line input #1, z$
loop until (left$(z$,1) = "0" or left$(z$,1) = "1")
else
print #2,z$
line input #1, z$
endif
loop until (eof(1) or instr(1,z$,"0 TRLR") > 0)

I only know about FH and FTM, so

For FH or other programs, where the program uses a Gedcom file (not a "own" format file), you can just use the original suggestion and insert the three lines immediately after the appropriate 0 @I..@ INDI line in that file (it will get moved next time the file is opened); it's a bit cleaner than adding duplicate 0 @I..@ INDI lines by inserting a separate file, though that should also work.

Edit: It appears it is possible to merge a Gedcom into FTM without duplicating census data etc, but it only works for me if it is an exported version of the tree and, therefore, the records are in exactly the same order (which they are not with my normal merge routine). In this case, Tim's method works.

The following is only relevant if, like my normal "merge" file, the Gedcom is generated in a different order or with different individual/family numbering.

For FTM, because it reads a Gedcom and saves its own ".ftm" file, it can be a bit more complex. If you are running it as a stand-alone program, then you may get away with loading the modified Gedcom as a new tree (not something I have tried, so I'm not sure about media links) but you have to either change the name of the Gedcom or delete the old .ftm file, as far as I know (feel free to contradict me). What you cannot do is merge a Gedcom into a tree loaded in FTM without the census entries etc being duplicated, they have to be removed from the Gedcom first or within FTM.

If you have FTM linked to an Ancestry tree, deleting the ".ftm" file or all bar one of the individuals is a major hassle, all the hints will start from scratch . I believe the best way for FTM is the single approach of deleting all the CENS, RESI and OCCU entries from the Gedcom and adding in the LC entries, preferably after the appropriate 0 @I..@ INDI line though creating a second file and adding that into the Gedcom will (I believe) work and be simpler for most people.

Phil

 

Tim,

I'm getting inconsistent results from various tests I have done, so need to find why.

Regarding a new tree, the export from FTM automatically adds a date reference to the name in the save dialogue box, so you can edit that as needed and that will be the default name if you just load the file into FTM as a new tree.

You can usually tell the success status by the merge dialogue sequences, but it is not definitive.

Phil

 

Doesn't FTA provide this already or am I missing the point?

The Lost Cousins tab in FTA shows the number of Lost Cousins facts missing from the Gedcom file by census year.
Clicking on the corresponding button for country/year then displays a window showing the individuals concerned.
That can then be verified by looking at the Colour Census Report where the missing fact occurrences are indicated by Yellow tiles.

 

A quick update: things are not looking promising.

If the new data is merged from a file with the appropriate "0 @I " entry and standard header/footer entries only, a new unnamed person is added without links to anyone else.
If the new data is added within the persons data after an existing LostCousins entry or at the end of the Gedcom file with the appropriate "o @I " reference, the entry is being ignored.
If the new data is added immediately preceding an existing entry, it is added and the existing entry is duplicated.

The only (slightly) good news is that altering an existing entry by inserting the new date does work.

Also, the event has to be added to the list manually for every tree - it does not appear globally nor get created automatically (though both happen with FH, I believe, the latter definitely with automatic creation turned on).

Phil

 

I agree that manual updating of possibly hundreds of tree individuals with custom LC facts is a daunting task. However, FTA does supply the required information to allow such manual updating to be performed.

The problem seems to be not so much with getting the right information into the Gedcom file for loading into the FHS but rather with the limitation of the FHS itself. Gedcom merge facilities appear to perform the addition of new individuals into an existing tree but not the addition of new data for existing individuals unless such individuals are completely replaced in their entirety.

I suggest that as many people as possible should request that the FHS developers of each product extend such merge facilities as soon as possible. There has been a general discussion on the GenoPro Forum claiming implementation difficulties where new data conflicts with existing data but I believe that new data should simply over ride existing data without any consistency checking unless that is already provided for manual updates.

 

Bryman, whilst in basic agreement with you, there are many programming/logic difficulties within matching processes to determine whether an item should be added or replaced during a merge - even the apparently simple such as whether a census entry with a "DATE 1881" is the same as one with "DATE Apr 1881" or "DATE 3 Apr 1881".

At this time, there are probably more pressing items to change, as far as the developers are concerned, such as the memory leak in FTM that locks out 3.5GB of memory in a minute or two when running an Ancestry sync on a "largish" tree causing FTM or Windows 10 (in my case) the collapse in a heap

So, while we wait for changes, we need a simple and effective method of using the information available from the LC website and FTA to update the LC list and add LC events in our preferred FH program.

The mechanism for adding the custom event is already available and the advice should be to always add this manually in the FH program before starting to add the entries - though most FH programs would add them automatically.

To avoid getting out-of-step, it is best to add an entry to the LC website and to the Gedcom file before moving to the next person. If you are trying to add LC flags for entries already on the LC website, treat each one as a new entry - for this it is advantageous to sort the FTA list data by Census Reference, which will put them in the LC website order. It is probably safest to only add data for one census from the LC list at a time.

Adding data to the LC website is already covered so we need advice on updating our trees. I believe this falls into two groups - FHPs that use Gedcom files as the primary storage and those that use a proprietary format file. I have used 1881 below, but this should be replaced by the appropriate census year for the entry you are making.

Where the storage file is a Gedcom file add the following three lines:

1 EVEN
2 DATE 1881
2 TYPE LostCousins

to each individual from the FTA list that is (now) on the LC website without a flag (by searching for I... in the Gedcom, where I... is the Ind Id provided in the FTA list) immediately before any line starting with a 1 after the 0 @I...@ INDI line. There is no further action needed.

For proprietary format files, create a file with a .ged extension (such as update.ged) using a text editor and insert the lines:

0 HEAD
1 SOUR FTM
2 VERS 22.0.1.1474
1 GEDC
2 VERS 5.5.1
2 FORM LINEAGE-LINKED

Now add the following four lines for each individual - incrementing the number by 1 as each new entry is added

0 @I1@ INDI
1 EVEN
2 DATE 1881
2 TYPE LostCousins

Finally, add the line

0 TRLR

to the end of the file.

When complete, content of this file should be added to the original tree, in the case of FTM, by using merge and selecting add individuals without merging. This will just add a series of unnamed individuals with no links to anyone else and will not affect the original data. Finally, each person to whom the data applies is merged with one of the unnamed entries using Person->merge two specific individuals.

A bit strange, but it worked reliably every time I tested it - though I'm not convinced it saves a lot of time/effort

Phil
(Apologies for the length!)

 

I agree Phil. That is why I only suggest a simple extension of any merge process to avoid getting wrapped up in such discussions. When manually updating, changes are only made if they are considered to be beneficial. If in doubt, don't update. I believe that the same should be the rule for mechanised situations too. Unfortunately, it seems to me that FHS has taken that to an extreme and is unnecessarily restrictive.

FTA provides the required information to allow missing LC flags to be identified and inserted manally. That is fine for situations where only a 'few' updates are needed. Where a much larger number of updates is required, for whatever reason, I appreciate that something faster than manual updating is to be preferred and Gedcom input seems to be the only realistic means possible.

The basic use of Gedcom input is to supply data for a new tree. Some programs have an additional 'merge' capability in which individuals/families in the Gedcom input are to be added to an existing tree. If such individuals/families already exist in the specified tree then they are replaced by the new data. That is not particularly useful in our situation where we need just the minimally supplied data to be used to update existing individuals rather than to replace them.

Without such a minor enhancement to the FHS program, I fear that all of your meticulous trial and testing will come to naught. Although you have identified a multi-step process (which you say may not save much time/effort), I think that it will appear too difficult for most users to attempt. Somewhat similar to assembly of flat-pack furniture where instructions appear daunting and inadequate at first but become unnecessary when fully understood by an experienced assembler.

Is it worth asking for an extension to FTA in which the appropriate Gedcom file is created automatically when requested by the user? That might make your approach more appealing.

 

As my conclusion to the recent series of posts is that automation is not the way forward, I wish to make a request for consideration.

The output of FTA for "Lost Cousins" is the best way to process data, so I would think that exchanging the columns "Likely Census Location" and "Census Reference" would improve the layout by putting all the information needed together in the early columns of the list.

This would allow the browser Lost Cousins page and FH program to be placed over the right hand part of the FTA page for ease in copying the data across.

Phil

 

Tim,

I thought it would be possible (now done!), but didn't think of dragging My excuse is I was time limited as I had to go out

Anyway, there is a minor flaw in my thought process - we need the "census name" not the actual name to get the matches. For me, that's relatively easy as I include a note of the form:
Name on Census: Martha Fussell; Age: 65 (1776)
for each census record, so I can find that when adding the LC event in the FH software.

Phil

 

Tim,

FTA gives "the name at the time of the census", plus the "NAME" surname where appropriate, as the "census name" so, as an example, I get Margaret Creese (Williams) for 1841 Piece:751 Book:18 Folio:46 Page:34, but the name on the census (and FMP transcription) is "Margaret Cross". Also the age is based on the FHS date which can vary considerably from the age given on the census.

I'm not criticising FTA in any way, it is extremely useful for checking so many things, but it can only display data it has been given in the Gedcom and users must be (repeatedly) made aware of the possible differences between the data FTA displays and the data that should be entered on the LC website, to ensure an effective match result, in addition to the FAQs available. Most people know that you can add "actual" information as supplementary data on LC, so my example contains both Creese and Williams there.

Phil