scrambled flat file import - a larger issue

[john harding]

In addition to vetoing two of my duplicates during import from a flat file, Sellerdeck has comprehensively scambled much of the content that it has not vetoed. The sort of things it has done are (1) to load Sections in an order other than that of the spreadsheet, (2) load Products within some Sections in the wrong order, (3) create in some instances two identically named Sub-sections and (4) either divide the Products between them or,(5) in the worst case, throw out all the Products except one and list that twice, in each of the two Sub-sections.

I have tried loading numerous times, and while some shots have been better than others, none has been correct.

It may be that, with help from Tech Support, I can get an accurate catalogue in the end. But that leaves me feeling that I dare not modify the catalogue ever again. What happens the next time I have to raise all my prices to cover inflation?

I feel that an enterprise to which I've devoted 20 years is hanging by a thread. One thing I'm now wondering is whether - if an accurate catalogue can be arrived at - updating it by using the "Link to an External Source" facility (which I know nothing about) would allow incremental updating in a safer manner. I suspect not, because I suspect that it will be in effect an updating full Import, and will screw things up again just like a manual Import.

Somebody please tell me I'm wrong about that.

[NormanRouxel]

I wrote a program that creates a flat file import from a stock system database for Actinic v3 in 2000. It's been running daily since then on several sites from the original V3 to nowadays V11 of SD. It has imported about 5K - 10K products per site every day since then. That's about 5000 flat-file imports per site. It's never created a corrupted catalog AFAIK.

If what you're seeing was the general behaviour of SD there would be many posts like yours.

Some things that may be worth considering to help narrow down what's wrong are:

Have you a faulty database? If so that may cause further corruption when you alter it. Sending it to SD for checking / repairing may be worth considering.

Is your datafile correct? Duplicate products / Sections could be because of extraneous spaces in Section names. E.g. Tools|Widgets may be treated differently from Tools | Widgets.

Is it deterministic? If you import and get a corrupted site, do you get identical or different results if you restore to a pre-import backup and import the same file again?

[john harding]

Thanks, Norman. I'm as certain as I can be that there are no contradictions in the file of the kind you mention. Referring the database to SD sounds like a good idea. But first I'll try deleting the catalogue and reimporting a few times to see, as you suggest, if the errors are consistent.

[NormanRouxel]

One way to check if a site is identical would be to Export the site (File / Export) to a textfile. Repeat after restoring and trying again with your import data. Then re-export to a different file and compare.

You could use WinMerge (www.winmerge.org) to highlight differing records.