confused: Can Actinic handle large amounts of data?

[mark_xigen]

Hi, can anybody help me redarding large quantities of data:

I'm working on a new site for automotive accessories. There's going to be about 10,000 ish products (which I understand is the limit for Access Dbase).

I have just imported a flat file of 'Car Air Filters' with about 4,000 products and LOTS of sub-sections in sub-sections.

Firstly - am I creating a nightmare for myself with this amount of data? Should I be using some kind of MySQL Dbase and calling the data into the store at the web somehow???

Secondly - I've been uploading the site for about 4 hours and it's still going!!!! (At the moment Its only got 3,500 air filters) the .cat files are upto A003020.cat is there a .cat file for every product??

I haven't started designing any page templates yet, I just wanted to get some data in the store to see how it handles it, here's the link:

www.motionperformance.co.uk/acatalog

Finally, if anyone's know a better way to present this kind of data with actinic, then please let me know.

Many Thanks,
Mark
Xigen Ltd.

[chrisbrown]

There's nothing wrong with having >10000 products. That number just indicates the sort of maximum size suitable for an Actinic site. There is no upper limit built into the software.

The upload time for a large site will be long the first time around. There is indeed a separate .cat file for every product, a separate .html for every sub-section and a separate file for every image. The good news is that in subsequent uploads only those files that have changed are uploaded. Your actual upload duration depends on the speed of your link of course.

BTW Did you lose interest when uploading? I can see the page that you linked to in your posting but all the section links give page not found errors.

[mark_xigen]

Sorry, me again.

I've just had a look at the computer after failing to upload. I'm not surprised as it's generated 380mb worth of files (.cat's i believe). And has overloaded my web space (only at 50mb at the moment).

I don't understand why a 800kb csv (flat file import) can generate 380mb of data!! there's no images. Is it due to the amount of HTML pages that's been generated?

I have a horrible feeling a crash course in MySQL/PHP is required.

Any ideas?

Thanks.
Mark.

[mark_xigen]

Thanks Chris, sorry I posted another message the same time as you.

Yes the site has not fully uploaded due to insufficiant web space.

I'm glad it only needs uploading once. It's certainly a shock to find 3500 products = appox 400mb and that's without any images! I've looked at the SiteHTML and that's got 100mb worth of HTML.

I guess my sites gonna be best part of 2gb when finished! Ouch!

I think I might setup a test server on my local (LAN) machine, I have Apache and perl running on XP. If i can get it running, at least I can find out the size of the site, and test it faster etc.

Mark.

[mark_xigen]

Right then - I've just set-up the site running on a local LAN test server, it uploaded in 15 secsonds over the LAN!!

www.xigenserver.co.uk

or

IP address (alternative)

Need sleeeeep!

[NormanRouxel]

A003020.cat is there a .cat file for every product??

There is one .cat file on the server for every Section (i.e Section pages or Product pages).

If you've uploaded 3000+ sections before running out of space then you must have imported nearly every product into a separate section.

This may make it very difficult for your customers to navigate the site.

Norman

[mark_xigen]

Hi Norman,

After uploading the site to my test server is actually totalls 101mb which isn't as bad as I first thought, but still very high.

Yes, I guess it's due to the amount of section (which = .html pages).

I think there's about 12,000 sections! It's running well on the local test server but I do agree, it could be slow to navigate. Will have to continue testing.

Any ideas on an alternative structure?

I only have the Air Filters in so far:

motion performance

Thanks,
Mark.

[nickchad]

Hi Mark

just seen your post - our site www.a1memorabilia.co.uk has around 1600 sections with just under 60,000 products in at the moment.

Because we use Mole-Ends product replicator our products are transfered from Excel into Access before being read into Actinic.

The biggest problem is the amount of memory needed (we had to upgrade 1gb Ram) to read to index.fil file when uploading.

we've had to turn the search function off to stop the server crashing if 2 people are searching at once.

It also takes along time to crank up when starting Actinic each day & refreshes take us about 3 hours & it's closer to 6 for a full site refresh.

But - yes Actinic can handle large amounts of data - it just takes abit of planning whenever your making changes.

Regards

Nick
A1 Memorabilia

[mark_xigen]

I shall keep an eye on the machine running Actinic then, it's only running 512mb Ram.

When you say it crashed the server when using the search, was that a high spec web server running a fast web T1 connection?

I'm slightly worried about the cost to host the site!
At 1gb, it's going to cost a bit on a good quality server.

Mark.

[nickchad]

Hi Mark

We have our site hosted thro' Harlequinn domains (www.harlequindomains.com) - who despite our best efforts to break their server have been very helpful - our last hosts weren't so pleasant (no names or pack drill - but they weren't interested in helping us).

We found them thro' this community - they seem to specialise in actinic sites - so it might be worth getting in touch with them.

Regards

Nick

[NormanRouxel]

Mark,

I tried searching on your site ( the motion performance link you gave earlier) and the server timed out.

You may well need a dedicated high-spec server for this.

Norman

[mark_xigen]

Yes, I will be running it on a good spec server once's it's build. It's only running on XP Home (apache2 + Perl), using BT 512d/256up broadband.

I'm just using it to test. I would imagine it's quite slow via ADSL when your looking at it.

I've see some dedicated 10gb hi-spec servers at about £1000 per year. This is worst case i'm sure.

I just want to be sure this is gonna work and handle the data this way, before i spend too much time on it. I still need to compile the data anyway, so if actinic isn't suited it's not the end of the world!