Clara upgrading and settings issue

[fleetwood]

There's definately something 'funny' going on.
How can a stats package make your transfer rates increase this much?

You'll also notice that if you click on 'subdomains' you get a nice new bar graph in bright red, warning you how close you are to using up your allowances.
We've already blown ours for this month, and were not even half way through the month!!
Everythings still working, but I'm just waiting for an email telling me I need to upgrade.
Actually, I tried to upgrade just to protect my position (we were nearing limits before all this happened, just it hadn't gotten as critical as the stats make it look now), but they don't wanna make it easy - the automatic upgrading site dosen't function, and the telephone dept didn't wanna answer to take my money!!
Looks like we will all have to watch this one closely...

Martin

[fleetwood]

Hi Jo
Our access.logs are around 35MB per day. No great worry on their own, just they rapidly eat up valuable webspace if left unchecked - 1 months worth = 1GB - more space than most of us have in total.

Is there any configuration you use that cleans the old files down, and is there actually any worth in keeping anything other than the most current day or two?

thanks

Martin

[pinbrook]

ok

in the awstats /data folder we keep one file per month called awstatsmmyyyydomain.txt. I am looking at a site that has 600-1000 visitors per day, and June's file is just under 1MB.

Each day (4am job in the morning) the previous day's traffic is appended to the txt file for graphical display.

Therefore the whole year is using less than 12MB.

In the top level logs folder we hold log files too but we recycle these every5 days, these hold transfer data and error logs which can be downloaded for analysis in other packages, these are on average 0.3MB stored as gz files.

[fleetwood]

so basically, something is going a bit pear shaped in our Clara AWSTATS folders.
I have 1500 or so visitors a day, and looking at 35MB for each days stats!
These are the files called access.logs.14102005 etc.
Haven't got to the end of a month to see a monthly file yet.
There has got to be an explanation for this, as there is no way we can be expected to delete our own logs on a daily basis, just to keep webspace under control!

[buspassjohn]

so basically, something is going a bit pear shaped in our Clara AWSTATS folders.
I have 1500 or so visitors a day, and looking at 35MB for each days stats!
These are the files called access.logs.14102005 etc.
Haven't got to the end of a month to see a monthly file yet.
There has got to be an explanation for this, as there is no way we can be expected to delete our own logs on a daily basis, just to keep webspace under control!

Just a thought, could it be a cunning plan to sell more bandwidth?

[Bob Isaac]

Depending on how Clara have set up Awstats and access log files, they may be deleted automatically after 4 weeks or after reaching a certain size. Again, depending on setup, the daily stats may be converted to .txt files and again may be deleted after a set period. They should be kept for at least 12 months to get your annual overview. Awstats can also be configured to delete log files after processing them.

I don't think this is anything to worry about, and I am not sure if the log files are part of your web space allowance.

Bob

[fleetwood]

Awstats can also be configured to delete log files after processing them.

I don't think this is anything to worry about, and I am not sure if the log files are part of your web space allowance.

Bob

I think it may be these raw log files that are the cause of the problem.
I don't know how all the stats stuff works, but reading between the lines from what you have said Bob, it sounds like there is a daily log file created of raw data, which is then processed to produce the stats. I am guessing it is these raw data files (mine are 35MB a day) that are what are being left behind to fill up our space, and yes, clara is counting these as using web space allowance. Maybe this is unintentional, as they have only just switched over to Awstats, and to be fair, they have not hassled me to say "your over your space allocation - upgrade or else!", although when you view the control panel sections, it does warn that your over allocation, which is a scare!
Deleting these 35MB daily files dosen't seem to affect the ongoing stats ie you can still view all the previous days report. I assume these daily files are of use to those of you know who know how to extract further data from them.
Bob - do you do further analysis work on your stats data, over and above the page awstats produces which shows all the graphs of data, hits, bandwidth used, keywords searched etc?

Martin

ps I'm taking a well earned break over the next few days - off to explore Yorkshire, so I'm going to try and forget all about stats for now - yipeee

[Bob Isaac]

First you need to read this about web stats:

http://www.analog.cx/docs/meaning.html

I use them more to ensure there are no errors such as incorrect links and missing files and graphics, although my site search software picks up a lot of those anyway. They are usefull to get a feel of what is going on but don't be ruled by them.

Awstats are the best of the free bunch and having looked at the likes of http://www.surfstats.com which you have to buy, I cannot see anything that Surfstats does that Awstats does not. Except Surfstats has a more polished look about it.

I presume you access the logfiles via an admin panel they provide, since deleting the raw logfile does not affect previous stats then awstats is converting each days stats into a text file. On our server these text files get to about 6Mb for the month, and that is with about 90,000 unique vistitors a month. The access logfile that is the raw access data gets to about 370000K a week and at the end of the week when it rotates it gets compressed and then deleted after 4 weeks.

Ask Clara how the server is set up to handle the logfiles and awstats files.

Bob

[party_pete]

my site search software

What product are you using Bob?

This is something i've been looking into for some time but cannot make a decision.

[Bob Isaac]

I use htdig which is open source. A bit fiddly to install at first but when you get the hang of it it is quite good...and free.

To ensure server security you should **not** install it off web site root. It is a Perl program. I have no idea how to install it on a Widows server, but if you want to try it and have root access to a *nix server then drop me an email.

Bob

[pinbrook]

Fluid dynamics have a good site search too.

[george]

Awstats tweaking....

Rather than start a new thread I'm hoping the onsite Awstats experts can help me. Two things.

Can it be adjusted to show an individual days stats, as opposed to the complete current month?

And. My old stats software (Advance Logger) showed the complete url of where visitors came from, ie, the individual thread on a message board, whereas Awstats only brings up the forum index. Not likely, I know, but...

[Bob Isaac]

Rather than start a new thread I'm hoping the onsite Awstats experts can help me. Two things.

Can it be adjusted to show an individual days stats, as opposed to the complete current month?

And. My old stats software (Advance Logger) showed the complete url of where visitors came from, ie, the individual thread on a message board, whereas Awstats only brings up the forum index. Not likely, I know, but...

The first question, awstats will show figures by the hour for pages, hits and bandwidth but will not tie it down to what page during what hour.

The second question, it will show the actual refering site file by name, if that is what you want to know. (Links from an external page (other web sites except search engines) ). It will also show the URL of each page visited on your web site, but not individual forum threads.

I maybe wrong but it sounds like you are trying to split the atom.

My awstats is set to generate the previous days stats at 0001 and in that minute it is quite intersting to see how much activity the web site has had, and how it varies from day to day. Stats do not portray an acurate picture, only a tend.

Bob

[george]

Thanks for the reply Bob. I could've explain that a lot better. With Advance Logger you gould see a complete days stats, including that days referrers. So much so that you could see unusually high traffic, trace it quickly, and even respond on the forum if it was appropriate. Awstats tends to hide the daily stuff within the month.

Best thing I've noticed are the nice folk who link to our badge images as avatar's in forums. A `hit` every time. Grrr.

[Bob Isaac]

If awstats does that I have not found it in the conf file. It may be something they will include in later versions, if people want it.

Bob