Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo User (http://www.linux-archive.org/gentoo-user/)
-   -   syslog-ng filtering (http://www.linux-archive.org/gentoo-user/342602-syslog-ng-filtering.html)

Ralph Slooten 03-16-2010 10:22 PM

syslog-ng filtering
 
Hi all,
Has anyone here worked out how to filter out syslog messages using�syslog-ng v3? The old syntax doesn't work (well complains bitterly about�performance and says to use regex), and no matter what I try I cannot get the new syntax to work :-/ I have a syslog-ng server which logs to MySQL for multiple clients in a network, however the database just keeps growing with irrelevant data I'd prefer to just quietly ignore on the server side.�

I'm trying to filter out (exclude) messages such as:
��(root) CMD (/root/bin/vmware-checker)
and
��(root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

==============
filter myfilter {

�� � � �not match("regex" value("/usr/sbin/run-crons"))
�� � � �and not match("regex" value("vmware-checker"));
}
log {
�� � � �source(src);
�� � � �source(remote);

�� � � �filter(myfilter);
�� � � �destination(d_mysql);
};
===============
However they just keep coming through the filter (ie: not matching the "not match" filter). I've tried escaping the slashes, not escaping them ... even partial words, but I obviously am missing something somewhere.

Anyone have any ideas?
Thanks in advance,Ralph

Roy Wright 03-16-2010 11:00 PM

syslog-ng filtering
 
On Mar 16, 2010, at 6:22 PM, Ralph Slooten wrote:

> Hi all,
>
> Has anyone here worked out how to filter out syslog messages using syslog-ng v3? The old syntax doesn't work (well complains bitterly about performance and says to use regex), and no matter what I try I cannot get the new syntax to work :-/ I have a syslog-ng server which logs to MySQL for multiple clients in a network, however the database just keeps growing with irrelevant data I'd prefer to just quietly ignore on the server side.
>

I just started with the example at:

http://en.gentoo-wiki.com/wiki/Syslog-ng

HTH,
Roy

Ralph Slooten 03-17-2010 01:49 AM

syslog-ng filtering
 
On 17 March 2010 13:00, Roy Wright <roy@wright.org> wrote:
>
> I just started with the example at:
> http://en.gentoo-wiki.com/wiki/Syslog-ng

>
> HTH,
> Roy

Thanks Roy, however they have the same syntax which isn't working on my side.

filter f_shorewall { not match("regex" value("Shorewall")); }*
I just tried a single rule (to make sure it wasn't my syntax):
filter killVmMessages {

** * * *not match("regex" value("vmware-checker"));
};
yet the "(root) CMD (/root/bin/vmware-checker)" messages still go through?!*
log {
** * * *source(src);** * * *source(remote);** * * *filter(myfilter);** * * *filter(killVmMessages);** * * *destination(d_mysql);};
I'm really stumped here. All other filters (non regex) works fine though, such as*facility() &*host().

Are you able to filter by content?
Ralph

Alan McKinnon 03-17-2010 09:39 AM

syslog-ng filtering
 
On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
> Hi all,
>
> Has anyone here worked out how to filter out syslog messages using
> syslog-ng v3? The old syntax doesn't work (well complains bitterly about
> performance and says to use regex), and no matter what I try I cannot get
> the new syntax to work :-/ I have a syslog-ng server which logs to MySQL
> for multiple clients in a network, however the database just keeps growing
> with irrelevant data I'd prefer to just quietly ignore on the server side.
>
> I'm trying to filter out (exclude) messages such as:
> (root) CMD (/root/bin/vmware-checker)
> and
> (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
>
> ==============
> filter myfilter {
> not match("regex" value("/usr/sbin/run-crons"))
> and not match("regex" value("vmware-checker"));
> }

Hah! this caught me out too.

The value of "value" cannot be anything arbitrary - syslog-ng has no clue what
you mean. The value is a field name, either a pre-defined one, or something
you defined using a parser. The docs are ambiguous on this, it's not clear
that the supplied values are abstracts. You are truing to search for the
string "regex" in a field called /usr/bin/vmware-checker.

Which obviously will not work.

I think you want:

match("/usr/sbin/run-crons" value "MESSAGE")

Note that it is MESSAGE. You want the field name, not it's dereferenced value.



> log {
> source(src);
> source(remote);
> filter(myfilter);
> destination(d_mysql);
> };
> ===============
>
> However they just keep coming through the filter (ie: not matching the "not
> match" filter). I've tried escaping the slashes, not escaping them ... even
> partial words, but I obviously am missing something somewhere.
>
> Anyone have any ideas?
>
> Thanks in advance,
> Ralph

--
alan dot mckinnon at gmail dot com

Fred Leon 03-17-2010 09:48 AM

syslog-ng filtering
 
Ralph Slooten <axllent@gmail.com> a crit*:


On 17 March 2010 13:00, Roy Wright <roy@wright.org> wrote:


I just started with the example at:
http://en.gentoo-wiki.com/wiki/Syslog-ng

HTH,
Roy


Thanks Roy, however they have the same syntax which isn't working on my
side.

filter f_shorewall { not match("regex" value("Shorewall")); }


I just tried a single rule (to make sure it wasn't my syntax):

filter killVmMessages {
not match("regex" value("vmware-checker"));
};

yet the "(root) CMD (/root/bin/vmware-checker)" messages still go through?!

log {
source(src);
source(remote);
filter(myfilter);
filter(killVmMessages);
destination(d_mysql);
};

I'm really stumped here. All other filters (non regex) works fine though,
such as facility() & host().

Are you able to filter by content?

Ralph



Perhaps you could try this which is working for me and let me filter
all messages coming from iptables:


# firewall logging
destination iptables { file("/var/log/firewall/iptables.log"); };
filter f_iptables { message("iptables"); };
log { source(s_all); filter(f_iptables); destination(iptables); };

# all messages coming from kern
destination df_kern { file("/var/log/system/kern.log" ); };
filter f_kern { facility(kern) and not filter(f_iptables); };
log { source(s_all); filter(f_kern);destination(df_kern); };

Fred

Robert Fekete 03-17-2010 10:12 AM

syslog-ng filtering
 
That's right, the value() parameter specifies which part of the message to
check. This helps to cut down the performance cost of filtering, because there
is no need to process the entire message if you are filtering on the program
name, for example.

Also, check the syslog-ng Administrator Guide
(http://www.balabit.com/support/documentation/?product=syslog-ng&type=all&language[en]=en&)
if you run into problems. And let me know if you do not find something that
should be in the guide so I can add it some time.

Regards,

Robert Fekete
maintainer of the syslog-ng documentation
*

Ralph Slooten 03-17-2010 07:16 PM

syslog-ng filtering
 
Fantastic, you hit the nail right on the head! Works like a charm now.
Now I'm wondering how it is you found out that it was this way and not the other? Robert maintains the documentation for rsync which I did look at, but with 225 pages I wasn't able to find this useful piece of information. Man syslog-ng.conf does not explain it either, in fact I searched Google and found several "tutorials", none mentioning this ;-)

Maybe I'm the idiot here, however I thought that this was a common way of getting rid of unwanted crud from the syslog?*
Also, I just read the gentoo-wiki site page again and it says :


filter f_shorewall { not match("regex" value("Shorewall")); }; # Filter everything except regex keyword ShorewallSurely this is the exact same mistake I made? Either that or I'm reading it wrong....





On 17 March 2010 23:39, Alan McKinnon <alan.mckinnon@gmail.com> wrote:

On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:

> Hi all,

>

> Has anyone here worked out how to filter out syslog messages using

> syslog-ng v3? The old syntax doesn't work (well complains bitterly about

> performance and says to use regex), and no matter what I try I cannot get

> the new syntax to work :-/ I have a syslog-ng server which logs to MySQL

> for multiple clients in a network, however the database just keeps growing

> with irrelevant data I'd prefer to just quietly ignore on the server side.

>

> I'm trying to filter out (exclude) messages such as:

> * (root) CMD (/root/bin/vmware-checker)

> and

> * (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

>

> ==============

> filter myfilter {

> * * * * not match("regex" value("/usr/sbin/run-crons"))

> * * * * and not match("regex" value("vmware-checker"));

> }



Hah! this caught me out too.



The value of "value" cannot be anything arbitrary - syslog-ng has no clue what

you mean. The value is a field name, either a pre-defined one, or something

you defined using a parser. The docs are ambiguous on this, it's not clear

that the supplied values are abstracts. You are truing to search for the

string "regex" in a field called /usr/bin/vmware-checker.



Which obviously will not work.



I think you want:



match("/usr/sbin/run-crons" value "MESSAGE")



Note that it is MESSAGE. You want the field name, not it's dereferenced value.







> log {

> * * * * source(src);

> * * * * source(remote);

> * * * * filter(myfilter);

> * * * * destination(d_mysql);

> };

> ===============

>

> However they just keep coming through the filter (ie: not matching the "not

> match" filter). I've tried escaping the slashes, not escaping them ... even

> partial words, but I obviously am missing something somewhere.

>

> Anyone have any ideas?

>

> Thanks in advance,

> Ralph



--

alan dot mckinnon at gmail dot com

Alan McKinnon 03-17-2010 07:31 PM

syslog-ng filtering
 
On Wednesday 17 March 2010 22:16:20 Ralph Slooten wrote:
> Fantastic, you hit the nail right on the head! Works like a charm now.
>
> Now I'm wondering how it is you found out that it was this way and not the
> other? Robert maintains the documentation for rsync which I did look at,
> but with 225 pages I wasn't able to find this useful piece of information.
> Man syslog-ng.conf does not explain it either, in fact I searched Google
> and found several "tutorials", none mentioning this ;-)

I read documentation, man pages and google all day every day, some things just
get intuitive :-)

Seriously though, there are a few hints. Syslog-ng's config file format was
written by programmers for programmers to be understood by programmers. That
may not have been the stated intent, but it is how things turned out. The
syntax is exactly that of C, all the way down to braces and statement
terminators. So, when reading the docs, I flicked the switch that puts my
brain in C-mode.

Also, there's an example in the admin guide pdf chapter 3 "Configuring syslog-
ng", something like:

match("string" value(MESSAGE);

It says that MESSAGE is exactly that and must not be dereferenced with "$"

That was a dead give-away

>
> Maybe I'm the idiot here, however I thought that this was a common way of
> getting rid of unwanted crud from the syslog?

It IS the ideal way to pre-filter logs based on the message content. Pre
version 3, you could only match on the entire message, so the feature to be
able to search just a user-defined chunk of the log entry is a major plus

> Also, I just read the gentoo-wiki site page again and it says :
>
> filter f_shorewall { not match("regex" value("Shorewall")); }; #
> Filter everything except regex keyword Shorewall
>
> Surely this is the exact same mistake I made? Either that or I'm reading it
> wrong....

No, you are not reading it wrong - the gentoo guide is wrong. It's a common
mistake, as the syntax looks like it's a name-value pair. To my mind, the
label "value" should instead be "field" or some synonym of that.

All the evidence indicates to me that the syntax makes sense once you "get"
how it works, but most folks' initial assumption about it is wrong, and the
developer never spotted his serious case of being blinded by his own
understanding.

I see Robert responded here earlier. Perhaps he'll see this post and re-look
at that section in a new light with a view to making a patch



>
> On 17 March 2010 23:39, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> > On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
> > > Hi all,
> > >
> > > Has anyone here worked out how to filter out syslog messages using
> > > syslog-ng v3? The old syntax doesn't work (well complains bitterly
> > > about performance and says to use regex), and no matter what I try I
> > > cannot get the new syntax to work :-/ I have a syslog-ng server which
> > > logs to MySQL for multiple clients in a network, however the database
> > > just keeps
> >
> > growing
> >
> > > with irrelevant data I'd prefer to just quietly ignore on the server
> >
> > side.
> >
> > > I'm trying to filter out (exclude) messages such as:
> > > (root) CMD (/root/bin/vmware-checker)
> > >
> > > and
> > >
> > > (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
> > >
> > > ==============
> > > filter myfilter {
> > >
> > > not match("regex" value("/usr/sbin/run-crons"))
> > > and not match("regex" value("vmware-checker"));
> > >
> > > }
> >
> > Hah! this caught me out too.
> >
> > The value of "value" cannot be anything arbitrary - syslog-ng has no clue
> > what
> > you mean. The value is a field name, either a pre-defined one, or
> > something you defined using a parser. The docs are ambiguous on this,
> > it's not clear that the supplied values are abstracts. You are truing to
> > search for the string "regex" in a field called /usr/bin/vmware-checker.
> >
> > Which obviously will not work.
> >
> > I think you want:
> >
> > match("/usr/sbin/run-crons" value "MESSAGE")
> >
> > Note that it is MESSAGE. You want the field name, not it's dereferenced
> > value.
> >
> > > log {
> > >
> > > source(src);
> > > source(remote);
> > > filter(myfilter);
> > > destination(d_mysql);
> > >
> > > };
> > > ===============
> > >
> > > However they just keep coming through the filter (ie: not matching the
> >
> > "not
> >
> > > match" filter). I've tried escaping the slashes, not escaping them ...
> >
> > even
> >
> > > partial words, but I obviously am missing something somewhere.
> > >
> > > Anyone have any ideas?
> > >
> > > Thanks in advance,
> > > Ralph
> >
> > --
> > alan dot mckinnon at gmail dot com

--
alan dot mckinnon at gmail dot com

Keith Dart 03-17-2010 07:40 PM

syslog-ng filtering
 
=== On Thu, 03/18, Ralph Slooten wrote: ===
> Maybe I'm the idiot here, however I thought that this was a common
> way of getting rid of unwanted crud from the syslog?

===

Probably the best method is to not send it there in the first place.

For example, the script run by cron, /usr/sbin/run-crons, has this line
in it:

[ -x /usr/bin/logger ] && /usr/bin/logger -i -p cron.info -t run-crons
"(`whoami`) CMD ($SCRIPT)"

You can comment that out and then those annoying run-cron entries won't
be logged.



-- Keith Dart

--
-- --------------------
Keith Dart
<keith@dartworks.biz>
=======================

Ralph Slooten 03-17-2010 08:43 PM

syslog-ng filtering
 
On 18 March 2010 09:40, Keith Dart <keith@dartworks.biz> wrote:

You can comment that out and then those annoying run-cron entries won't

be logged.*Yes,*dropping*those entries on the client side is an option, however then I have to do it for each client in the network. Doing it on the server means just once... and it's all local network, no bandwidth isn't an issue either.

There are also some cron jobs I do want logged ~ things that run maybe weekly or monthly, but some run every minute and really don't need to be logged.


All times are GMT. The time now is 12:14 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.