FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 10-10-2012, 07:10 PM
Tomas Mraz
 
Default replacing rsyslogd in minimal with journald

On Wed, 2012-10-10 at 15:01 -0400, Matthew Miller wrote:
> On Wed, Oct 10, 2012 at 02:44:53PM -0400, Konstantin Ryabitsev wrote:

> > case, then even if I require rsyslog for a package, that won't work
> > unless rsyslog is started and running. So, sysadmin's experience
> > changes:
> > Was: Install logwatch.
> > Becomes: Install logwatch. Make sure you install and enable rsyslog.
> > I just want to make sure people are aware of the change.
>
> Well, we've got: http://fedoraproject.org/wiki/Features/PackagePresets and
> it seems like we could probably come up with a preset selection for
> non-desktop system use. I'd say "server-presets", except it goes beyond
> server, of course. But yeah, we'd need to make that easy -- a list of "now
Then call it unix-presets perhaps?

> you get to jump through these hoops because we've made things better!" won't
> make anyone happy with us.

We are just dropping another part of the UNIX API - this time the system
logs. Who cares? (I do.)

--
Tomas Mraz
No matter how far down the wrong road you've gone, turn back.
Turkish proverb

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 07:20 PM
Kay Sievers
 
Default replacing rsyslogd in minimal with journald

On Wed, Oct 10, 2012 at 9:01 PM, Matthew Miller
<mattdm@fedoraproject.org> wrote:
> Additionally, it _would_ be cool for log monitoring and analysis tools to
> gain journald support, so that users of those tools can take advantage of
> all the features Lennart lists. If we could have some of those in place
> along with the proposed feature, that would be a win.

Along with the ability to retrieve data from the journal, tools should
probably start at the same time to support real message ids. They will
allow us reliable recognition without weird regex matches in human
readable syslog lines, allow catalogization of messages,
documentation, metadata handling, or even localization.

What we have in systemd so far is:
http://cgit.freedesktop.org/systemd/systemd/tree/src/systemd/sd-messages.h

We also have proper identifiers for devices/hardware in the kernel
logs now. The journal reads them already and connects them to the
current udev supplied data. These identifiers should also be used to
identify a device instead of the unreliable guessing of strings in
human readable syslog messages:
http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html

Kay
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 07:31 PM
Konstantin Ryabitsev
 
Default replacing rsyslogd in minimal with journald

On Tue, Oct 9, 2012 at 5:24 PM, Lennart Poettering <mzerqung@0pointer.de> wrote:
> I am not generally against adding time-based rotation, but really, this
> is much less of a "necessity" than other things the journal provides,
> which syslog does not: for example per-service rate limits, and
> unfakable meta-data for log messages. I mean, really, how can we ship
> a syslog where every random user can fake messages, say they are from a
> privileged process and offer no way how to detect that?

I think you overestimate how much a sysadmin cares about fake
messages. The thing that's really important to a sysadmin is to make
sure that none of the REAL messages are lost. If someone fakes root
login entries by using something as trivial as "logger", I can easily
establish they are fake by looking at auditd logs. And then I would
*really* make that user regret their actions by using blunt
cryptanalysis tools.

So, it's not accurate to say that we don't currently have ways to detect that.

Regards,
--
Konstantin Ryabitsev
LinuxFoundation.org
Montréal, Québec
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 07:44 PM
Kay Sievers
 
Default replacing rsyslogd in minimal with journald

On Wed, Oct 10, 2012 at 9:31 PM, Konstantin Ryabitsev
<icon@fedoraproject.org> wrote:
> On Tue, Oct 9, 2012 at 5:24 PM, Lennart Poettering <mzerqung@0pointer.de> wrote:
>> I am not generally against adding time-based rotation, but really, this
>> is much less of a "necessity" than other things the journal provides,
>> which syslog does not: for example per-service rate limits, and
>> unfakable meta-data for log messages. I mean, really, how can we ship
>> a syslog where every random user can fake messages, say they are from a
>> privileged process and offer no way how to detect that?
>
> I think you overestimate how much a sysadmin cares about fake
> messages. The thing that's really important to a sysadmin is to make
> sure that none of the REAL messages are lost. If someone fakes root
> login entries by using something as trivial as "logger", I can easily
> establish they are fake by looking at auditd logs. And then I would
> *really* make that user regret their actions by using blunt
> cryptanalysis tools.
>
> So, it's not accurate to say that we don't currently have ways to detect that.

That works only for very very few of the logged messages, and it is a
good example how things should really not be designed or work today.

We need one source of system log and not a bunch of daemons with all
overlap but still have only parts of the picture, store their own
stuff all over the place.

Manual matching between the different data sources can sometimes be
used to find out what was really going on, but that's really not good
enough today.

The journal daemon uses similar close-to-the-kernel properties to
establish trust in logged messages, and in the future it is planned
that it will also rad all audit messages directly. The audit daemon
will then mostly be a policy execution engine for (rather exotic)
requirements like "crash the box if the message does not go to disk".

Kay
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 07:49 PM
Simo Sorce
 
Default replacing rsyslogd in minimal with journald

On Wed, 2012-10-10 at 21:44 +0200, Kay Sievers wrote:
> On Wed, Oct 10, 2012 at 9:31 PM, Konstantin Ryabitsev
> <icon@fedoraproject.org> wrote:
> > On Tue, Oct 9, 2012 at 5:24 PM, Lennart Poettering <mzerqung@0pointer.de> wrote:
> >> I am not generally against adding time-based rotation, but really, this
> >> is much less of a "necessity" than other things the journal provides,
> >> which syslog does not: for example per-service rate limits, and
> >> unfakable meta-data for log messages. I mean, really, how can we ship
> >> a syslog where every random user can fake messages, say they are from a
> >> privileged process and offer no way how to detect that?
> >
> > I think you overestimate how much a sysadmin cares about fake
> > messages. The thing that's really important to a sysadmin is to make
> > sure that none of the REAL messages are lost. If someone fakes root
> > login entries by using something as trivial as "logger", I can easily
> > establish they are fake by looking at auditd logs. And then I would
> > *really* make that user regret their actions by using blunt
> > cryptanalysis tools.
> >
> > So, it's not accurate to say that we don't currently have ways to detect that.
>
> That works only for very very few of the logged messages, and it is a
> good example how things should really not be designed or work today.
>
> We need one source of system log and not a bunch of daemons with all
> overlap but still have only parts of the picture, store their own
> stuff all over the place.
>
> Manual matching between the different data sources can sometimes be
> used to find out what was really going on, but that's really not good
> enough today.
>
> The journal daemon uses similar close-to-the-kernel properties to
> establish trust in logged messages, and in the future it is planned
> that it will also rad all audit messages directly. The audit daemon
> will then mostly be a policy execution engine for (rather exotic)
> requirements like "crash the box if the message does not go to disk".

It seem your intention is to make the journal so much better that it
will be the preferred choice (and indeed the default).

So make it really better and support time-based rotation. You don't need
to make time-based rotation the default, but you'll make a lot of people
happy to have the option.

Simo.

--
Simo Sorce * Red Hat, Inc * New York

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 08:02 PM
Kay Sievers
 
Default replacing rsyslogd in minimal with journald

On Wed, Oct 10, 2012 at 9:49 PM, Simo Sorce <simo@redhat.com> wrote:
> On Wed, 2012-10-10 at 21:44 +0200, Kay Sievers wrote:
>> On Wed, Oct 10, 2012 at 9:31 PM, Konstantin Ryabitsev
>> <icon@fedoraproject.org> wrote:
>> > On Tue, Oct 9, 2012 at 5:24 PM, Lennart Poettering <mzerqung@0pointer.de> wrote:
>> >> I am not generally against adding time-based rotation, but really, this
>> >> is much less of a "necessity" than other things the journal provides,
>> >> which syslog does not: for example per-service rate limits, and
>> >> unfakable meta-data for log messages. I mean, really, how can we ship
>> >> a syslog where every random user can fake messages, say they are from a
>> >> privileged process and offer no way how to detect that?
>> >
>> > I think you overestimate how much a sysadmin cares about fake
>> > messages. The thing that's really important to a sysadmin is to make
>> > sure that none of the REAL messages are lost. If someone fakes root
>> > login entries by using something as trivial as "logger", I can easily
>> > establish they are fake by looking at auditd logs. And then I would
>> > *really* make that user regret their actions by using blunt
>> > cryptanalysis tools.
>> >
>> > So, it's not accurate to say that we don't currently have ways to detect that.
>>
>> That works only for very very few of the logged messages, and it is a
>> good example how things should really not be designed or work today.
>>
>> We need one source of system log and not a bunch of daemons with all
>> overlap but still have only parts of the picture, store their own
>> stuff all over the place.
>>
>> Manual matching between the different data sources can sometimes be
>> used to find out what was really going on, but that's really not good
>> enough today.
>>
>> The journal daemon uses similar close-to-the-kernel properties to
>> establish trust in logged messages, and in the future it is planned
>> that it will also rad all audit messages directly. The audit daemon
>> will then mostly be a policy execution engine for (rather exotic)
>> requirements like "crash the box if the message does not go to disk".
>
> It seem your intention is to make the journal so much better that it
> will be the preferred choice (and indeed the default).

The journal is nothing really to choose, it's a mandatory core part of
the operating system, systemd needs it itself, and it always runs.

A running syslog daemon always gets its data forwarded only from the
journal daemon. Syslog is by fact today already an "add-on", and not a
required component, it is just installed by default today. I don't use
or run syslog on any of my boxes since quite a while.

> So make it really better and support time-based rotation. You don't need
> to make time-based rotation the default, but you'll make a lot of people
> happy to have the option.

I really don't mind someone implementing a "maximum retention policy"
for the journal, surely sounds useful for some setups, but I'm
personally not really interested in implementing it.

Kay
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 08:07 PM
Konstantin Ryabitsev
 
Default replacing rsyslogd in minimal with journald

On Wed, Oct 10, 2012 at 3:44 PM, Kay Sievers <kay@vrfy.org> wrote:
>> I think you overestimate how much a sysadmin cares about fake
>> messages. The thing that's really important to a sysadmin is to make
>> sure that none of the REAL messages are lost. If someone fakes root
>> login entries by using something as trivial as "logger", I can easily
>> establish they are fake by looking at auditd logs. And then I would
>> *really* make that user regret their actions by using blunt
>> cryptanalysis tools.
>>
>> So, it's not accurate to say that we don't currently have ways to detect that.
>
> That works only for very very few of the logged messages, and it is a
> good example how things should really not be designed or work today.

Yeah, I wasn't saying it's a stellar system, but it is well-understood
by sysadmins -- syslog messages are "discretionary logging" vs. auditd
messages, which are "compulsory syscall logging." I monitor the
former, since it's my first-line alert system for something strange
going on, but I certainly don't rely solely on syslog for forensics.

> We need one source of system log and not a bunch of daemons with all
> overlap but still have only parts of the picture, store their own
> stuff all over the place.

Well, the counter-argument is that we also don't want to put all our
proverbial eggs in one basket. I was kinda fond of not mixing
discretionary free-for-all "I-think-I-just-burped" random junk that
ends up in syslog from hard auditd data. My favourite was always
seeing syslog entries in other languages if workstation user happened
to select something other than "English" for their desktop.

> Manual matching between the different data sources can sometimes be
> used to find out what was really going on, but that's really not good
> enough today.

It is nearly always inevitable, especially in large heterogeneous
environments. I've done quite a few forensic analyses in the past and
you always have to correlate logs from multiple sources. You'll have
Apache log files, PHP error log files, database log files, FTP log
files, etc. I'm not even sure I want to put it all into journal -- and
a lot of it can't go into journal for various reasons. Apache can
either log to syslog or to a file, unless you do some horrible magic
with piping it to tee and logger.

Not saying that the situation won't be improved with journal, but it
will have less of an impact on "real" people for whom log analysis and
correlation is bread-and-butter.

> The journal daemon uses similar close-to-the-kernel properties to
> establish trust in logged messages, and in the future it is planned
> that it will also rad all audit messages directly. The audit daemon
> will then mostly be a policy execution engine for (rather exotic)
> requirements like "crash the box if the message does not go to disk".

I'm not sure anyone actually cares to join the two, honestly. Ausearch
and aureport are well understood and cherished by (admittedly few)
people that know what they do.

Best,
--
Konstantin Ryabitsev
LinuxFoundation.org
Montréal, Québec
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 08:19 PM
Tomasz Torcz
 
Default replacing rsyslogd in minimal with journald

On Wed, Oct 10, 2012 at 03:49:11PM -0400, Simo Sorce wrote:
> So make it really better and support time-based rotation. You don't need
> to make time-based rotation the default, but you'll make a lot of people
> happy to have the option.

Journald will rotate logs when signalled with SIGUSR2. So you need something
like “systemctl kill --signal=USR2 systemd-journald.service” executed by cron
or from .timer unit.
BTW, .timer units will grow calendar scheduling in future, so cron will
go after rsyslog, too.


(Johann, I've stolen your idea
--
Tomasz Torcz Morality must always be based on practicality.
xmpp: zdzichubg@chrome.pl -- Baron Vladimir Harkonnen

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 08:39 PM
Chris Murphy
 
Default replacing rsyslogd in minimal with journald

On Oct 10, 2012, at 2:02 PM, Kay Sievers wrote:

> Syslog is by fact today already an "add-on", and not a
> required component, it is just installed by default today. I don't use
> or run syslog on any of my boxes since quite a while.

How is rsyslog properly disabled?

sockets.target syslog.target rsyslog.service all seem related.


Chris Murphy

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-10-2012, 08:54 PM
Lennart Poettering
 
Default replacing rsyslogd in minimal with journald

On Wed, 10.10.12 22:19, Tomasz Torcz (tomek@pipebreaker.pl) wrote:

> On Wed, Oct 10, 2012 at 03:49:11PM -0400, Simo Sorce wrote:
> > So make it really better and support time-based rotation. You don't need
> > to make time-based rotation the default, but you'll make a lot of people
> > happy to have the option.
>
> Journald will rotate logs when signalled with SIGUSR2. So you need something
> like “systemctl kill --signal=USR2 systemd-journald.service” executed by cron
> or from .timer unit.

Note that this will not really implement something that would be useful
for data retention policy enforcement. Sending USR2 will cause journald
to rotate the files, but not delete more than necessary to fulfill the
disk space limits. To enforce data retention policy enforcement we need
to bump this logic up to delete all journal files which contain entries
older than a specific time.

Implementing this is actually not hard... happy to take patches.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 08:34 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org