FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 10-09-2012, 09:24 PM
Lennart Poettering
 
Default replacing rsyslogd in minimal with journald

On Tue, 09.10.12 16:53, Simo Sorce (simo@redhat.com) wrote:

> > > Also: what is the equivalent for logrotate in the systemd journal
> > > case?
> >
> > Rotation happens in-line, i.e. each time before we are about to write an
> > entry we check if rotation is necessary and execute it. This should make
> > things a lot more robust, as this fixes a common issue with syslog where
> > a lot of data generated in bursts could flood the fs until a much later
> > time-based rotation took place. This time window goes away with the journal.
>
> Lovely so now I have a perfect way to remove all traces of access all we
> need is to cause a lot of logging to go through until it is merrily
> deleted ?

We enforce per-cgroup ratelimiting. That means a service can run amok,
but this will cause its stream to be throttled while leaving other
services unaffacted.

> This *must* be configurable, there are places where rotation is not
> allowed at all and the system *must* crash and stop if logs can't be
> written and preserved.

Well, I am not sure what you are requesting. We cannot store more on a
disk than fits on a disk. So we need to do things based on disk
size/free space. Almost everybody would rather have log messages being
dropped than the machine come to a standstill because log messages can't
be written anymore.

If you want audit-like semantics with crashing if we cannot write, then
use something else, not the journal. The journal is supposed to be
robust and do the right thing so that you can leave it unnatteneded and
whatever happens it didn't spill the disk or become unavailable. It's
supposed to be "zero maintainance".

> > See SystemMaxUse= resp. SystemKeepFree= in journald.conf(5).
>
> You need to allow boundless configurations, see above.

No, I don't. The journal is not a reimplementation of auditd. If people
want a "yes, please, crash my machine if a client enters a log spam
loop" policy, then we have other options than the journal, and which can
run side-by-side with the journal. I dont think this is in anyway
relevant for 99.9% of all installations. In those it is more interesting
to minimize the negative impact a misbehaving service can have, rather
than amplifying it. So no, I don't need, I don't must support all
thinkable setups.

I am not generally against adding time-based rotation, but really, this
is much less of a "necessity" than other things the journal provides,
which syslog does not: for example per-service rate limits, and
unfakable meta-data for log messages. I mean, really, how can we ship
a syslog where every random user can fake messages, say they are from a
privileged process and offer no way how to detect that?

> Also rotating based on use is generally annoying to admins, as it makes
> more difficult to predict where stuff will end up and what will
> deterministically be in backups.

For some sure, for most not.

> logrotate has time based policies for very good reasons.

Yeah, because Unix doesn't really allow much else...

> Your policy may make sense on space-constrained configuration but in any
> other system they make little sense, and log compression on rotation is
> all you really need (lots of repetitions in the logs allow big gains
> when compressing).

Well, I guess we have to agree to disagree on what is necessary to make
things robust, safe and secure.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 09:26 PM
Lennart Poettering
 
Default replacing rsyslogd in minimal with journald

On Tue, 09.10.12 14:57, Chris Murphy (lists@colorremedies.com) wrote:

>
> On Oct 9, 2012, at 11:20 AM, Lennart Poettering wrote:
>
> > On Tue, 09.10.12 15:35, Bryn M. Reeves (bmr@redhat.com) wrote:
> >> Setting PIPE or piping to a pager is even worse - the lines are
> >> truncated at 77 chars regardless of the term width so for now I'm
> >> running journalctl --no-pager -a | less to get wrapped lines in a
> >> pager.
> >
> > Fixed in F18.
>
>
> Not for me. I get one result for the first command, and 20+ for the second:
>
> journalctl | grep btrfs
> cat /var/log/messages | grep btrfs
>
> systemd-194-1.fc18.x86_64

Well, you have to actually enable the journal on persistent storage
first. Try "mkdir /var/log/journal". Without that we will only keep a
very small set of logs in RAM, so that things are flushed out quickly.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 09:40 PM
Matthew Miller
 
Default replacing rsyslogd in minimal with journald

On Tue, Oct 09, 2012 at 11:26:19PM +0200, Lennart Poettering wrote:
> Well, you have to actually enable the journal on persistent storage
> first. Try "mkdir /var/log/journal". Without that we will only keep a
> very small set of logs in RAM, so that things are flushed out quickly.

So, minutes before seeing this message, on my F17 test system, I did:

$ mkdir /var/log/journal
$ sudo systemctl restart systemd-journald.service

And now I get

Logs begin at Sat, 29 Sep 2012 09:23:21 -0400, end at Tue, 09 Oct 2012 17:30:59 -0400.
Failed to iterate through journal: Bad message
Sep 29 09:23:21 localhost.localdomain systemd-journal[313]: Journal started

Is this the expected behavor?

--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm@fedoraproject.org>
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 09:41 PM
Stephen John Smoogen
 
Default replacing rsyslogd in minimal with journald

On 9 October 2012 15:24, Lennart Poettering <mzerqung@0pointer.de> wrote:
> On Tue, 09.10.12 16:53, Simo Sorce (simo@redhat.com) wrote:

> If you want audit-like semantics with crashing if we cannot write, then
> use something else, not the journal. The journal is supposed to be
> robust and do the right thing so that you can leave it unnatteneded and
> whatever happens it didn't spill the disk or become unavailable. It's
> supposed to be "zero maintainance".

So in those cases rsyslog would be required, but would be seen as a
post-install step.

EG what you are looking at is building a GNOME-OS and for those sorts
of tablets, etc the journal is right for that. The other cases like at
a Hospital, trading firm or various .gov.XX then having rsyslog
installed with audit post would be the way to get the needed features.


--
Stephen J Smoogen.
"Don't derail a useful feature for the 99% because you're not in it."
Linus Torvalds
"Years ago my mother used to say to me,... Elwood, you must be oh
so smart or oh so pleasant. Well, for years I was smart. I
recommend pleasant. You may quote me." —James Stewart as Elwood P. Dowd
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 09:50 PM
Matthew Miller
 
Default replacing rsyslogd in minimal with journald

On Tue, Oct 09, 2012 at 03:41:51PM -0600, Stephen John Smoogen wrote:
> > If you want audit-like semantics with crashing if we cannot write, then
> > use something else, not the journal. The journal is supposed to be
> > robust and do the right thing so that you can leave it unnatteneded and
> > whatever happens it didn't spill the disk or become unavailable. It's
> > supposed to be "zero maintainance".
>
> So in those cases rsyslog would be required, but would be seen as a
> post-install step.
>
> EG what you are looking at is building a GNOME-OS and for those sorts
> of tablets, etc the journal is right for that. The other cases like at
> a Hospital, trading firm or various .gov.XX then having rsyslog
> installed with audit post would be the way to get the needed features.

If so, this seems unfortunate, because the other features discussed (e.g.,
trustable metadata) would be very welcome in these environments. Can't the
enterprise have nice things?



--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm@fedoraproject.org>
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 09:57 PM
Stephen John Smoogen
 
Default replacing rsyslogd in minimal with journald

On 9 October 2012 15:50, Matthew Miller <mattdm@fedoraproject.org> wrote:
> On Tue, Oct 09, 2012 at 03:41:51PM -0600, Stephen John Smoogen wrote:
>> > If you want audit-like semantics with crashing if we cannot write, then
>> > use something else, not the journal. The journal is supposed to be
>> > robust and do the right thing so that you can leave it unnatteneded and
>> > whatever happens it didn't spill the disk or become unavailable. It's
>> > supposed to be "zero maintainance".
>>
>> So in those cases rsyslog would be required, but would be seen as a
>> post-install step.
>>
>> EG what you are looking at is building a GNOME-OS and for those sorts
>> of tablets, etc the journal is right for that. The other cases like at
>> a Hospital, trading firm or various .gov.XX then having rsyslog
>> installed with audit post would be the way to get the needed features.
>
> If so, this seems unfortunate, because the other features discussed (e.g.,
> trustable metadata) would be very welcome in these environments. Can't the
> enterprise have nice things?

Sorry I didn't mean to make that either/or. The enterprise gets the
journald but does not get to keep its contents unless there is a
program that sends it to say rsyslog.

--
Stephen J Smoogen.
"Don't derail a useful feature for the 99% because you're not in it."
Linus Torvalds
"Years ago my mother used to say to me,... Elwood, you must be oh
so smart or oh so pleasant. Well, for years I was smart. I
recommend pleasant. You may quote me." —James Stewart as Elwood P. Dowd
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 10:08 PM
Lennart Poettering
 
Default replacing rsyslogd in minimal with journald

On Tue, 09.10.12 15:41, Stephen John Smoogen (smooge@gmail.com) wrote:

> On 9 October 2012 15:24, Lennart Poettering <mzerqung@0pointer.de> wrote:
> > On Tue, 09.10.12 16:53, Simo Sorce (simo@redhat.com) wrote:
>
> > If you want audit-like semantics with crashing if we cannot write, then
> > use something else, not the journal. The journal is supposed to be
> > robust and do the right thing so that you can leave it unnatteneded and
> > whatever happens it didn't spill the disk or become unavailable. It's
> > supposed to be "zero maintainance".
>
> So in those cases rsyslog would be required, but would be seen as a
> post-install step.
>
> EG what you are looking at is building a GNOME-OS and for those sorts
> of tablets, etc the journal is right for that. The other cases like at
> a Hospital, trading firm or various .gov.XX then having rsyslog
> installed with audit post would be the way to get the needed features.

This is BS. The journal is for most folks, not just GNOME users.

How many people actually enable "auditctl -f2"? There's probably not
many except a few three letter agencies and similar folks.

I don't really want to play in the three letter agency area. That
doesn't mean I want to break things for them, I am just saying that the
super-strict policies they want should not dictate how the system works
for everybody else. As long as we make their setups possible (and yeah,
installing rsyslog is not that hard), that's fine. But really, a webshop
couldnt care less for such a mode. For most people reliability is more
important than "auditctl -f2".

Really, I have no intention to provide anything like "auditctl -f2" in
journald. Not going to happen. People can install auditd/rsyslog for
that. It's not my turf, I don't want those bugs.

And anyway: it is really confused to believe that people care more for
"auditcl -f2" than unfakable logs... I am not a security guy, but having
logs where unprivileged users cannot insert undetectable fakes is much
much much much much much more interesting to me thatn "auditctl -f2"
like behaviour. And if that's any standard we never would have allowed
syslog in the distro at all...

To stress this:

With the feature I am planning to propose for F19:

- I just want to change what is installed/enabled by default
- I do not want to break rsyslog or auditd or make them unavailable
- I do plan to support the equivalent of most things syslog offers, but
do not plan to provide *everything* syslog offers. One of these things
is UDP syslog proto support.
- I just want to provide something that is robust and secure and works
for the vast majority of people without reconfiguration
- something that brings a number of important improvements over syslog at
a lower footprint
- That works for server folks, embedded folks, desktop folks alike, but
not necessarily all thinkable usecases of these uses.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 10:17 PM
Stephen John Smoogen
 
Default replacing rsyslogd in minimal with journald

On 9 October 2012 16:08, Lennart Poettering <mzerqung@0pointer.de> wrote:
> On Tue, 09.10.12 15:41, Stephen John Smoogen (smooge@gmail.com) wrote:
>
>> On 9 October 2012 15:24, Lennart Poettering <mzerqung@0pointer.de> wrote:
>> > On Tue, 09.10.12 16:53, Simo Sorce (simo@redhat.com) wrote:
>>
>> > If you want audit-like semantics with crashing if we cannot write, then
>> > use something else, not the journal. The journal is supposed to be
>> > robust and do the right thing so that you can leave it unnatteneded and
>> > whatever happens it didn't spill the disk or become unavailable. It's
>> > supposed to be "zero maintainance".
>>
>> So in those cases rsyslog would be required, but would be seen as a
>> post-install step.
>>
>> EG what you are looking at is building a GNOME-OS and for those sorts
>> of tablets, etc the journal is right for that. The other cases like at
>> a Hospital, trading firm or various .gov.XX then having rsyslog
>> installed with audit post would be the way to get the needed features.
>
> This is BS. The journal is for most folks, not just GNOME users.

Ugh.. look I was trying to restate exactly what you said in previous
emails to make sure I understood what you were saying and to show I
agreed where that is coming from.. and it looks like I dropped some
packets somewhere

By GNOME-OS I meant a particular use-case where a journal would be
useful and it is built for like you said in previous emails. Change
GNOME-OS to Fedora, KDE-OS, Mozilla-OS, whatever.. it is a use case
for a lot of people.

Sites that need specialized big business needs are going to need
something like rsyslog because they have limited case issues.. like
never over-writing logs, halting when logs fill up, etc. Those are
written in regulations that aren't going to change in anytime before
say RHEL-10 comes out.

> How many people actually enable "auditctl -f2"? There's probably not
> many except a few three letter agencies and similar folks.

The hospital servers I helped work with had to have it for HIPAA and
SOX Banes. The money systems that had PCI-DSS also had it on some. But
I am agreeing with you it is a small case.

--
Stephen J Smoogen.
"Don't derail a useful feature for the 99% because you're not in it."
Linus Torvalds
"Years ago my mother used to say to me,... Elwood, you must be oh
so smart or oh so pleasant. Well, for years I was smart. I
recommend pleasant. You may quote me." —James Stewart as Elwood P. Dowd
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 10:20 PM
Stephen John Smoogen
 
Default replacing rsyslogd in minimal with journald

On 9 October 2012 17:18, Dan Williams <dcbw@redhat.com> wrote:
> On Tue, 2012-10-09 at 15:57 -0600, Stephen John Smoogen wrote:
>> On 9 October 2012 15:50, Matthew Miller <mattdm@fedoraproject.org> wrote:
>> > On Tue, Oct 09, 2012 at 03:41:51PM -0600, Stephen John Smoogen wrote:
>> >> > If you want audit-like semantics with crashing if we cannot write, then
>> >> > use something else, not the journal. The journal is supposed to be
>> >> > robust and do the right thing so that you can leave it unnatteneded and
>> >> > whatever happens it didn't spill the disk or become unavailable. It's
>> >> > supposed to be "zero maintainance".
>> >>
>> >> So in those cases rsyslog would be required, but would be seen as a
>> >> post-install step.
>> >>
>> >> EG what you are looking at is building a GNOME-OS and for those sorts
>> >> of tablets, etc the journal is right for that. The other cases like at
>> >> a Hospital, trading firm or various .gov.XX then having rsyslog
>> >> installed with audit post would be the way to get the needed features.
>> >
>> > If so, this seems unfortunate, because the other features discussed (e.g.,
>> > trustable metadata) would be very welcome in these environments. Can't the
>> > enterprise have nice things?
>>
>> Sorry I didn't mean to make that either/or. The enterprise gets the
>> journald but does not get to keep its contents unless there is a
>> program that sends it to say rsyslog.
>
> Ah; I think what you meant to say is:
>
> "*IF* what you are looking at..."

In my head I thought I wrote that *IF* until you pointed out I missed it.

> but I'd suggest instead:
>
> "If you have strict requirements on time-based logging rotation or
> certain audit requirements, then something like rsyslog(?) is required
> in parallel with the journal. In most other cases (desktops, tablets,
> many servers) the journal is sufficient."

*patch acked*


> No?
>
> Dan
>
> --
> devel mailing list
> devel@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel



--
Stephen J Smoogen.
"Don't derail a useful feature for the 99% because you're not in it."
Linus Torvalds
"Years ago my mother used to say to me,... Elwood, you must be oh
so smart or oh so pleasant. Well, for years I was smart. I
recommend pleasant. You may quote me." —James Stewart as Elwood P. Dowd
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 10-09-2012, 10:24 PM
Matthew Miller
 
Default replacing rsyslogd in minimal with journald

On Tue, Oct 09, 2012 at 04:20:14PM -0600, Stephen John Smoogen wrote:
> > "If you have strict requirements on time-based logging rotation or
> > certain audit requirements, then something like rsyslog(?) is required
> > in parallel with the journal. In most other cases (desktops, tablets,
> > many servers) the journal is sufficient."
> *patch acked*

Okay, so, given that: isn't systemd with time-based rotation logging more
desirable than pushing that aspect off to rsyslog, because rsyslog loses the
secure logging aspect?

I would also note that the scope of organizations that have requirements for
time-based rotation are much, much larger than than the set of organizations
who need their servers to crash on error. It's an important use case, not
just a thought experiment.

--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm@fedoraproject.org>
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 09:27 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org