lost udp packets
 

I have a CentOS release 5.8 that has snmp traps being sent to it. I've
been trying to forward the snmp traps to another system. I've tried
forwarding with snmpd/snmptrapd, iptables, and some forwarding programs.
I can see snmp traps getting delivered to the system with tcpdump and
wireshark, but no matter what app I run, the traps do not appear to be
reaching the application or port 162. It seems like the packets are
possibly being dropped right away.

iptables is wide open:

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

If I run the apps I can see port 162 open and closed depending on what I
have running, so I'm sure there's not a specific app running already on
that port.

Anyone have any ideas on what could be happening to these packets and
why they might not be reaching port 162 on this host?

Thanks,
James
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

On 10/4/2012 9:40 AM, James Pifer wrote:
> I have a CentOS release 5.8 that has snmp traps being sent to it. I've
> been trying to forward the snmp traps to another system. I've tried
> forwarding with snmpd/snmptrapd, iptables, and some forwarding programs.
> I can see snmp traps getting delivered to the system with tcpdump and
> wireshark, but no matter what app I run, the traps do not appear to be
> reaching the application or port 162. It seems like the packets are
> possibly being dropped right away.
>
> iptables is wide open:
>
> # iptables -L
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> If I run the apps I can see port 162 open and closed depending on what I
> have running, so I'm sure there's not a specific app running already on
> that port.
>
> Anyone have any ideas on what could be happening to these packets and
> why they might not be reaching port 162 on this host?
>


Just a follow up. I ran tcpdump for port 162 for a little while and when
I stopped I see this at the end:

737 packets captured
737 packets received by filter
0 packets dropped by kernel

So I guess the kernel is not dropping them. Still can't explain why
applications are not picking them up.

Any help is appreciated.

Thanks,
James
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

On Thu, Oct 4, 2012 at 12:17 PM, James Pifer <jep@obrien-pifer.com> wrote:
> On 10/4/2012 9:40 AM, James Pifer wrote:
>> I have a CentOS release 5.8 that has snmp traps being sent to it. I've
>> been trying to forward the snmp traps to another system. I've tried
>> forwarding with snmpd/snmptrapd, iptables, and some forwarding programs.
>> I can see snmp traps getting delivered to the system with tcpdump and
>> wireshark, but no matter what app I run, the traps do not appear to be
>> reaching the application or port 162. It seems like the packets are
>> possibly being dropped right away.
>>
>> iptables is wide open:
>>
>> # iptables -L
>> Chain INPUT (policy ACCEPT)
>> target prot opt source destination
>>
>> Chain FORWARD (policy ACCEPT)
>> target prot opt source destination
>>
>> Chain OUTPUT (policy ACCEPT)
>> target prot opt source destination
>>
>> If I run the apps I can see port 162 open and closed depending on what I
>> have running, so I'm sure there's not a specific app running already on
>> that port.
>>
>> Anyone have any ideas on what could be happening to these packets and
>> why they might not be reaching port 162 on this host?
>>
>
>
> Just a follow up. I ran tcpdump for port 162 for a little while and when
> I stopped I see this at the end:
>
> 737 packets captured
> 737 packets received by filter
> 0 packets dropped by kernel
>
> So I guess the kernel is not dropping them. Still can't explain why
> applications are not picking them up.
>
> Any help is appreciated.

I'd try strace'ing the app that is supposed to be receiving them to
see if the socket opens are working and what happens with a packet
arrives on the port.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

> I'd try strace'ing the app that is supposed to be receiving them to
> see if the socket opens are working and what happens with a packet
> arrives on the port.
>


No idea what this means. snmptrapd keeps running (strace snmptrapd -f
-Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the
initial start:

gettimeofday({1349372532, 120897}, NULL) = 0
gettimeofday({1349372532, 120917}, NULL) = 0
gettimeofday({1349372532, 120934}, NULL) = 0
gettimeofday({1349372532, 120950}, NULL) = 0
select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
gettimeofday({1349372537, 120615}, NULL) = 0
gettimeofday({1349372537, 120637}, NULL) = 0
gettimeofday({1349372537, 120655}, NULL) = 0
gettimeofday({1349372537, 120670}, NULL) = 0
gettimeofday({1349372537, 120686}, NULL) = 0
gettimeofday({1349372537, 120703}, NULL) = 0
gettimeofday({1349372537, 120721}, NULL) = 0
gettimeofday({1349372537, 120737}, NULL) = 0
select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
gettimeofday({1349372542, 119701}, NULL) = 0
gettimeofday({1349372542, 119726}, NULL) = 0
gettimeofday({1349372542, 119744}, NULL) = 0
gettimeofday({1349372542, 119760}, NULL) = 0
gettimeofday({1349372542, 119776}, NULL) = 0
gettimeofday({1349372542, 119794}, NULL) = 0
gettimeofday({1349372542, 119813}, NULL) = 0
gettimeofday({1349372542, 119829}, NULL) = 0
select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
gettimeofday({1349372547, 118753}, NULL) = 0
gettimeofday({1349372547, 118777}, NULL) = 0
gettimeofday({1349372547, 118794}, NULL) = 0
gettimeofday({1349372547, 118811}, NULL) = 0
gettimeofday({1349372547, 118827}, NULL) = 0
gettimeofday({1349372547, 118844}, NULL) = 0
gettimeofday({1349372547, 118862}, NULL) = 0
gettimeofday({1349372547, 118878}, NULL) = 0
select(9, [3 5 7 8], [], [], {0, 1760}) = 0 (Timeout)
gettimeofday({1349372547, 120727}, NULL) = 0
gettimeofday({1349372547, 120745}, NULL) = 0
gettimeofday({1349372547, 120761}, NULL) = 0
gettimeofday({1349372547, 120777}, NULL) = 0
gettimeofday({1349372547, 120793}, NULL) = 0
gettimeofday({1349372547, 120809}, NULL) = 0
send(7, "1

307203225!", 20, 0) = 20
gettimeofday({1349372547, 120884}, NULL) = 0
gettimeofday({1349372547, 120908}, NULL) = 0
gettimeofday({1349372547, 120929}, NULL) = 0
select(9, [3 5 7 8], NULL, NULL, {0, 1}) = 1 (in [7], left {0, 1})
getsockname(7, {sa_family=AF_FILE, path=@""}, [2]) = 0
recv(7, "122
307203225!10K9",
65536, 0) = 28

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

James Pifer wrote:
>> I'd try strace'ing the app that is supposed to be receiving them to
>> see if the socket opens are working and what happens with a packet
>> arrives on the port.
>>
>
>
> No idea what this means. snmptrapd keeps running (strace snmptrapd -f
> -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the
> initial start:
>
> gettimeofday({1349372532, 120897}, NULL) = 0
> gettimeofday({1349372532, 120917}, NULL) = 0
> gettimeofday({1349372532, 120934}, NULL) = 0
> gettimeofday({1349372532, 120950}, NULL) = 0
> select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
<snip>
Do you have ntp running on all the servers?

mark

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

On 10/4/2012 1:56 PM, m.roth@5-cent.us wrote:
> James Pifer wrote:
>>> I'd try strace'ing the app that is supposed to be receiving them to
>>> see if the socket opens are working and what happens with a packet
>>> arrives on the port.
>>>
>>
>> No idea what this means. snmptrapd keeps running (strace snmptrapd -f
>> -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the
>> initial start:
>>
>> gettimeofday({1349372532, 120897}, NULL) = 0
>> gettimeofday({1349372532, 120917}, NULL) = 0
>> gettimeofday({1349372532, 120934}, NULL) = 0
>> gettimeofday({1349372532, 120950}, NULL) = 0
>> select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
> <snip>
> Do you have ntp running on all the servers?
>
> mark
>
> _________

Not necessarily. SNMP traps are coming from all different kinds of
devices. I can't imagine wrong times would mess up snmptrpd. Are you
thinking that's what it's having a problem with?

Even if I try to just to a udp forward, with socat, iptables, or a
couple specific forwarding apps I tried, nothing seems to get to the apps.

I might just try restarting this server during the night. Maybe
something is just hosed.

Any other ideas?

Thanks,
James
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

On Thu, Oct 4, 2012 at 12:45 PM, James Pifer <jep@obrien-pifer.com> wrote:
>
> No idea what this means. snmptrapd keeps running (strace snmptrapd -f
> -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the
> initial start:
>
> recv(7, "122
307203225!10K9",
> 65536, 0) = 28

Do you know what it was receiving here?

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

On 10/4/2012 2:52 PM, Les Mikesell wrote:
> On Thu, Oct 4, 2012 at 12:45 PM, James Pifer <jep@obrien-pifer.com> wrote:
>> No idea what this means. snmptrapd keeps running (strace snmptrapd -f
>> -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the
>> initial start:
>>
>> recv(7, "122
307203225!10K9",
>> 65536, 0) = 28
> Do you know what it was receiving here?
>

Difficult tell as a lot of snmp traps are received.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

Reboot didn't help, but modifying my snmpd.conf and adding "master
agentx" did the trick. Apparantly snmpd was quietly denying snmptrapd
from connecting. Just happened to come across the suggestion.

Now I need to figure out how to have the traps forwarded but retain the
real source of the trap.

Thanks for the help.

James
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

lost udp packets
 

On 10/05/2012 11:53 AM, James Pifer wrote:
> Now I need to figure out how to have the traps forwarded but retain the
> real source of the trap.

If you want to forward the traps without modifying the source address on
the UDP packet, you'll need to use iptables. Add a DNAT rule to the
PREROUTING chain in the nat table.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos