System: arp: [ip] is on [if 0] but got reply from [mac] on [if 1]
(last edit: 2001-07-01)
                   
Introduction
You have a machine with two NIC's both hooked up to the same switch but both have a 
different subnet assigned. Everything works fine but you get allot of these messages:

  arp: 192.168.1.69 is on rl0 but got reply from a8:0d:62:1c:42:03 on rl1

A switch works on a low level: it only knows MAC addresses. When it receives a broadcast 
message it will send that message to all of its ports (like it is supposed to do). Your 
machine maneges an arp table (try 'arp -a'). This table makes sure that your machine 
knows on which MAC address which ipaddress is so it can do an Internet-to-Ethernet 
translation. When the broadcast reaches your machine it will be received on both your 
NIC's. Arp notices that it gets a package from a MAC address of which it thought was on 
interface 0 on interface 1 and reports this problem. I think you might agree with me that
this is a good thing. But everything works fine and there are situations where such a 
setup is desired and this problem generates allot of messages. These messages are going 
to hide some of the important other messages because of the amount of them so you realy 
want to get rid of them.

Solution
There are four solutions to this problem: ignore the error, strip the error message from 
the source, rewire or set the kernel state. The first option realy isn't a good one for 
the above reason. I've read somewhere that this error reporting is just 4 lines of code 
in some c file (forgot which one) and you can remove those lines without problems from 
the source. I'm not so fond of this sollution. If you connect one NIC with subnet A to a 
switch with only subnet A attached to it and the other NIC with subnet B to a switch with 
only subnet B attached to it your problem will be solved. So now I have mentioned all the
solutions of which you imediatly had to sigh and here is IMHO the propper one:

There is a kernel state for this:

  root@host:/#sysctl -a | grep -i arp
  net.link.ether.inet.log_arp_wrong_iface: 1

You can change this one for as long as your machine runs (when you reboot your change 
will be lost) with the following command:

  root@host:/#sysctl -w net.link.ether.inet.log_arp_wrong_iface=0

Just test your machine for a while and if this solves the problem then you can set this 
options permanently: Create the file '/etc/sysctl.conf' if it doesn't exists (it isn't 
there by default) and put the following line in it:

  net.link.ether.inet.log_arp_wrong_iface=0

Reboot to check if it works (not necesary but a good test)