So, I got this setup.

Here is my scenario. We had the first switch (the one with a cable to ISP, here 1.1.1.1) set up with an SVI (1.1.1.2). Default route to ISP. Everything is cool.
Then the company used up the adresses they got. 1.1.1.2 for the SVI, 1.1.1.3 for their ISA that all user traffic passes through (left out of this), 1.1.1.3-6 for VPN's.
So, they get a few more add's. 2.2.2.1/28.
I set it up like this. I removed the SVI at the first switch and instead set it up on the first switch by removing the address from the interface vlan 50, then added it to the other switch on the same interface. It worked just fine, no problems to ping over the trunk, through the first switch, to the 1.1.1.1 (isp). Also the other public ip's worked just fine, they were reachable (few servers set up there).
Then, later (5-30 minutes) it crashes. Does not work. I had already left for the evening.
My thoughts is that the arp-entry in the first switch timed out, and it didnt send a new one because it did not have a SVI.
What is the solution? Creating a SVI on both switches rly is not popular, since it eats up and public IP and it creates some trouble with the VPN's.