Posts Tagged tips

IPSEC VPN connection between Racoon and Checkpoint

When connecting a Checkpoint firewall with a Linux or BSD server with Racoon software, the following error is quite typical in the very beginning of  Phase 1 negotiation:

2011-04-14 15:47:21: DEBUG: 40 bytes message received from 62.x.x.x[500] to 2
2011-04-14 15:47:21: DEBUG:
30652081 6d92a9ee 00000000 00000000 0b100500 4bd389ff 00000028 0000000c
00000000 0100000e
2011-04-14 15:47:21: DEBUG: malformed cookie received or the initiator's cookies collide.

The ting is, racoon uses AES key length of 128 bit by default, and Checkpoint firewalls use AES-256 (for Phase 1 only 256-bit keys are supported).

The following configuration should fix the problem. Also “sainfo” line shows how to set AES-256 for  Phase 2:

remote 62.x.x.x {
        exchange_mode main;
        proposal {
                encryption_algorithm aes 256;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group 2;
                lifetime time 1440 minutes;
        generate_policy off;
        nat_traversal force;

sainfo address 213.x.x.x[any] any address 62.y.y.y/24[any] any {
        encryption_algorithm aes 256;
        authentication_algorithm hmac_sha1;
        compression_algorithm deflate;

, , ,