IPSEC VPN connection between Racoon and Checkpoint

When connecting a Checkpoint firewall with a Linux or BSD server with Racoon software, the following error is quite typical in the very beginning of  Phase 1 negotiation:

2011-04-14 15:47:21: DEBUG: 40 bytes message received from 62.x.x.x[500] to 2
2011-04-14 15:47:21: DEBUG:
30652081 6d92a9ee 00000000 00000000 0b100500 4bd389ff 00000028 0000000c
00000000 0100000e
2011-04-14 15:47:21: DEBUG: malformed cookie received or the initiator's cookies collide.

The ting is, racoon uses AES key length of 128 bit by default, and Checkpoint firewalls use AES-256 (for Phase 1 only 256-bit keys are supported).

The following configuration should fix the problem. Also “sainfo” line shows how to set AES-256 for  Phase 2:

remote 62.x.x.x {
        exchange_mode main;
        proposal {
                encryption_algorithm aes 256;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group 2;
                lifetime time 1440 minutes;
        generate_policy off;
        nat_traversal force;

sainfo address 213.x.x.x[any] any address 62.y.y.y/24[any] any {
        encryption_algorithm aes 256;
        authentication_algorithm hmac_sha1;
        compression_algorithm deflate;

, , ,

  1. #1 by Morpheus on February 3, 2012 - 9:32 am

    Tnx man, this worked!

  2. #2 by txlab on February 3, 2012 - 9:33 am

    yeah, it took me awhile to figure out 🙂

  3. #3 by prajith p on June 7, 2014 - 10:06 am

    how can we use username and password instead of pre_shared_key? is it really possible?

    • #4 by txlab on June 7, 2014 - 10:22 am

      I don’t know. I simply documented my experience

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: