Redis is a key-value in-memory database (Redis.io). You can use Redis since version 10.0.1 in IBM Verify Access Manager as an alternative to the DSC (Distributed Session Cache) in WebSEAL and also in the Federation component as a replacement for the HVDB in some situations. Although it’s not possible yet to store everything in Redis , I think this will be definitely the case in the future!
There’s a couple of advantages to using Redis instead of the traditional DSC:
- it no longer requires a cluster on ISAM - this is a huge advantage. In my opinion, you should not use ISAM clusters anymore nowadays.
- it scales independently of ISAM
Redis clusters are not supported in ISAM, but Redis Sentinels are. So to achieve high availability, you must turn to a setup using at least 1 Redis replica with the supporting Redis Sentinels. Redis sentinel brings resilience to the solution, but it’s not a scaling solution.
Redis Sentinels work together with a master-replica setup.
Redis Sentinel is a separate service that runs alongside the Redis datastore and that monitors the health of the master. The Sentinels need to agree between themselves to promote a replica to master. To do that, they use a quorum mechanism. Because of this mechanism, you MUST have (at least) 3 sentinel servers, so you can set the required quorum to 2 (at least).
An important side note here is that the Redis client that is used should support Sentinel as well. The logic to switch to a new master, should be provided by the client. I don’t know what client ISAM uses, but it’s visible in the tracing what the logic is. See logging and tracing.
So you need (at least) 3 systems
- 1 master (+sentinel)
- 1 slave (+sentinel)
- 1 additional sentinel
In all cases, the
redis package includes both the redis database and sentinel.
Red Hat Enterprise Linux 8
In RHEL 8, Redis 6 is supplied as a Yum module. Don’t install using ‘simple’ yum commands, because you’ll end up with Redis 5.
sudo yum module install "@redis:6/common"
In Fedora, you can simply install Redis using yum or dnf.
sudo yum install redis
All redis servers should have the same configuration, except for the “replicaof” setting. These keys should be modified for a working setup:
bind <external ip address of your server> 127.0.0.1 port 6379 daemonize yes protected-mode yes supervised auto replica-read-only yes
Specifically, the users that you’ve defined for replication, for sentinel and for your clients should be in ALL redis.conf files (both master and replica).
user default off nocommands allkeys >SOMERANDOMPASSWORD user isam on +@all -CONFIG allkeys >ThePassWordForISAM user sentinel-user on +client +subscribe +publish +ping +info +multi +slaveof +replicaof +config +client +exec >SENTINELRANDOMPASSWORD user replica-user on +@all allkeys >REPLICARANDOMPASSWORD
NOTE If you are going to use the same Redis Sentinel setup for your Federation Runtime as well, you need to use a slightly different configuration for the default user:
user default on nopass -@all +hello
This has to do with the Lettuce Java Redis client that is used in the Liberty Runtime.
The masteruser/masterauth that are used for replication, should also be in ALL redis.conf files.
masteruser replica-user masterauth REPLICARANDOMPASSWORD
This needs to be in the master’s redis.conf file as well, because when the failover happens the master becomes a replica and won’t be able to replicate if it doesn’t have the credentials.
Configuration for replica
The replica should be configured to connect to the host and port of the master.
replicaof <ip address of the master> 6397
This configuration parameter should not exist on the master, because that will break things.
The ACL (the users we created in the redis.conf) does not magically make your Redis system secure, the passwords are stored in clear text in the configuration files and you can brute force Redis (if you can connect to it) - Redis is very fast and there is no rate limit protection.
A hacker could brute-force test 1000’s of passwords in seconds… So passwords should be very long (+20 characters).
That said, it’s still a good idea to disable the default unnamed user, and create separate users for the different tasks. So in this case, we’ll create a user for the ISAM systems to use, a user for Sentinel and a Replica user (for replication).
The isam user needs to have some specific rights as documented here: (https://www.ibm.com/docs/en/sva/10.0.3?topic=limiting-redis-commands)
|isam||user for the isam client|
|sentinel-user||user for sentinels|
|replica-user||user for replicas|
For a production setup, you really should configure TLS. I’ve not done that in this case (yet).
You should have firewall rules in place so only systems that are allowed to connect to Redis, can connect to Redis.
Redis should definitely not be exposed to the internet!
The only configuration that is necessary on the sentinels, is connecting it to the master.
Since we use protected mode, we also need to supply the authentication data for our
#MUST use bind <external_ip> 127.0.0.1 bind <external ip address of your server> 127.0.0.1 port 26379 sentinel monitor mymaster <master ip address> 6379 2 sentinel auth-user mymaster sentinel-user sentinel auth-pass mymaster SENTINELRANDOMPASSWORD # Default is 30 seconds. sentinel down-after-milliseconds mymaster 5000 # Default is 3 minutes - 180000 sentinel failover-timeout mymaster 9000
NOTE : the order of interfaces for
bindis very important: the loopback interface should not be the first one!
You should check the sentinel log, to verify the sentinel can connect to the master, detects the other replica(s) and that it can detect failures.
It is possible to make a ‘cluster’ out of the setup, by using additonal components like haproxy : https://www.willandskill.se/en/setup-a-highly-available-redis-cluster-with-sentinel-and-haproxy.amp.html/.
I don’t think this is necessary in a scenario with IBM Access Manager - a relatively small Redis server should be able to store a very large number of concurrent ISAM/ISVA sessions (100000+) without issue.
Introducing components also makes the setup more complex and simpler is better.
Start Redis and Sentinels
To start everything, you can use the services that are created:
sudo systemctl start redis sudo systemctl start redis-sentinel
You can also enable them to start on boot, if you like.
sudo systemctl enable redis sudo systemctl enable redis-sentinel
Redis for WebSEAL
Redis can be used as a replacement for the DSC in WebSEAL. The configuration uses ISAM-specific terms as “redis collection” and “redis servers” .
The Redis client in WebSEAL uses some Redis commands to find what server is the master (specifically, the
ROLE command) .
The Redis client in WebSEAL does not automatically “discover” the replica(‘s) in a Redis Sentinel setup, so you do have to define the Servers in the Collection, in ISAM.
Configuration in ISAM
The documentation is a bit sparse at the moment, although the actual implementation is pretty simple.
There’s a ‘wizard’ type of configuration, but this is not documented at the moment:
- Create redis collections: go to Web/Redis configuration (https://
/wga/redis_config/collections) Use the 'New' button to add a collection. You can just give it a name , and keep the defaults .
Then, go to the ‘servers’ tab and add your Redis servers (not the Sentinels, only the servers).
This seems more like convenience than anything else , this is not strictly necessary (if you would choose to edit the webseald.conf by hand).
- Edit your instance, go to the Redis tab
This creates the redis collection and server stanzas in the webseald.conf (you could just as well do that by hand)
- Enable distributed sessions in the webseald.conf
[session] dsess-enabled = yes dsess-server-type = redis
- Restart the reverse proxies
NOTE : the mechanism in ISAM is the same or similar to what is used in IAG, and there’s better documentation on Redis with IAG : https://docs.verify.ibm.com/gateway/docs/tasks-sharing-sessions.
If you now login to the reverse proxy, your session will be stored in Redis.
You can use a multitude Redis client to look at the session in the datastore, but the installation of redis comes with the
redis-cli -h <master_ip> -p 6379
This connects to the redis server, so you can authenticate and run commands.
AUTH isam ThePassWordForISAM KEYS *
You can now see the isam keys. Up to 3 keys can be created per user session (https://www.ibm.com/docs/en/sva/10.0.2?topic=overview-redis-keys)
Test session persistence on WebSEAL
If you don’t use DSC or Redis, you are logged out if the Reverse Proxy restarts.
So you can now restart the reverse proxy, and you will remain logged in.
Test fail over of Redis
To force a failover, you can use the DEBUG command. This will make the replica take over the role of the master, and demote the master to replica.
AUTH isam ThePassWordForISAM DEBUG sleep 30
During this action, you should remain logged in to ISAM.
After the failure occured, the master should now be a replica. You can check that using the
If you still have the redis-cli open, you’ll have to login again.
AUTH isam ThePassWordForISAM info replication
Logging & Tracing
First the bad news : there’s hardly any logging in the webseal log for Redis. If things don’t work, you’re in the dark - it’s not possible to determine from the webseal log that there even is a problem. Users will experience an HTTP
503 code and will not be able to login nor continue existing sessions.
You can enable the pdweb.redis trace setting to see what actually happens. Go to Web/Reverse Proxy. Select the reverse proxy you enabled Redis for and select ‘Troubleshooting/Tracing’
pdweb.redis and enable it.
Trace file example
This example shows what happens and how the Redis client in ISVA determines that it’s connected to the “master” or the “replica” (by issuing the ROLE command)
2022-04-25-13:08:18.003+00:00I----- thread(6) trace.pdweb.redis.mgr:5 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:388: 0x7fe65002dda0: Connected to the redis server: redis1.domain.com:6379 2022-04-25-13:08:18.003+00:00I----- thread(6) trace.pdweb.redis.mgr:7 /build/isam/src/i4w/pdwebrte/redis/AMWRedisPool.cpp:366: 0xf80c30: Created a new connection to redis1.domain.com:6379. 2022-04-25-13:08:18.003+00:00I----- thread(6) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:586: 0x7fe65002dda0: Executing the redis command: ROLE. 2022-04-25-13:08:18.004+00:00I----- thread(6) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:648: 0x7fe65002dda0: The command succeeded. 2022-04-25-13:08:18.004+00:00I----- thread(6) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:443: 0x7fe65002dda0: Reply: type[-1]: 5. 2022-04-25-13:08:18.004+00:00I----- thread(6) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:443: 0x7fe65002dda0: Reply: type: 2. 2022-04-25-13:08:18.004+00:00I----- thread(6) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:537: 0x7fe65002dda0: Reply: string: slave. 2022-04-25-13:08:18.004+00:00I----- thread(6) trace.pdweb.redis.mgr:8 /build/isam/src/i4w/pdwebrte/redis/AMWRedisCollection.cpp:500: 0x12522b0: The server is not a master! 2022-04-25-13:08:19.705+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:796: 0x7fe650001580: Failed to retrieve the next reply: Server closed the connection 2022-04-25-13:08:19.705+00:00I----- thread(5) trace.pdweb.redis.mgr:7 /build/isam/src/i4w/pdwebrte/redis/AMWRedisSubscription.cpp:224: 0xf863d0: The connection for the __redis__:invalidate channel to the collection1 collection has finished. 2022-04-25-13:08:20.000+00:00I----- thread(5) trace.pdweb.redis.mgr:8 /build/isam/src/i4w/pdwebrte/redis/AMWRedisCollection.cpp:310: 0x12522b0: Checking the health of the collection: collection1 2022-04-25-13:08:20.000+00:00I----- thread(5) trace.pdweb.redis.mgr:8 /build/isam/src/i4w/pdwebrte/redis/AMWRedisCollection.cpp:465: 0x12522b0: Checking the health of the redis1.domain.com:6379 server in the collection1 collection. 2022-04-25-13:08:20.000+00:00I----- thread(5) trace.pdweb.redis.mgr:5 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:243: 0x7fe650001580: Creating a connection to the redis server: redis1.domain.com:6379 2022-04-25-13:08:20.004+00:00I----- thread(5) trace.pdweb.redis.mgr:5 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:344: 0x7fe650001580: Attempting to authenticate to the redis server. 2022-04-25-13:08:20.004+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:688: 0x7fe650001580: Executing the redis command: AUTH xxx 2022-04-25-13:08:20.007+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:749: 0x7fe650001580: The command succeeded. 2022-04-25-13:08:20.007+00:00I----- thread(5) trace.pdweb.redis.mgr:5 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:367: 0x7fe650001580: Successfully authenticated to the redis server. 2022-04-25-13:08:20.007+00:00I----- thread(5) trace.pdweb.redis.mgr:5 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:374: 0x7fe650001580: Attempting to set the client name: isva-3861eec8-c492-11ec-b2f7-0050569e684a. 2022-04-25-13:08:20.007+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:586: 0x7fe650001580: Executing the redis command: CLIENT SETNAME isva-3861eec8-c492-11ec-b2f7-0050569e684a. 2022-04-25-13:08:20.009+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:648: 0x7fe650001580: The command succeeded. 2022-04-25-13:08:20.009+00:00I----- thread(5) trace.pdweb.redis.mgr:5 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:388: 0x7fe650001580: Connected to the redis server: redis1.domain.com:6379 2022-04-25-13:08:20.009+00:00I----- thread(5) trace.pdweb.redis.mgr:7 /build/isam/src/i4w/pdwebrte/redis/AMWRedisPool.cpp:366: 0xf80c30: Created a new connection to redis1.domain.com:6379. 2022-04-25-13:08:20.009+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:586: 0x7fe650001580: Executing the redis command: ROLE. 2022-04-25-13:08:20.010+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:648: 0x7fe650001580: The command succeeded. 2022-04-25-13:08:20.010+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:443: 0x7fe650001580: Reply: type[-1]: 5. 2022-04-25-13:08:20.010+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:443: 0x7fe650001580: Reply: type: 2. 2022-04-25-13:08:20.010+00:00I----- thread(5) trace.pdweb.redis.mgr:6 /build/isam/src/i4w/pdwebrte/redis/AMWRedisConnection.cpp:537: 0x7fe650001580: Reply: string: master.