Storytime! Because of a series of recent events I felt it is time to publish a few stories of what could go wrong when implementing ServiceNow® from a security perspective.
While some of the stories I gathered over time show pure ignorance to basic security best practices, some show a lack of understanding and foresight regarding system security.
While the stories might seem (not) funny, I will try to give general solutions or patterns to avoid the underlying problems the customers were facing.
Stories
If you make it admin, it will work definitively
A while ago I had to implement an integration to Zabbix, which is a monitoring solution and provides a share and store similiar to ServiceNow. As another ServiceNow customer already built an implementation, I thought it might be a good idea to base on the existing work already done.
While I was working my way through the implementation to understand it‘s functionalty and adapt it‘s patterns, I discovered „forgotten“ credentials (username, password, instance URL)… The story would be over here if they followed a few security best practices.
The credentials still worked on their instance… „Okay“ I thought, „these are only sub-productive instances“. I already need to highlight, it was the same username and password that was used for the sub-productive instances! This becomes important now: because the instance names followed a naming pattern (*dev, *test) it was easy to determine their productive instance. And, well, the same username and password combination worked there as well.
And another time, the already not so funny story would be over, if not another security best practice was ignored: the user had the admin role!
You might think, „Well who grants the admin role to an integration user?!“… Surprisingly many customers do that and it‘s not just a veeery bad habbit! I‘ve seen that too many times already! There are a couple of reasons:
- Time: somethings needs to get done and none is taking the time to determine the needed access rights… admin just makes it working. It‘s a provisional solution — which will never be challenged after being in place, because, well, it works just fine
- Configuration: Some third-party tools require access on an admin level, e.g. because they provide a list of tables and fields for configuration reasons. After the configuration the admin is not necessary anymore because access is only needed for the configured tables and fields. But again: no one is raising a hand, because, well, it works just fine.
Try to avoid these reasons or try to work around them!
Lessons learned
- Never use the same password (and user) for different instances, especially for productive systems!
- Remove all credentials on your publications, always double-check!
- Never ever grant admin role to integration users on productive systems!
- Clearly define, what an integration user has to do and needs access to. Assign the most restrictive role available fulfilling the access requirements!
I made it work without admin, right?
It seems very popular no one cares much about technical users. They are technical, (two) machines speak to each other, in a clearly defined way. While this is true, it ignores the fact, that a technical user could be misused by a human being.
This story resembles the first one, except the technical user didn’t have the admin role. The rest is very very similar. The interesting fact with this story is that the user was able to modify itself.
It had the flag „integration user“ set which inherently just allows access via web services by the ServiceNow security model. Interactive logins are blocked. But what if the user modifies itself, removing that flag? The integration still works — no one will notice because there are no hick-ups — and we can login into the instance. And, of course, we can remove obvious traces on the user record itself then.
Being able to access the UI also makes misuse much easier: we see our access level at one glance because applications and modules are hidden for certain roles. We therefore can work our way through the available information.
Lessons learned
- see Story 1
- Under no circumstances an integration user should be able to modify itself! (This applies to everything related to the user, including roles and group memberships)
- Monitor interface and user interactions:
- The Transaction log provides information which transactions were done. Lookout for Form and List transactions which shouldn‘t appear for integration users.
- Setup a monitoring of changes made to users.
- Log authorization requests for integration users.
Admin isn’t even necessary if you just don’t care enough
On the list of more complex security incidents is a case, which could be produced by every normal developer… but good news on this one: this, in many cases, is just a habit which can be changed.
The story started with a Helper functionality to facilitate a very responsive UI, which loads bits of information just in time. It was designed to react on user input and deliver desired information. The scope was very narrow — but the solution was very general.
From an architect’s point of view I can totally understand the urge to create reusable, general solutions rather than highly specific solutions. From the security perspective, things should be as scoped as possible to only deliver as much information as needed and as less as possible. Those two views don’t need to be contrary, if the user input is validated correctly.
And eventually that’s what didn’t happen.
There’s been no input validation so that the helper function could be misused to retrieve information besides the actually intended ones. This was going so far that any arbitrary table (incl. the system tables with the prefix sys_) could be queried without any access control. So a normal (even a role less) user was able to retrieve (security relevant) properties and fields containing two-way-“encrypted” passwords.
One word to two-way-“encrypted” passwords: they just seem to be secure. A system that stores them for usage in an automated way needs to decrypt those passwords for that exact usage. As the system needs to decrypt them, a user could use the system itself to extract the clear text password. You could say, the passwords are rather encoded than encrypted — and the codec to encode and decode the password is the same for all ServiceNow instances.
On top of that there would have been an easy solution: using GlideRecordSecure instead of GlideRecord.
GlideRecordSecure automatically only returns Records and Fields the current user can access. This, regardless of a user input validation, would have been a saftey net — and doesn’t behave much different to the normal GlideRecord from a developers perspective.
Lessons learned
- Always try to find the right balance between a general and a purpose specific implementation.
- Do proper input validation!
- Do this even for technical interfaces as they are potentially misusable!
- This also allows a greater system stability.
- Use the safety net that are there! It’s like on a construction site — put the safety net in place!
- Use the safety net that are there! It’s like on a construction site — put the safety net in place!
- Use the safety net that are there! It’s like on a construction site — put the safety net in place!
- Putting this three times was intentionally.
Credentials leak!
If you think about it, Story 3 can lead to Story 2… a role less user has access to system parts it shouldn’t have access to and is able to retrieve credentials for a different user with much higher access rights. From there we work our way through much more efficiently.
Or simply someone just leaves his/hers credentials somewhere unintentionally / by accident.
There are so many ways a system can get compromised — or some of your (external) employees are simply forgetting something.
This applies to any credentials, but especially to integration user credentials — because a compromised integration user always means compromised access to at least two systems.
This is why you should see that it’s the nature of credentials to leak — and that’s why there are password policies demanding a password change every x days. Stick to those rules, they are a safety net as well!
Anti-Patterns and better Patterns
So let me conclude what I found so far:
- Never use a combination where user name = password
This is a very very sad behavior and everyone doing that should be fired immediately! - Always remove your credentials from places, where you don’t store them primary or for recovery.
- Never ever grant admin rights to non-admin users!
- Make integration users unable to change parts outside of their integration scope, especially information regarding the user itself!
- Build an active monitoring system for integration user activity.
- Build awareness among your developers and architects for security perspectives.
- Promote security-aware patterns
- Promote to think outside the actual scope
- Develop in pairs — one being the good guy, one being the bad guy that want’s to break out of the solution
- Discuss solutions from an access perspective
- Use secure mechanisms that are already in place!
One last word
These are real world scenarios I was going through. They are not constructed. They all had a great potential of compromising the affected companies. All the companies reacted very fast! If you could do yourself a favor, have a Security Incident Response process in place — also in regards to the GDPR and similar laws.