Decision Management, revisited

Our Decision Management blog aims to provide educational materials to practitioners from business and IT. We share best practices, new trends and thought leadership pieces.

Take a look at SMARTS
Carole-Ann Berlioz-Matignon

Carole-Ann Berlioz-Matignon

Co-founder and CPO
Read More
Carlos Serrano-Morales

Carlos Serrano-Morales

Co-founder and CTO
Read More
Colleen McClintock

Colleen McClintock

VP Products
Read More
Marc Lerman

Marc Lerman

VP User Experience
Read More

Live from Decision CAMP 2014 – Tobias Vigmostad – Digitalizing Business and Legislative Rules in the Norwegian Immigration Administration

Decision CAMP 2014

TobiasTobias visited us from Norway to share his experience with the immigration administration.

The objective was to allow business and legal to manage their own rules without any technical assistance.  Automation of immigration rules in a prudent way also means that decisions needs to stay in their jurisdiction, rather than being outsourced to external IT services.  Control had to remain in house.

Ease of automate depends on rules complexity, but they also considered how complexity facts were.  Banking decision are simple in comparison to criminal or immigration cases.  This is an interesting aspect that I have not seen being described with such clarity before.  Taxes or pension rules can be fairly complicated, but the facts are usually black and white.  You are eligible or you are not.  Calculation might be complicated but they can systematically determined based on each application.  In the case of immigration or building code, there is a lot of grey area.  This talk is getting my attention because of all the work we are currently doing with permit applications.

Although decisions in the category of “complicated facts” should be handled differently, there might be some cases that could be more easily automated, to start with.

We all had a chuckle in the room when Tobias translated for us how the press reacted to the initiative.  This is a reaction, a fear we have seen at times about automating decisions, replacing human worker.  We all know that the benefits are great for all constituents.  Though reducing the human workload could be seen as a threat, it is also an opportunity to spend their attention to more interesting work.  Oh my, I could go on and on…  Accelerating immigration decision, as well as permits or criminal investigation seems like a good thing to me.

I am digressing though…  UDI’s approach started with decision support, continued with standardizing, to achieve automation.

As I have seen in many projects, business rules elicitation combined with true collaboration within a team allows to improve the quality of the business rules.  The ideation to implementation to testing to testing cycle is enabled by business rules.  It used to take 6 months to get these ideas tested, now it takes barely a week.

As for the Tax Administration project in New Zealand, business people were thrilled with the system, and asked for more.  The system now processes 5-10k cases per month.

Live from Decision CAMP 2014 – Marcia Gottgtroy – An Intelligence Led Approach to Decision Management in Tax Administration

Decision CAMP 2014MarciaWhat is intelligence led?  You need to start by thinking about your objectives, but you don’t always have to get top down.  there is a little of iteration between top-down and bottom-up.

The Inland Tax Revenue administration aimed to achieve straight through processing, including risk management, with a 360-degree view of customers with real-time data.

In order to do that, it was critical to collect as much data as possible.  The platform they put in place allows for the collection, analysis and

  • data layer
  • integration layer – analytical store
  • decision layer – inefrence engine
  • services layer – business intelligence
  • intelligence team

They started with GST (Goods and Services Tax) because of its large volume of daily transactions.  It was a big issue for the business.  New Zealand is great lab for technology because big is not huge.  Usually they process 4,000 GST refunds for small and medium businesses and another 4,000 for larger companies.  They used to review all the refunds but manual process was becoming too much to handle.  One important detail is that management was fully supporting the initiative.

They applied the CRISP methodology.  Core principles:

  • Collaboration
  • Education
  • Continuous Improvement

The pilot group was composed of 3 wise men, who make the final decision, and 6 champions.  Wise men started capturing the first rules.  The champions then developed on the initial work.

The knowledge was captured in declarative form.  The expert-based risk assessment model took 4 months.  It was captured in rules, including rationale and clarifications.

The pilot was a success.  95% of GST refunds were automatically released.  They reduced by more than 2 days the GST processing time.  Customers got GST refunds faster, over the nightly processing and release of funds.

They reduced the costs for Inland Revenue and for customers.

The feedback after running the pilot for 3 years.  The system is now vital.  They do not want to go back to the way they used to do it.

One fraud case they investigated and found within a month.  The scam had been going for months before.  By printing the whole network, they could visualize the relationships and solve the case.  Check the picture in the slides!

After the first results, they started receiving tons of requests to add to the scope.  Victims of their own success!  They do have to make sure that they control the expansion of the system, and stay focused.  The team has grown a bit to 15 people, but it is stable now.

Live from Decision CAMP 2014 – CTO Panel

Decision CAMP 2014

CTO PanelOnce again, I had to use my persuasion skills to convince our team of CTOs to meet on stage.  It is funny how these guys need to be forced to meet in a panel when they have so much to share and can be so entertaining too!

In any case, once again, Carlos Serrano-Morales (Sparkling logic), Jacob Feldman (OpenRules) and Mark Proctor (Red Hat) shared their views on Decision Management.  James Taylor is our moderator today.

Talking about their current interests, Carlos stressed again the importance of data.  Data, and big data, is becoming increasingly more available as I was recently stressing too.  Data is becoming more open than ever.  Young companies, young engineers, young professionals have less reluctance to share data.  Carlos’s point is that experts will have no choice but to take data into account for more insightful decisions.

Mark and Jacob preferred not to talk about big data.  Jacob explored the use of rules and machine learning to prepare data.

One more point that is concerning Carlos is the ecosystem that youngsters are interested in.  They don’t care about Java like we have been, they want Python and technologies like that.  They want to go fast and integrate stuff dynamically.  That allows them to go fast, and to add more data sources quickly.  The impact for us is that we need to be declarative and dynamic.

Back to analytics, a fair comparison of business rules and predictive models is that models are extremely compressed.  They embed a ton of knowledge in a single formula.  They do not move fast though.  They need to be refreshed on a regular basis, which is a difficult task that is performed by data scientists.  In order to deal with the in-between state, as models start to age, companies use business rules that complement the models, and deal with situations the models do not know about.  At refresh time, the many rules created in the meantime get compressed inside the model.  And the cycle starts again.

We have been extracting rules out of data for a long time.  We just did not call them rules, we called them decision trees or scorecards or something like that.

Live from Decision CAMP 2014 – James Taylor – Decision Management 101

Decision CAMP 2014DCAMP-JTRight after my Sparkling welcome to the attendees, James has been taking the floor to share his experience in Decision Management.  The tone is both educational and informative, sparkled with examples.  I love customer anecdotes.

One important starting point for Decision Management is that it is focusing on operational decisions.  In contrast with strategic or tactical decisions, they are made in large volumes.  The impact of a single decision may not be significant in itself, but the scale of operational decisions compensates.

By automating decisions, you can re-allocate human resources to refine a consistent behavior, or to manage escalations I would add.  The ‘free time’ of expert resources can be effectively spent managing risk or improving decisions in any way that is relevant to the business.

Unfortunately enterprises do not always use Decision Management.  In many instances, they:

  • wait instead of acting — as they do not know how to handle exceptions
  • escalate instead or empowering — as they do not know how to handle exceptions
  • report but not learn — as they do not know how to handle exceptions
  • inconsistently respond — as they do not know how to handle exceptions

Decision Management aims at addressing these issues.

James recommends 3 stages to better operational decisions:

  1. Decision Discovery: identify the decisions that are most important to your operational success
  2. Decision Services: design and build independent decision services to automate these decisions
  3. Decision Analysis: create a closed loop between operations and analytics to measure results and drive improvement

James claims that Decision Discovery is the most neglected stage.  The most valuable decisions to address are often clearly identified by the business owners or by the company board.  I do agree that elicitation has been traditionally a more problematic exercise due to a lack of effective techniques.  There have been methodologies that emerged from the early efforts.  It is exciting to see that nowadays standards are starting to get more widely adopted.  Look for more talks on DMN this week.

Why Decision Management for the Internet of Things (IoT)?

Internet of ThingsHere is a hint…  Earlier this week I shared my views on why Data was such a significant enabler for Decision Management, and vice versa.

With the Internet of Things, we get surrounded by ‘things’ that we want smarter.  The idea is that devices like switches, batteries and any appliance could make decisions in our stead.  I remember the days when we thought that it was science fiction…  Instead of programming the furnace to turn on at this or that time or temperature, we were dreaming of a house that could adapt, anticipate and react to current events or information.  And now, it’s happening.

Continue reading “Why Decision Management for the Internet of Things (IoT)?” »

The Role of Data in Decision Management

AnalyticsIn the early days, we were very focused on knowledge.  Figuring out how to extract, capture and model knowledge biased our approach to business rules the same way that we can get obsessed with nails once we have a hammer in hand.

I am not saying that knowledge isn’t important or valuable of course.

The point I want to make is that knowledge in the abstract isn’t as valuable as it could be with data.  Data is the life blood of decision management.  I came to realize that a few years ago, once I finally took a step back to rethink where we were at in this industry.  It was ironic that we did not see that working for an analytics company back then.

Continue reading “The Role of Data in Decision Management” »

Let’s meet at Decision CAMP

I am thrilled to Decision CAMP 2014present this year again at Decision CAMP.  To make things interesting, Jacob and I will be sharing a project…  Both he and I will be showing off how to build the same tax example.  If you have never seen business rules written from scratch, that should be quite interesting!  Don’t miss our Monday challenge.

I am truly looking forward to the awesome presentations from Charles Forgy, Carlos and the rest of the gang.  The best part of the show though is the networking that takes place.  We really have a wonderful group of practitioners!

I encourage you to join us again this year…  it is absolutely free, and it takes place in the great PayPal facility.  There is really no excuse for not coming if you are in the Bay area.

Decision Camp 2014 – Call for Speakers

dc2014logolongLast year we were one of the organizers and sponsors of the inaugural  Decision Camp.  Decision Camp is a  free event where practitioners, industry thought leaders, and vendors come together to share and exchange information on decision management technologies  (business rules,  business intelligence, analytics, and optimization).  Over 300 people attended last year’s conference over the 3 days of the event!

Decision Camp is unique.  Not only is it free, it is one of the few conferences focusing on decision management in practice where you’ll learn how decision management has transformed the business for organizations in healthcare, manufacturing, insurance, banking, retail, utilities, and telecommunications ! You won’t want to miss this year’s event as it promises to be bigger and better.  The call for speakers is open until May 31, so if you are currently involved in a decision management project and would like to share your experiences with other practitioners, you should consider joining us as a speaker this year!



Learn about our latest SMARTS release – Marrakesh!

Our latest SMARTS release, Marrakesh is out!  This release introduces a significant number of improvements and enhancements, primarily driven by customer requests.  This release is a bit like a souk (an open-air marketplace) in Marrakesh where there are lots of interesting things to see and something useful for everyone!



Browse through the list of new features in Marrakesh, and see if something doesn’t catch your eye!

Support for Functions – You can now define functions in the SMARTS form and invoke them in your decision logic.  Functions are like computations that allow you pass arguments when you invoke them.  Two types of functions are supported, local functions and remote functions.

A good use for local functions is formatting an output message with additional details from your input data and/or a date/timestamp.  The message itself could be passed to the function as a parameter.   In the screen shot below, we pass a message as a parameter to the Message function.

function invocation

The formatted output from the function is shown in the Decision Outcome section including the name of the applicant and date the application was submitted.

Remote functions are used when you need to retrieve information from an external service as part of decision evaluation.  Remote functions are defined with a JSON-RPC 2.0 service entry point.  Obviously, you should exercise some caution when invoking functions in rule conditions as they can be executed many times resulting in poor performance.

Support for Decimal types – Decimal types allow you to define the format and precision of your data.  Formats supported include Currency, Scientific, Percent, Hexadecimal and more.  The decimal data type supports up to 29 significant digits.  It is particularly suitable for financial calculations that require a large number of digits and can’t tolerate rounding errors.

In the screen shot below, the field Premium has been defined as Currency with two decimals.

Screen Shot 2014-04-22 at 4.43.28 PM

Segmented Simulations – SMARTS has always had the ability to run local simulations, allowing you to execute your decision logic against the currently loaded data sample and produce reports in the SMARTS dashboard.  In our Honolulu release, nearly two years ago, we introduced the ability to run remote simulations against very large data samples with potentially millions of records.  These remote simulations execute asynchronously on the server using the SMARTS out-of-the-box MapReduce framework.   New in this release is the ability to run segmented simulations.  A segmented simulation allows you to select one or more segmentation fields (included computed fields).  SMARTS segments the data sample based on the distinct values in the field(s) and executes the simulation against each segment.  You can then view the dashboard reports for each segment in addition to the overall simulation results for the entire data sample.

The screen shot below shows a dashboard report, Decision Outcome,  produced by executing a segmented simulation by state, showing the decision outcome (number of applications approved, referred, and denied) for the entire data sample (in blue) and the number for each unique state.

segmented simulation

Customized Form Import – SMARTS makes it easy to define your decision inputs, outputs, and intermediate values.  In SMARTS  you can import a data sample (from a JSON, XML, or CSV file)  and automatically create a form describing the structure (sections and fields) of the data sample.  SMARTS  infers the field type based on the values in the data sample.  Once the form is imported,  you can modify the form to make it more readable and business-friendly by renaming fields and adding new fields, including computations.  New in Marrakesh is the ability to customize the form on import, allowing you to change the display name, data type, or cardinality.  This is a convenient feature that can save you some time when starting a new project.

Disable/Enable Rules and Rule sets –  During development and testing you may need to disable some rules or rule sets to help you understand how your changes are impacting your decision results.  One way to do this is to temporarily add a guard condition (setting the guard to false) to keep the decision logic from executing.  Prior to Marrakesh, this was the only option.   In this release you now have an option in the rule and rule set menus to explicitly disable and enable.  Disabling a rule or rule set makes it invisible to compilation and execution, so you can temporarily remove rules or rule sets from the decision’s execution.

Guards are different- they are compiled and evaluated at execution time so that the rule or ruleset will only execute if the guard condition evaluates to true.  You can use guards when you need to dynamically determine whether or not to apply a rule or rule set to the current document or transaction.

Assume you want to disable your Senior / Low Mileage rule for testing.  In the screen shot below, you can see the menu item to disable the rule.

enable disable menu

Once the rule is disabled, the name of rule (or rule set) will have a line through it as shown here:

disabled rule

Markdown in Descriptions – The ability to maintain traceability from a rule or rule set to its source requirements is a common requirement in business rule and decision management applications.  Many of you requested the ability to provide a link to an external document or repository containing the source policy, regulation, or requirement.  In Marrakesh, we now support Markdown in descriptions for decision artifacts.   Markdown syntax is very readable, easy to learn, and can be converted into HTML.  So now, you can include hyperlinks in your descriptions and you can format your descriptions too!




In addition, Marrakesh includes some exciting new features and enhancements to BluePen, the analytics module for SMARTS.

Improved Scalability and Performance – You can now run analytics jobs against much larger data samples.  Also, many of the key algorithms for rules generation have been enhanced for better scalability and performance.

More Algorithms – new algorithms have been implemented for variable identification, variable discretization, and rules generation.

Segmented Analytics Jobs – Just as for simulations, you can now run segmented analytics jobs, selecting one or more segmentation variables and triggering jobs that will execute against each one of the corresponding segments.  This will potentially yield results that are better tailored to each segment.

Streaming Rules Generation Algoritms –  Beta support for streaming rules generation algorithms has been added.  These algoritms don’t require that all the documents in the data sample be in memory and therefore can scale to very big data sizes with very little memory.

We hope you enjoy our latest SMARTS release.  We  appreciate all the suggestions you provided that shaped this release.  Please let us know what you think.

If you haven’t yet used SMARTS, some of the new feature descriptions likely lack context.  SMARTS is the most intuitive and powerful decision management offering on the market.  Attend one of our product tour webinars or sign-up for a free trial to learn more!



From Business Intelligence to Better Decisions

Last week we jointly hosted a webinar with our consulting and implementation partner, Mariner.  Shash Hedge, BI Solution Specialist from Mariner, described operational BI, its challenges, and some traditional and recent implementation approaches.  He concluded with a few cases studies of operational BI projects that were missing an important piece — the ability to make decisions based on the operational insight provided by the system.

Operational BI systems provide critical insight on business operations and enable your front-line workers to make more informed decisions.  But as Shash highlighted, insight delivered in the right format, to the right people, at the right time is often not enough, you need to make decisions based on that insight in order to take action…

I lead the second half of the webinar, introducing decision management and describing how it complements operational BI.  Watch the recorded webinar to learn more.

YouTube Preview Image

The recording is a bit rough when the video gets to my part; it sounds like I am presenting from another country!  We’re planning another joint webinar in May where we will cover the topic in more depth and demonstrate how these two technologies complement each other.  Stay tuned for dates and registration information.  I’m sure we’ll get the sound issues resolved next time!