In assessing state and territory rule communication, it is also important to note the quality of policy manual access and deployment across states given the crucial role these manuals play in communicating rule updates to public organizations.

Ben Molin, the co-founder of SNAP Screener, an unofficial calculator that helps users to find out whether they are eligible for food stamps and estimate the amount of benefits they can receive, provides insight into his experience using state SNAP policy manuals in an interview with our team:

“The most important resources a state can provide about SNAP are the accurate (BBCE) gross income and asset tests for their state. So my two criteria for evaluation of a state’s resources would be: (1) How easy is it to find the BBCE income limits? And (2) How easy it is to find the utility allowances. If they have utility allowances information easily available, it is a great indicator about the rest of their resources.”

SNAP Screener maintains a publicly-accessible table of eligibility parameters for SNAP, which brings together information from federal and state resources.

Molin identifies Pennsylvania and Alaska as two states that have excelled in communicating eligibility rules in their policy manuals. On further examination, we highlight several important features of policy manual communication:

1. All policy rules are presented in a single interface. The interface is accessible and includes links to all pages via an easy-to-use navigation bar that can toggle between different policy rules. (See both Pa. and Alaska, Figure 1).
2. There are separate pages for figures presented in easy-to-read tables with clear effective dates (see Pa. and Alaska, Figure 2).

‍

Documentation is vital to the success of any rules as code approaches. We define three categories of documentation: limited, moderate and extensive. Limited documentation explains how to run the code and includes some general background information. Moderate documentation explains the variables and provides some basic information about how a decision about a specific benefit is being made. Both limited and moderate documentation will also typically lack some organization. Extensive documentation shows how the software can be used and explains the different design decisions. It maps actions that a typical user would conduct to definitions, variables, and code logic snippets. Calculations are also clearly explained. Together, the documentation makes it easy for developers and analysts to see what needs updating when policies and rules change. We find that in general, documentation tends to be moderate or extensive. This is important since many developers who want to use these code bases may not be well versed in SNAP regulations or in a rules as code framing.  

We see that each of our four case study examples uses a different programming language and different framework. For example, for its Policy Rules Database, the Federal Reserve Bank of Atlanta has built a custom rules engine. It is not built using a rules as code framework, such as an open source template for implementing rules as code that can be expanded on to fit a specific use case, (e.g.: 18F or OpenFisca), or any tool specifically used for encoding rules (such as Drools).

Instead, it uses its own structure, implementing the logic for each program in a function using information from data files that contain the eligibility criteria. The tool was built using the R programming language. While R is a general purpose statistical programming language, it is not as ubiquitous as Java, Python, or JavaScript. The program data files do persist over time, though and show the state eligibility criteria for each program annually for different household sizes. This is a beneficial component since it maintains a historical record of policy changes and gives users a way to  understand how changes impact benefits usage.

We do have one concern with the Policy Rules Database code. There are some specific rules or calculations that are hard-coded in the program. If those rules change over time, not only do the data files need to be updated, but so does the code itself. This means that the code base is probably less flexible than one with a higher level of abstraction that was designed with more modularity in mind. In general, this tight coupling between the rules and the code makes it difficult to identify what information should be updated and where in the code it needs to be updated in response to a change. The Atlanta Fed is in the process of updating their tooling with a different language with the goal of addressing some of these concerns.

ACCESS NYC decouples the policy logic from the application code by using the Drools framework. Drools is a rules engine that was developed using Java, a general purpose programming language. This rules engine allows for rules to be written at a higher abstraction level than programming them directly in Java or a different general purpose programming language. The Drools engine does allow for the rules logic and the data to be separated, new rules can be added without updating existing rules, and the rules themselves are readable. While this solution has a number of strengths, including codebase standardization and straightforward creation of new rules, there are some drawbacks as well. First, there is a steep learning curve required to use the Drools framework. Second, the format of the rules is specific to the Drools’ rules engine and is not easily adjustable to other formats. This limits its interoperability.  So even though the framework is built using Postgres, an open-source relational database, and the Java programming language, it is a highly restrictive language with limited exporting capabilities, making it difficult to integrate easily with a different technology stack.

The web-based ACCESS NYC eligibility screener serves as an example of the type of tool that can be built using the business logic stored in the rules database. The organization’s team also created an Application Programming Interface (API) that calls the rules engine in order to determine potential eligibility for various programs outside of the ACCESS NYC front end screener. This can be used for batch screening households for benefits or creating new benefits screening tools for a specific use case as it enables running household data against eligibility rules for city, state, and federal programs.  

Starting in 2017 through 2020, a team at 18F, part of the General Services Administration, prototyped how the federal government could communicate benefit eligibility rules as computer code for integration into state systems. Under their Eligibility APIs Initiative, the team built a prototype API and calculator for the federal rules for SNAP.  Through this development, they created the open source 18F prototype framework, which has now been reused by other tools like SNAPScreener. SNAPScreener extends the data model to all U.S. states and territories. In the framework, the program data is stored in JavaScript format. The correct values for information such as state deductions amounts, income limit factors, and broad based categorical eligibility determination are retrieved for the state and year from the JavaScript file, and applied to common functions that can be used for any state to retrieve an individual’s estimated SNAP benefit. We view this solution as one that mixes the benefits and functionality of the Federal Reserve Bank of Atlanta and ACCESS NYC. The rules data are stored in a separate file that can be updated without updating the code associated with the user application itself. Simple text files allow for the decoupling of the data from the application interface code logic. Traditionally, these simple files can lead to complexities when formats change since all the data in the file will need to be rewritten in the new data format. However, this framework uses the JSON format for the data stored in these text files. That format is a well established lightweight format that is as interoperable as data stored in a traditional relational database. Also, the JSON files can be shared publicly, making it more accessible than many government databases. The disadvantage of formats like JSON and XML compared to SQL (the language used to access data in traditional relational databases like MySQL, Postgres, and SQLite) is that code needs to be written to manage, access, and manipulate the JSON files. With SQL, the data management can be taken care of by the database management system.

Finally, PolicyEngine also uses tools specifically designed for handling rules logic. It is built on the international open source OpenFisca framework. The tool uses YAML files to store information about the benefit programs, while its backend is implemented in Python. Its client-facing website was built using JavaScript. Similar to the Policy Rules Database, it computes an applicant’s income eligibility threshold using a multiplier against the federal poverty line threshold. This multiplier, like the threshold itself, is stored in YAML. The language the tool uses is generally used for working with data and, similar to JSON, is serializable. Some developers prefer using YAML because it allows for commenting and supports more complex data types than JSON.

Because PolicyEngine supports a wide range of programs and holds information about tax policy as well as benefits, it contains significantly more data than the other tools we analyzed. In general, YAML has the same strengths and limitations as JSON in terms of interoperability, scalability, and comparisons to traditional relational databases. We note that all of the files generated using these different markup languages can be readily moved to databases as the number of rules and files grows. During the development period, using files that allow for flexible schema adjustment may be more efficient and enable more projects to be developed concurrently.

Overall Design Considerations

We can see from the high level comparison that each team had different constraints when designing their tools. The most noticeable difference was the variation in the programming languages used. While the languages each of the organizations used are mainstream, general purpose, or data languages, they are different in terms of the use cases for which they are best suited. The rules developed are also not language agnostic. As previously mentioned, the examples we examined have varying levels of application and data coupling. Those with higher levels of abstraction may be more readily portable to different contexts, such as other states or programs. The tools also differ in terms of when conditions are applied. For example for SNAP, the gross income test is not applied to households with elderly or disabled members. PolicyEngine includes this check as part of its calculation, while the 18F prototype keeps the test as a part of the gross income computation threshold and only calls the test when applicants have specific household information. While this may seem like a trivial distinction since both approaches lead to the same result, this policy rule was translated into different code logic. If there are no basic guidelines as to which logic structure to use, variability will occur and portability will decrease.

These differences highlight a number of design considerations that are important moving forward. First, a broad design consideration is whether or not the software should be developed for a specific benefits use case. In this study, three of the four examples were designed specifically for SNAP. This means that it may require more effort to adapt to other benefits programs.  Second, there are rules that are fairly consistent across states and rules that are very specific for certain jurisdictions. It is important to design tools that compute rules that are compliant with policies in the jurisdiction of interest. But it may also be beneficial to standardize how policy and rules logic map to rules code, irrespective of the state the rules are being developed for. This would help make the rules code (and rules engine) reusable for different benefits programs. Finally, if we want plug and play rules as code, we need to modularize and separate out different components so that standards can be established for each component. As an example, designers may want to think about what should take precedence in terms of determining eligibility: categorical eligibility or standard eligibility thresholds? Consistency across these types of design decisions are important for increasing modularity and reusable components.

To make some of these design decisions more concrete, we suggest considering the following categories of decisions: (1) rules parameter data format, (2) the mapping between policy, rules logic, and rules data, (3) rules engines, and (4) developer interfaces. The rules parameter data format consists of the naming and data value conventions that are being used to determine benefit eligibility, e.g. income level or poverty rate. The mapping between policy, rules logic and rules data is the pseudocode that connects the policy and data fields to conditional logic. For example, add A + B + C to produce TOTAL and check that TOTAL < D. Rules engines generate and interpret the rules to ensure compliance given the set of data fields and rule logic. Developer interfaces are what the application developer uses to build interfaces that users can interact with when determining their benefit eligibility.

Ultimately, each of these design decisions will influence how uniform the rules are across contexts and how easily different states can take advantage of the work done by the rules as code community. When looking at these components in the context of the specified goals, all of these design decisions are important for portability and standardization. This includes standard file formats, data standards, rule languages and specification standards, etc. The usability of the rules as code framework hinges on a clear and easy to understand mapping between policy, rules logic, and rules data and an implementation of them within rules engines that is simple to use and straightforward to add new policies to. If the rules and data are very tightly coupled, the code will not be usable outside of its original context without custom code for adapting the rules code to the new context.

In the case of our examples for SNAP, the SNAP Screener prototype does use a portable data format (JSON), but the mapping between the policy, rules logic and rules data are custom implementations that would need re-implementing for other use cases. These rules were not generated using a rules engine and the developer interfaces are also not generic. One strength is that the code-base is straightforward and could be manually adjusted for different use cases. If we consider ACCESS NYC, it is the best example of decoupling data and application logic. It accomplishes this by using a rules engine. The Drools engine is used to standardize the generation of rules, ensuring both a consistent data format and a strong mapping between the policy, rules logic and rules data. While the generation of the rules can typically not be accessed by the public, the API gives access to the outcome and these outcomes can be integrated into different applications that users can access. As a general principle, the more independent each of our four design decisions are, the more versatile the code-base will be and the easier it will be for multiple disparate states to use different modules.

Recommendations

As we work to advance the use of rules as code, each of the different case study examples we examined can be used to evaluate the strengths and weaknesses of software design decisions. While this cursory look has uncovered advantages and disadvantages, there are many aspects of information we are still missing. For example, what resources were available for each of the different initiatives? What resources does each entity need in order to successfully maintain and update software when program changes occur? How stable are the languages used to code each application and do any of the organizations have concerns about software or stability moving forward?

Given these questions, our community needs to come to an agreement on a basic set of standards in order to support states considering using rules as code. We also need to think about ensuring interoperability between rules engines in order to reduce the burden of development for different states. Finally, a more comprehensive auditing of code, even those created by states that have more proprietary codebases, should be conducted. Doing so will ensure that states will implement these code bases and rules correctly and that every rules as code implementation is designed to be extensible and can be easily adapted to constraints that exist in different states.

‍

‍

Policy Rules Database

Policy Engine

None of the screening tools base their determination of categorical eligibility purely on whether or not an individual self-reports being a recipient of one of the programs that would make them eligible for SNAP. Instead, a user's answers to the eligibility questions are used to compute whether the individual is eligible for benefits that would also make them eligible for SNAP. Therefore, categorical eligibility appears to often be treated in the same manner as a standard eligibility threshold rule.

The different approaches for coding these rules provide insight into potential methods for standardization. As can be seen in the examples above, even within a single organization, details such as variable names are handled very differently. The Policy Rules Database, for example, uses both camel case (eg: GrossIncomeEligibility) and snake case (eg: not_categ_elig_ssi). While this does not impede functionality, it is not generally considered a best practice as it has a high chance of errors.

On a broader scale, we can see that there is a disparity between organizations in terms of how modular the code is, and where in the process a value is computed. For example, organizations such as PolicyEngine use more complicated tests for eligibility criteria, while the 18F prototype approach involves simpler tests that are or are not called, depending on the specific household information. This leads to redundant code that could be made more modular by using functions. The same holds true across organizations. In general, many organizations are recreating the same functionality since these applications have been developed independently, in a grassroots way, rather than around a standardized framework.

Even when an organization has a publicly available repository with code that can be extended for a particular use case, they rarely have public APIs. The end result is the need to recreate the work within a new project, rather than being able to hit an endpoint to retrieve the information in question and build the desired extended functionality. Supported collaboration and standardized rules syntax with open APIs could improve efficiency and interoperability.

‍