Monday, September 19, 2016

Traffic Light Logic - A Finite State Machine Example

A "finite state machine" (FSM) is a model of computing in which the program is conceived as a machine which can be in one of a finite number of states. The FSM model is proposed as having advantages for a variety of applications and the example of a traffic light is often given, so it seems worthwhile to see how the Blockly for Business Logic "flat functional specification" would look for a traffic light example. For comparison, a traffic light example implemented in the machina framework can be found here.

The example implements BBL for a traffic light having the following requirements:

  • a north-south main street is green for 180 seconds, then yellow for 5 seconds
  • an east-west secondary street is green for 20 seconds, then yellow for 3 seconds
  • there is a 2 second delay after one street turns red before the other turns green
  • the east-west walk signal (for crossing the north-south main street) has:
    • a 5 second start-walking signal
    • a 20 second continue-walking signal
  • the north-south walk signal (for crossing the east-west secondary street) has:
    • a 3 second start walking signal
    • a 5 second continue-walking signal
  • there is a walk-request button for the east-west walk signal

Top Level Specification

Here is the BBL "flat functional specification" that implements the requirements:

This top level specification contains five outputs:

  • a possibly new color for the east/west light, or empty for no change
  • a possibly new color for the north/south light, or empty for no change
  • a new state for the east/west walk light (start walking, continue walking, or don't walk)
  • a new state for the north/south walk light
  • an output to reset the east/west walk request when it has been handled

East/West Light Change

The specification for the first output, possible east/west light change, begins as follows:



If the east/west light is green and has been for 20 seconds, turn it yellow. Otherwise, if it is yellow and has been for 3 seconds, turn it red. This is the rest of the specification:

If the light for the other direction, the north/south road, is red and has been for at least 2 seconds, then turn the east/west light green. If none of these conditions are met, return an empty result, which means to leave the light unchanged.

North/South Light Change

The specification for light changes for the north/south road is similar, except that the durations are different and there is logic for handling the east/west walk request.


The north/south light changes from green to yellow if it has been green for 180 seconds, or if an east/west walk request is being processed. The east/west walk request is handled if the north/south light has been green for at least 20 seconds and the north/south walk signal is at "don't walk", because obviously you don't want the light to change while the north/south walk signal is still at "walk" or "continue walking".

Walk Signals

Below is the first part of the specification for changing the east/west walk signal.


When the east/west light is green, the walk signal changes to "walk" until it's been green for 5 seconds. This is the continuation:



When it has been green for 5 seconds, it changes to "continue walking" until it has been green for 25 seconds, when it changes to "don't walk". The north/south walk signal is similar, but with different durations.

Resetting the Walk Request

There's a pedestrian button to request the light to change when crossing the main north/south street, which otherwise has a long, 180 second green light. The pending request is reset when the east/west light turns yellow.

Processing Loop

The code created from the flat, functional specification defined in BBL is called in an implied loop implemented outside of BBL. The BBL-generated code would be called at least every 1/2 second, passing the current state of the lights and the length of time they have been in that state, and receiving back the new state.

Generating Code

One of the most important benefits of the BBL model of active specification is that the specification can be translated directly to multiple target platforms. In this example, a simulator may be created for validation, and when validated, a translation to a completely different target language may be required if, as is likely, an embedded platform is targeted. Many cities now have centralized network control of traffic signals. With the logic specified in BBL independently of these implementation details, it becomes more efficient and more reliable to change the target platform. Once a new translator is implemented for the new target environment, migration should go much more quickly and smoothly.

Conclusion

There are quite a lot of different possible configurations for traffic lights. Some intersections have more than two roads intersecting. There may be left and/or right turn signals. Sometimes all the traffic lights are supposed to be red when pedestrians are crossing. Sometimes you don't get the walk signal at all unless you request it. The BBL model lets you get control of a wide range of logic like this, and analyze and validate it with a minimum of distracting and irrelevant verbiage, with the objective of achieving rapid and highly reliable development and maintenance.  

Saturday, May 7, 2016

Getting Super Good at Making Large Complex Objects

Elon Musk says he wants to know how to "get super good at making large complex objects". BBL can help.

No one builds 747s. People build parts of 747s, parts of parts, and parts of parts of parts, then put them together. It is perhaps the essence of engineering that, to do a task that is impossible, you decompose it into parts that are less impossible, then further decompose those parts into parts that are less difficult, and so on, until you end up with parts that are simple. Then you do all the simple things and put it all together, starting at the bottom.

The idea of "Blockly for Business Logic" (BBL) can be summarized as: decompose into sub-problems that can be expressed as functional logic on flat data, expressed in Google's "Blockly" visual programming tool, then combine them.

Blockly makes it easy and practical to express logic "functionally", and logic that is expressed this way can be more rigorously validated. Encapsulating the "flat" part of the processing separately from data, user, or machine I/O is a powerful simplification that eases comprehension and encourages rigorous unit testing. The impossible becomes almost too easy. I would say that doing the impossible is in fact impossible unless you first make it simple.

I'm not ready to attempt the construction of a 747 in BBL quite yet. A simpler example, though, may convey the idea, with a little imagination, that BBL can be scaled up to any project.

The Pinewood Derby is a racing event for Cub Scouts in the Boy Scouts of America. Cub Scouts, with the help of parents, build their own cars from wood, usually from kits containing a block of pine, plastic wheels and metal axles. The step-by-step instructions for building a Pinewood Derby racer found here are the basis for this BBL example.

At the top level, the racer is complete when certain steps are complete, as follows:

The question arises whether the steps need to be performed in the order specified. The answer is "no", because this is supposed to be "functional programming", in which there are no implicit dependencies. In the example, some steps could be performed in a different order than what is shown but not others. It should not be painted before the block is shaped, at least. To enforce this dependency, block shaping completion before painting is included as a condition of "(is) painted", as follows:
A successful car must meet all such conditions. What happens if a condition is not met? The build of the car fails. What happens then? How exactly are these conditions detected? That's outside the scope of BBL.

A bigger but related question is, what does this program "do"? Does it build a car? Does it help someone build a car? The answer is that it could do those or perhaps many other things. All it says, so far as it goes, is that these things must be done to build a car. How it's used exactly is outside the scope of BBL. What BBL produces is an XML analog of the visual program. How that XML is used is another matter.

But isn't this utterly trivial? Yes, and that is exactly the point. And no. The process should be decomposed into parts, each of which is trivial. If a piece of the composition isn't so simple as to be boring, then it hasn't been sufficiently decomposed. But the end result is not trivial. The end result is a slam-bang guaranteed successful car build, with much higher probability than using conventional software methods.

What this module "does", then, is just this:  it controls the building of a car. If "painted" is removed, the car will not be painted. If something completely new needs to be done, it must be added or it won't get done. It's a sort of checklist, but an active one, though its activity is implemented elsewhere. It's seems to me that, in order to be "super good at making large complex objects", just this sort of active checklist must be maintained, in exactly one place, and in a form that is boringly simple and which ignores everything except what it does, which is to specify that these steps make a Pinewood Derby car.

This sort of thing, when implemented in conventional software, is usually called "configuration", and is often done these days using XML. So one way to look at BBL is as a configurator. It has five things that configuration schemes usually don't have, though:

  • It uses Blockly visual programming.
  • It adheres to the functional programming paradigm.
  • It adopts the powerful simplification of operating only on flat inputs and outputs.
  • It uses a small, simple, consistent, customized programming language.
  • It configures the entire process, not just higher level or partial components.
Continuing with the example, the "inspect and prepare" step breaks down further as follows:
When we look at the "block inspected and prepared" step, it starts to get a little more interesting.
If someone worked on this, and told you they made a mistake, what would you think of them? Who could possibly make a mistake doing something this simple? Well, I did. My first attempt looked like this:

The blue triangle means that a validation has failed. It confused me for a moment, and at first I thought there must be something wrong with my validation JavaScript. Clicking on the blue diamond gives the error message, though:

This message means that only a single element is allowed here and I was trying to give it more than one. I forgot to warp it with the "All" block, which returns a single "true" element if all the boolean conditions it contains are true.

Next we look at "block is not warped". Finally we're deep enough into the logic that we're actually "doing" something. We're going to test the block to see if it's warped by seeing if it rocks when placed on a flat surface.
The "block side 1" test look like this:
You press each of the 4 corners in turn to see if it rocks, then turn it over and do the same on the other side.

If it rocks, the build returns false. How exactly you recover from that is, once again, outside the scope of BBL.

It seems like an awful lot is "outside the scope of BBL". Yes, that's true. Does it mean that using BBL is pointless? Not at all. By isolating and simplifying this logic, you end up with a powerful and enforceable view of the problem.

To "black belt" programmers, this may all look simple-minded, and it is. I'm basically simple-minded and I hate making things difficult for myself. In my 30 years as a software developer, I've always written small modules like this, if they let me. I have something of a reputation for it. One time my manager came to me with a new project and gave me strict instructions to write it as a single module, because he thought my style was "boring". I did, and it came out to about 1,000 lines. Then he went and made a change to it on his own initiative, which created a subtle error in the results that did not get caught in testing, which turned out not to be boring for him or the customer.

When you run your software, you want it to be boring, every time. You don't want it to end up in the evening news. 

Why BBL? Some day, perhaps, someone will use a robot to build Pinewood Derby cars. When that day comes, no change will be required to anything in the BBL. It is the same for cars built by robots as it is for cars built by Cub Scouts.

Why BBL? It produces boring results.

Why BBL? It can be created, modified and understood by people who like football more than they like software development.

Saturday, April 30, 2016

Inventory Reordering Example: What BBL is Good For and What It's Not Good For

"Blockly for Business Logic" (BBL) is not for everything. The case for using this approach can be summarized as:
  • Logic that needs to be customized for the particular organization. If every organization could use the same logic, then there would only be need for one off-the-shelf software solution for every organization, but even small organizations often find they need a high degree of customization to be effective. 
  • Logic that changes relatively often. Every organization that hopes to survive must adapt to changing laws, changing markets and evolving risks and opportunities, but the cost, time and dangers of changing the logic embedded in traditionally-structured software holds back this vital adaptation.
BBL is not intended to be a general-purpose programming language, though. The power of the concept is in its simplifications, but these simplifications mean that there are some use cases it can't handle well or can't handle at all. The following inventory reordering example shows what can be done with BBL and what needs to be handled outside of it.

The example involves determining whether a particular product needs to be re-ordered because inventory is low. On the face of it, it seems that it should be a relatively simple calculation. Many businesses simply create a "reorder point", which is the inventory level where the product is re-ordered. This simple approach has the following problems, however:
  • Shipping costs can be reduced and better volume discounts obtained if re-ordering is done less frequently and in larger shipments.
  • Limited storage space and spoilage restrict the size of re-orders.
  • Demand for particular products can sometimes be quite variable and difficult to predict, for example if Oprah features a product on her television show, making it impossible to be 100% sure that you will always be able to remain stocked in all items.
  • Some products are more important to the business than others and must be re-ordered more frequently if necessary. For example, if a grocery chain runs out of a popular brand of beer even occasionally customers will stop going to that chain, but they won't stop using the store just because once in a while the store doesn't have saffron.
In the example, these problems are handled with the following specification:
  • Products are assigned an "importance" value between 0 and 1, with a low importance, 0.1 for example, being assigned to a product like saffron, and a high importance, 0.9 for example, being assigned to a product like Bud Light.
  • Re-ordering regimens are identified as daily, weekly, monthly or quarterly re-orders. Daily re-orders are only performed for high importance products, and weekly, monthly and quarterly used for progressively less important products.
  • Each product has a "lead time", which is the time from placing the order until it is available for sale. The projected sales during this lead time for each product are calculated and compared to current inventory. For high importance products, the product is re-ordered when the amount on hand is below 115% of projected sales, and for low importance products, it's re-ordered when the amount on hand is below 95% of projected sales.
At the top level, we have two conditions that need to be met, as follows:
The "(is) product reorderable" determines if the product is important enough to be re-ordered more frequently, that is, daily or weekly. The "(is) product needed" determines if the inventory has fallen low enough in relation to the projected sales to be reordered. Both of these are "calculated inputs", meaning that they are defined in BBL. Here is the definition for "reorderable":
The "product importance" is compared to a cut-off value for each reorder type, daily, weekly or monthly, to determine if the product is eligible for re-order for that type of order. All products can be re-ordered if it's a quarterly re-order.

Finally, here is the definition for the "product needed" input:

The product is needed if the quantity in stock is less than the projected sales during the time it takes to receive the product and make it ready for sale, adjusted up 15% if it's a high importance product to reduce the chances of running out in the meantime, or adjusted down by 5% to reduce costs of frequent reordering when it's a product that doesn't matter so much of you run out.

The example shows the part of the process where BBL works well, but it's also necessary to point out what part of the process is being handled external to BBL. The input "projected sales" is a particular example of this. It's an input to the BBL and is calculated elsewhere as follows:
  • Determine the "lead time" for each product--the time it takes from ordering until the product is ready for sale. It includes the time it takes to process the order, the time it takes for it to be filled, shipped and received, and the time it takes to place it "on the shelf" ready for sale.
  • Estimate the cumulative sales of the product during this lead time, taking into account such factors as seasonality and promotions. This requires estimating daily sales and summing them for the lead time period.
The "projected sales" calculation is not suited to BBL calculation. BBL calculates only from "flat" inputs, and the "projected sales" calculation is not flat, because it requires summing over daily projections for a "lead time" number of days that is different for each product. That requires a "loop" and BBL does not support looping in order to achieve simplification.

The projection for a single day could possibly be done in BBL, though, as well as the lead time. These would be inputs to a conventional process, probably SQL, that produces the "projected sales" input to the inventory re-ordering. Whether it's a good idea to use BBL for daily projection and lead time calculations depends on how complex they are and how often they change.

In the example, the BBL can be expected to change relatively frequently in order to "fine tune" the algorithm to maximize cost savings and customer satisfaction. These two factors can be visualized as the two axes of a chart, as follows:
The chart shows a linear relationship between cost and customer satisfaction. It assumes a dollar of cost increase produces a dollar of increased customer satisfaction. A competitive, adaptive business strives to always make this curve more shallow, to get more than a dollar of satisfaction from a dollar of expense. In fact, that is practically the definition of "profit". "Fine tuning" business algorithms such as the inventory re-order calculation is an important way to achieve these gains. The changes which may be considered in the BBL to fine tune the algorithm may include:
  • Changing the "reorder type" options, for example by adding a bi-weekly or bi-monthly reorder type.
  • Changing the "importance" thresholds for allowing re-ordering of products with greater or lesser frequency.
  • Adding additional "importance" thresholds for determining the percentage of projected sales to be applied when comparing to current inventory, and changing those percentages for different importance categories.
Imagine how much more nimble a business can be if it can change these features of the algorithm without software development.

It's true that a conventional software application could be written to allow an analyst to modify these kinds of parameters. I myself have written a number of programs like that. BBL, however, allows greater flexibility as to how the algorithms can be modified without reprogramming. Also, the interfaces to such "configuration" tools, written by different teams over some period of years, tend to be inconsistent, leading to potential confusion and longer learning curves for users. BBL offers the possibility of a single, consistent, and highly flexible platform for development of logic that can be expressed as a flat, functional calculation, plus it facilitates use of the same logic on different software platforms and for different purposes, and migration to new platforms.

Not everything can be expressed in the BBL language, but a lot can be. By using BBL to express and maintain business logic that needs to adapt to changing conditions, an organization can achieve more rapid and more effective response to pressures and opportunities.

Saturday, April 23, 2016

Expenditure Approval Example

The Problem

Businesses and organizations live or die on the efficiency and effectiveness of their processes, and none is more important than procedures to control of expenditures. Improperly authorized spending can bankrupt an organization, but controls that are too burdensome can result in lost opportunities. It's one of the more difficult balancing acts faced by modern organizations in the face of sophisticated fraud schemes and a fast changing world. As a result, expenditure controls are often complex and often need to be updated on a regular basis to meet new threats while also responding to changing needs. For these reasons, expenditure approval procedures present a good case for the use of techniques I'm promoting via Blockly for Business Logic (BBL).

USAF/CAP Example

For my example, I found an online description of expenditure controls used the the United States Air Force Auxiliary Civil Air Patrol in this PDF and implemented the procedure in BBL. The main output is a boolean calculation I've called "expenditure approved" which is true if the expenditure is approved and false if it's not, which looks like this:

Blockly has a built-in feature for adding comments, which look like this:

In Blockly, you add comments by right clicking on a block, which creates a question mark in a blue button on the block. Clicking on the button opens a box for entering the comment. You can hide or show the comment by clicking on the button.

As this example, shows, there are two ways to approve an expenditure. Non-recurring expenditures have a complex approval process, while recurring expenditures are more or less automatic.

The following screen shot looks more closely at the non-recurring approval process:

The green blocks are inputs to this calculation. The first three are boolean inputs that indicate whether the request is documented, whether it's within guidelines, and whether documents have been reviewed. These inputs are generated outside of this logic and may be as simple as having the requester click on check boxes indicating they have been done, or they may involve more complex validations. 

The word "is" in parenthesis is the "type" of the input, and "is" indicates boolean. This is intended to be read by non-programmers, so the use of technical terms like "boolean" is avoided.

There are two forms of approval allowed: written, or online signature using Sertifi. 

Calculated Inputs

Next we look at some inputs that are calculated using sub-modules.


These three inputs are actually outputs from sub-modules. Within this module, though, they appear simply as inputs. BBL is able to figure out which module definitions depend on which other module definitions and perform the source calculations before running the higher-level dependent calculations. Recursion is not allowed. You can't call a module from itself or from any of its sub-modules. It could be implemented, of course, but the whole point is to create a simple way of expressing logic that works for perhaps 80% of business logic and is usable by non-programmers. More complex logic would be implemented outside of this context, either during generation of inputs to this process (preferable, I think) or by implementing custom-designed operations as blocks.

Unlike most programming languages, sub-modules are called without the use of parameters. Again, the point is to reduce complexity for use by non-programmers. All inputs to all modules are global. Because the process is "functional", there are never any local variables, so name conflicts between global inputs and local variables are never a problem. 

Here is the definition of the "approver approved" sub-module:


It checks for individual approver limits. In this simplified example, there is only one approver. If there were a small number of approvers, they might be handled as shown here, though for a large number of approvers it would be handled in a database outside the BBL mechanism and the approver limit would be passed as an input instead of the identity of the approver.

This is the definition of the "level approved" sub-module:

And this is the definition of the "sertifi approved" sub-module:


Unit Tests

When a module is defined, a template for creating unit test cases is automatically generated. In the case of the main module, it looks like this:

Test cases are set up by filling out the controls for a test and saving them. When the tests are run, the output is shown, and if it matches the "expect" value, a green check mark shows, otherwise a red circled X.

These are unit tests, not integration tests, so the inputs that come from sub-modules are not calculated, just entered. Likewise, the inputs used by sub-modules but not used by the main module are not included in these tests. Integration testing will also be required, but that is a separate process.

Generated Code

The immediate result of creating a valid definition is an XML representation of the calculation, for example:

  <output name="expenditure_approved" type="is">
    <conjunction operator="any">
      <conjunction operator="all">
        <input type="is">documented</input>
        <input type="is">within_guidelines</input>
      </conjunction>
      <conjunction operator="all">
        <selection name="form_of_approval">recurring_approval</selection>
        <input type="is">recurring_approved</input>
        <input type="is">documents_reviewed</input>
        <conjunction operator="any">
          <selection name="form_of_approval">written_approval</selection>
          <selection name="form_of_approval">sertifi_approval</selection>
        </conjunction>
        <input type="is">approver_approved</input>
        <input type="is">level_approved</input>
        <input type="is">sertifi_approved</input>
      </conjunction>
      <conjunction operator="all">
        <selection name="form_of_approval">recurring_approval</selection>
        <input type="is">recurring_approved</input>
      </conjunction>
    </conjunction>
  </output>

This XML can automatically be translated into any programming language. For purposes of running the unit tests, it's translated into JavaScript, which looks like the following. This code is somewhat difficult to understand because it's generated in a "functional" style, as one long expression. As I said in a previous post, the functional style has great advantages in terms of being able to validate correctness and for other reasons, but is difficult to work with if you're not using something like Blockly.

expenditure_approved:function(inputs)
   {
      return 
         (
            (
               (
                  (
                     validate(
                        "is", 
                        inputs.documented
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.within_guidelines
                     )
                  )
               )
            ) || 
            (
               (
                  (
                     (
                        validate(
                           "form_of_approval",
                           inputs.form_of_approval
                        ) === "recurring_approval"
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.recurring_approved
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.documents_reviewed
                     )
                  ) && 
                  (
                     (
                        (
                           (
                              validate(
                                 "form_of_approval",
                                 inputs.form_of_approval
                              ) ===  "written_approval"
                           )
                        ) || 
                        (
                           (
                              validate(
                                 "form_of_approval",
                                 inputs.form_of_approval
                              ) ===  "sertifi_approval"
                           )
                        )
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.approver_approved
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.level_approved
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.sertifi_approved
                     )
                  )
               )
            ) || 
            (
               (
                  (
                     (
                        validate(
                           "form_of_approval",
                           inputs.form_of_approval
                        ) ===  "recurring_approval"
                     )
                  ) && 
                  (
                     validate(
                        "is", 
                        inputs.recurring_approved
                     )
                  )
               )
            )
         )
   }

How It's Used

The ultimate question is, how is this used? It may be used in a variety of different ways. It can be used to generate the validations for a web form input, for example, or it can be used to validate a batch file containing expenditure records. It can be used to create a server application that validates via a server protocol. It can be used in multiple ways at the same time. If it's used to generate code for one platform, you can more easily switch to using a different platform, without having to pick the business logic out of the old code and rewrite it in new code.

The biggest advantage, though, is that the logic can be written by someone who is not a programmer, or at least by someone who has only "light" programming skills. Even more important, the logic can be READ by someone who is not a programmer. This allows the code to be "self documenting" and is important for software auditing. 

The Bottom Line

All organizations do battle with the two-headed monster of control versus responsiveness. Yes, an organization needs to maintain tight controls and yes, the organization needs to be flexible and responsive. Doing both things at once requires getting more control over the expression of business logic within the organization, and something like the BBL approach seems to offer a way forward.

Tuesday, April 19, 2016

Wash. Rinse. Repeat.

For many years, bottles of shampoo contained these instructions:
  1. Wash.
  2. Rinse.
  3. Repeat.
This is an example of what is called, in computer science, a "loop". You do something over and over again. Hopefully, at some point, you stop doing it. For example, these might be the instructions for adding up a list of numbers:
  1. If there are no more numbers, quit.
  2. Get the next number.
  3. Add it to the total.
  4. Go to step 1.
This and the shampoo instructions are examples of "procedural" programs. They tell you how to do something. There's another kind of programming, though, called "declarative" programming. With declarative programming, you say what you want done, but not how to do it.

For example, the shampoo instructions might say:
  1. Wash your hair.
This implies that the shampoo user knows that you need to rinse the soap out and possibly repeat the process a certain number of times.

In the case of a computer program, the "declarative" version might be:
  1. Sum these numbers.
It's not difficult to write a programming language that knows what to do when you give it a declarative instruction like this. I worked for 35 years programming in APL, a computer language invented in the 1960s that is largely declarative. The APL instruction for "sum these numbers" is:

      +/NUMBERS
If you've used computer spreadsheets like Excel, you're probably familiar with its declarative command for summing a list of numbers:

      SUM(A1:A5)

Most computers, deep down inside, still need to go through the list of numbers one at a time to add them up, but it's easy to write computer programs that accept declarative instructions like these and are smart enough to fill in all the steps necessary to produce the result.  In short, with declarative programming, you don't need to write "loops". The computer system knows how to do the loops for you. The loops are "implied" rather than "explicit".

Some years ago I had an assignment to convert an actuarial accounting system from RPG to a more modern programming language. RPG is an IBM programming language invented in 1959 and still used today, though in a much evolved form. In its original form, RPG was highly declarative and the looping was implied. At its simplest, you wrote a program that operated on one input record and produced one output record. RPG knew how to run this program over and over again on a lot of different input records to produce a lot of different output records. You didn't have to write this loop yourself.

I realized as I worked on this project that the RPG model of programming involved a powerful simplification. Of necessity, each step of a complex process was broken down into small bits. Each bit took in a record from an input file, did a bit of work, and produced record in an output file that was passed on as input to the next step in the process. RPG worked this way because it was designed to work efficiently on the very limited computers available at the time, but it also had the benefit of making it much easier to create correct programs. A very complex process, broken down in this way, became very manageable. It was something like "mass production" applied to programming. Unfortunately, as computers became more powerful, we started creating programs that weren't so simple. They ended up more artisanal than industrial, and correctness and maintainability suffered. I think we need to revisit the industrial model for maintaining business logic.

One of the most successful declarative languages is SQL, used to retrieve and manipulate data in relational databases. Declarative languages have the same difficulty as functional programming, though, in that they often end up having long "one-line" expressions that can be difficult to work with. The following is a typical expression from SQL:

SELECT DISTINCT Pnumber FROM PROJECT WHERE Pnumber IN   ( SELECT Pnumber FROM PROJECT, DEPARTMENT, EMPLOYEE WHERE Dnum=Dnumber AND Mgr_ssn=Ssn AND Lname=‘Smith’ ) OR Pnumber IN ( SELECT Pno FROM WORKS_ON, EMPLOYEE WHERE Essn=Ssn AND Lname=‘Smith’ );
These "one-liners" can be broken down and indented to make them more readable, but they're still difficult to work with:

SELECT DISTINCT Pnumber
FROM PROJECT
WHERE Pnumber IN   (
      SELECT Pnumber
      FROM PROJECT, DEPARTMENT, EMPLOYEE
      WHERE Dnum=Dnumber AND Mgr_ssn=Ssn AND Lname=‘Smith’ )
OR Pnumber IN (
      SELECT Pno
      FROM WORKS_ON, EMPLOYEE
      WHERE Essn=Ssn AND Lname=‘Smith’ );

Writing and testing SQL statements like this is a huge pain. Blockly visual programming, though, is just the thing for these kinds of expressions. Sure enough, some folks at a German firm have created a Blockly SQL editor. You can see a demo at this YouTube video.

The Blockly for Business Logic (BBL) idea I'm developing uses functional programming to process a single record. It relies on an external process to feed it records and handle results. The inputs would most likely be generated using SQL, and Blockly should be used to define the SQL-derived data inputs.

I've been working on developing examples of what BBL would look like when applied to various business requirements, which I'll share in future posts.

Wednesday, April 6, 2016

Flat World

I love people who believe the world is flat, I really do. They are simple people who live much simpler and happier lives than those of us who "watch the sun going down and see the world spinning round". Flat worlders believe that the flat world rides on the back of a turtle, and if you ask them what the turtle is on, they'll tell you it's another turtle. Turtles all the way down. 

Flat is good, turtles or not. I look at a map, I want to know how to get to Toledo--it's I-75 the whole way. Sweet.

If it's Toledo, Ohio. If it's Toledo, Spain, I need to use a "great circle" mapper that shows a curving arc that my jet will fly from Sarasota, Florida to Madrid airport, because the earth, so I'm told, "she's a-round".

But if I don't need to go to Spain, I-75 will do just fine for me. Why use round when flat will do? You're just making life difficult for yourself. You'll end up wasting a lot of time and you'll miss Monday Night Football.

Software development is like that. Sometimes you need to make round software, no getting around it (!?), but most of the time, flat will do nicely and it's SO much simpler.

What, you may ask, is flat software?

If you ask computer scientists, you'll get different answers, but to me, flat software is software that takes flat inputs and turns them into flat outputs.

OK, so what are flat inputs and what are flat outputs?

Here's a flat input:

Cash Client  Security Type  Transaction Date 
Yes          Stocks         June 7, 1995


Here's an input that's not flat:

Cash Client  Security Type  Transaction Date 
Yes          Stocks, Bonds  June 6-7, 1995

The not-flat input has more than one of each kind of thing.

To work with not-flat inputs or outputs, you need to repeat some part of the program more than once. In programmer-speak, you need to use a "loop" (or at least "recursion"). With flat inputs and outputs, though, whatever you do, you only need to do it once. Computer scientists have a special word for programs like this. It's called "functional programming". 

This is not really new. I'm old enough to remember when almost all programs were flat programs. The inputs were on punch cards. They weren't just conceptually flat, they were ACTUALLY flat. If there were 20 steps in your program, you wrote 20 different programs. Each step took in a punch card, processed it, and punched a new card as an output to that intermediate step. Then you took that card and fed it into the card read with the program for the second step, and it processed that and punched another intermediate result. And so on. This was "functional programming" though we didn't know it at the time.

As computer hardware and software got more powerful, we got away from this model of programming and things got more complicated. I have often wished that we could get back to this simpler "flat world" model where it was possible to be much more certain about what the programs did.

Writing programs that only do each thing once is a HUGE simplification. The Wikipedia article on "functional programming" says that it "makes it much easier to understand and predict the behavior of a program".

But wait, what if you need to go to Spain, uh, I mean, work with a LOT of the same kinds of things? Well, that's somebody else's problem. You only make doors. You don't make the whole car all by yourself.

It's not practical to write everything this way, but wow, if you can use functional programming for the things that change most often, you're WAY better off. I can't prove it, but I strongly suspect that 80% of the parts of software systems that change frequently can be written using functional programming.

There's a problem with functional programming though. When done using text-based languages, functional programs often end up being one long line of code. Some people love it, but you need to be smarter than the average bear. 

Doing it with text-based languages is harder than it needs to be. Here's the program I showed in my last post using a Blockly visual programming editor:

It has flat inputs and outputs! It's a "functional program"! But it's EASY to create using Blockly by dragging and combining colored blocks. It's easy to understand. It's easy to test. It's just EASY.

Easy is good. Just ask a Ford Motor Company door maker. He or she will tell you, "yeah, it's not rocket science". (Actually, they will tell you 这不是火箭科学.)

Bottom line: Don't do hard if you don't have to. Do easy when you can.

Mass Producing Software

Before about 1900, if you wanted to make something, you needed to know how to make the whole thing yourself. If you wanted to make a car, you had to be a car maker. Then along came mass production, and you only needed to be a door maker, so long as there were other people who knew how to make the rest of the car and you could figure out how to work together.

People who work in car factories today don't know how to make cars. They know how to make parts of cars. It's a lot easier to learn to make a door than it is to learn to make a whole car. Besides that, door makers use tools that are designed to be really good at making doors but which can't be used to make steering wheels. 

In software development, scripting is like mass production. A scripting language is a tool that can be designed to be really good at making one part of a computer program, even though it can't be used to make other parts of the program. It's also much easier to learn and much easier to use, so you don't need to be much of a genius to make really good programs.

I've created or helped develop scripting languages at various times in my career. Designing good scripting languages was difficult though. And users still needed to learn the specialized language and type in commands correctly. Then along came Blockly.

BOOM! Blockly simplifies the creating of scriping languages and makes using them 100 times easier.

Here's a calculation I created that does a simplified version of a calculation that was needed in a securities trading system I worked on at my last job, using a scripting language I created using Blockly. It calculates the number of days after a securities transaction before the securities are delivered and paid for. Even if you're not a computer programmer, you can probably look at this and understand what it does. Let's call this Way A. Give it a try.


Here's a program written in a general purpose, text based language, that does the same thing. Let's call it Way B.

settlement_days:function(inputs){
   return 
      (
      validate("boolean", inputs.cash_client) 
      || validate("security_type",inputs.security_type) 
         === "money_market" 
      || validate("security_type",inputs.security_type) 
         === "treasury_bills"
      )
      ? 0 
      : 
         lt(validate("date", inputs.transaction_date),802497600000
         ? 5 
         : 3
}

Some people love doing Way B. Let's call them WayBers. For them, there is no greater happiness than staying up all night typing computer code. WayBers make up about 1/10 of 1% of the population. Mostly they don't get invited to parties. A lot of them are still single by the time they are 35.

A lot more of us are WayAers. We want to finish working on our company's software by 5:00 so we can get home, have dinner with our families and watch the game.

There's a great effort being made just now to get WayAers to learn to write programs using Way B.

It ain't going to happen.

If you want WayAers to write software, you need to do it using Way A, which is the way we make cars and everything else. Once we figure that out, the lion will lie down with the lamb, all the people of the world will come together in harmony, and we will be mentally and spiritually at peace. Or not. But at least we will be able to get our work done so we can go home and watch the game.