See the main page for a Cress overview
The current version of Mustard is 2.0, dated 5th October 2012.
Mustard is a language and a tool for defining and validating test scenarios. Although Mustard was designed for use with Cress, it can be used independently of it. See the Cress description for an overview of Cress.
Mustard has been used to validate services in the following domains:
Mustard validates specifications/implementations using:
Mustard creates and runs tests expressed using a scenario notation. It relies on the cress_common.pm Perl module borrowed from Cress.
To run Mustard requires Perl 5 or similar. Mustard has been run on Unix (NextStep/OpenStep) and Windows (under CygWin).
A Unix-like installation is assumed in the following, though it should be possible to install and run on other platforms where Perl runs. It is assumed that the files are extracted to $HOME/bin/mustard.
In a few places in the code, a Unix-like environment is assumed. For example, Macintosh end-of-line may not be correctly handled. Paths and filenames are assumed to have '/' separators. Search paths are assumed to have ':' separators. On a Windows system, it is suggested that CygWin be used; ActivePerl may be suitable but has not been tried.
The following environment variables should be set up (e.g. in your .profile, .cshrc or Windows environment variables):
| Variable | Meaning |
|---|---|
| M4PATH (used by M4) | a colon-separated, Unix-like directory path used to locate M4 macro files, e.g. .:$HOME/bin/mustard/m4; if you are using Windows, you will not be able to cite drive letters unless you use CygWin references such as /C/home/me/bin/mustard. |
| PATH (used by command-line) | a directory path used by a shell to locate executables, e.g. to include $HOME/bin/mustard/bin |
| PERLLIB (used by Perl) | a directory path used by Perl to locate modules (cress_*.pm), e.g. $HOME/bin/cress/bin. |
| telelogic (used by Mustard) | for SDL with Unix only, Telelogic Tau directory (e.g. C:/usr/local/tau, no default) |
| TMP (used by Mustard) | a directory temporary files, /tmp by default |
| WINDIR (used by Mustard) | for SDL with Windows only, Windows system directory; used to determine if running on MS Windows (e.g. C:\Windows, no default). |
This script takes a filename on the command line. "file[.bpel|.lot|.pr]" should contain a BPEL/Lotos/SDL specification generated by Cress. The feature names in this are automatically extracted.
The main file can optionally be followed by feature or service names to restrict testing to these. A partner "feature" has the form <service>.<partner>.
For each such feature (e.g. CONFIRM), the location of its diagram is found (e.g. directory /home/kjt/bin/cress/vxml/confirm). If an XML file exists (e.g. confirm.xml), it is converted to Mustard form. The XML form of Mustard is defined by an XML schema. If an XML or converted Mustard file does not exist (e.g. confirm.mustard), tests of this feature are not performed.
If there is such a test file, it must contain tests defined in the Mustard scenario language. Tests are generated from this and combined with the original specification into a temporary file. The tests are then run automatically, with output/diagnosis of success, failure or inconclusive results.
Do not define the same test file in both XML and Mustard format, as the former will be converted into and overwrite the latter.
Command-line options are:
| Option | Meaning |
|---|---|
| -a | append to test log <system>.log (default create new test log) |
| -b memory | bit state hash memory size in MB (default 5) - Lotos |
| -d depth | maximum depth of exploration (default 100) - Lotos |
| -e level | use the given error reporting level (3 - panics, 2 - these plus errors, 1 (default) - these plus notes, 0 - these plus diagnostics) |
| -h | print help |
| -k key | key for server authorisation (BPEL, user:password@host) |
| -l library | Lotos library (default stir) - Lotos |
| -m | manual (run tests manually, default automatic) |
| -p mode[runs] | performance test: c concurrent or s sequential, optionally followed by the number of test runs (default 20) - BPEL |
| -q qualifier | qualify visible macros to disambiguate them by using the given qualifier as a prefix (e.g. 'must_') |
| -v vocabulary | use the named vocabulary (converted to lower case), default as follows: BPEL - ws; Lotos - the basic filename ignoring any '_' suffix, e.g. in.lot has vocabulary in, gs_matcher_scorer.lot has vocabulary gs; SDL - the basic filename ignoring any 'System' suffix, e.g. IVRSystem_Donation.pr has vocabulary ivr |
| -w | when waiting for a specified signal and parameters, do not allow other instances of the signal with different parameters - SDL |
For manual testing, test files are created as follows:
For automatic testing, a complete log file is created in <file>.log and test files are deleted as follows:
The script assumes certain formats for a BPEL specification (as respected by Cress):
| BPEL | Meaning |
|---|---|
| features: SERVICE1/PARTNER1,PARTNER2 SERVICE2 SERVICE3/PARTNER3 | services and associated partners |
The script assumes certain formats for a Lotos specification (as respected by Cress):
| Lotos | Meaning |
|---|---|
| features: FEATURE1 FEATURE2 | names of included features |
| features: SERVICE1/PARTNER1,PARTNER2 SERVICE2 SERVICE3/PARTNER3 | services and associated partners |
| specification SpecName [GateNames] | format of specification header |
| PutStatus(Feature,Number,...,StatusResult(Value)) | profile feature and details |
| process TestXXX | format of test process header |
The script assumes certain formats for an SDL specification (as respected by Cress):
| SDL | Meaning |
|---|---|
| features: FEATURE1 FEATURE2 | names of included features |
| EndSystem SpecName; | format of system specification end |
| StatusXXX((. Feature,Number,... .)) := Value | profile feature and details |
Mustard literals are as follows:
| Mustard | Meaning |
| ?Type | any value |
| list(type,value,...) | array |
| true, false | booleans |
| £7, L13.25, $67.12, D25.00 | currency ('£'/'$' converted to 'L'/'D' internally) |
| @2005 10 04, @10-04, @04 | date ('-', '/', space removed) |
| =output/mark.dta, =merge.spss:test/merge.svd | file to be checked (expect 'output/mark.dta' and check contents against 'output/.mark.dta', expect 'merge.spss' and check contents against 'test/merge.svd') |
| 0, +9, -3.14 | number |
| operation:sort, operation:constructor(...) | operation |
| :6091, :467 000 x 7423, :801-9134 | phone ('-', '+', space removed) |
| 'What arrival date?, '45 | string (forbidden characters " $ ' ( ) , : ; ? ! [ ] _ ` | removed) |
| !x, !inputCount | variable |
The core Mustard notation is as follows:
| Mustard | Meaning |
| // text | explanatory comment (removed during translation, use '%%' for literal '//' such as in a URL) |
| « text » | quoted text (not subject to translation) |
| call(feature,scenario) | invokes behaviour of another scenario; the feature name is optional, defaulting to the current feature |
| decide(behaviour) | non-deterministic (scenario-decided) choice of alternative behaviours |
| depend(condition,behaviour,...) | behaviour depends on whether the condition holds; an optional final behaviour acts as an `otherwise' case |
| exit(behaviour) | sequential behaviour with normal termination |
| interleave(behaviour) | concurrent execution of behaviours |
| offer(behaviour) | deterministic (system-decided) choice of alternative behaviours |
| present(feature) | holds if the feature is present in the system |
| read(signal,parameters) | inputs a signal from the system; the variant Read absorbs other signals before the desired one is input |
| refuse(behaviour) | sequential behaviour with abrupt termination if the final behaviour occurs, or successful termination if not |
| send(signal,parameters) | outputs a signal to the system; the variant Send absorbs other signals before the desired one is output |
| sequence(behaviour) | sequential behaviour with abrupt termination |
| succeed(behaviour) | sequential behaviour with successful termination |
| test(name,behaviour) | defines a test for the given name and behaviour |
A few examples are provided with Mustard:
A scenario has a name (which is automatically qualified by the current feature). As an example, the following is a test for a SIP proxy. It defines a simple sequence of events: address 1 goes off-hook, receives dial tone, and goes on-hook. If the specification respects this sequence, a pass verdict is recorded for the scenario.
test(No_Dial, test of call without dialling
succeed( successful sequence
send(OffHook,1), 1 goes off-hook
read(Announce,1,DialTone), 1 gets dial tone
send(OnHook,1))) 1 goes on-hook
The following scenario introduces a choice. After address 1 dials address 4, the outcome depends on the status of 4. If this is a valid number, it will start ringing from 1; but if it is an invalid number, then 1 will receive an 'unobtainable' message. The offer combinator allows a deterministic (system-decided) choice. This is appropriate here, as only the system knows whether the callee can be rung.
test(Ring_Or_Unobtainable, test of ring or unobtainable
succeed( successful sequence
send(OffHook,1), 1 goes off-hook
read(Announce,1,DialTone), 1 gets dial tone
send(Dial,1,4), 1 dials 4
offer( system choice
read(StartRing,4,1), 4 rings from 1
read(Announce,1,Unobtainable)))) 1 gets unobtainable
It is sometimes desirable to use the decide combinator for a non-deterministic (scenario-decided) choice. This ensures that all alternatives are explored.
The depend combinator makes a scenario conditional; the dependency is evaluated when the scenario is defined, not when it is executed. Dependencies are most commonly used when the specification has features. For example, a scenario can be varied according to the features that have been provisioned. The following tests the feature CFBL (Call Forward on Busy Line), deployed in either a user agent or a proxy. Call forwarding is handled differently in each case: a called user agent will announce a temporary change of number to the caller, while a proxy will automatically re-route the call. The following scenario depends on which feature has been deployed.
test(Busy_Forward, test of forward on busy
succeed( successful sequence
send(OffHook,1), 1 goes off-hook
read(Announce,1,DialTone), 1 gets dial tone
send(OffHook,3), 3 goes off-hook
read(Announce,3,DialTone), 3 gets dial tone
send(Dial,1,3), 1 dials 3
depend( feature dependency
present(AGENT_CFBL), agent forwarding present?
.., behaviour for agent forwarding
present(PROXY_CFBL), proxy forwarding present?
...))) behaviour for proxy forwarding
Scenarios can also be made finer-grained, depending on whether a feature applies to a particular pair of users.
Scenarios often begin in similar ways. It is undesirable if the same opening behaviour has to be repeated. Instead, opening sequences can be defined as scenarios that are called by other scenarios.
Many specification problems arise due to concurrency, for example race conditions. It is desirable to check if a specification suffers from these kinds of problems. This requires scenarios that independently execute multiple behaviours in parallel (i.e. through interleaving). In the following example, addresses 1 and 2 concurrently go off-hook and dial address 3. The outcome depends on the system, so there is a system-defined choice: 3 starts ringing from 1, and 2 receives busy; or 3 starts ringing from 2, and 1 receives busy. Note how sequence is used to group the interleaved and alternative behaviours.
test(Raced_Calls, test of simultaneous calls
succeed( successful sequence
interleave( concurrent behaviour
sequence( sequence
send(OffHook,1), 1 goes off-hook
read(Announce,1,DialTone), 1 gets dial tone
send(Dial,1,3)), 1 dials 3
sequence( plus sequence
send(OffHook,2), 2 goes off-hook
read(Announce,2,DialTone), 2 gets dial tone
send(Dial,2,3))), 2 dials 3
offer( system choice
sequence( sequence
read(StartRing,3,1), 3 rings from 1
read(Announce,2,BusyHere)), 2 gets busy
sequence( or sequence
read(StartRing,3,2), 3 rings from 1
read(Announce,1,BusyHere))))) 1 gets busy
Refusal tests express what a system must not do. Since the purpose of call screening is to block calls, it might be preferable to state this explicitly in a refusal test. If it is known that address 2 screens calls from address 1, then calls from 1 must not ring 2. The behaviour to be refused is the last parameter of refuse, here the act of ringing 2 from 1.
test(Screen_Caller_Refusal, test of caller refusal
refuse( refusal sequence
Send(OffHook,1), 1 eventually goes off-hook
read(Announce,1,DialTone), 1 gets dial tone
send(Dial,1,2), 1 dials 2
Read(StartRing,2,1))) 2 must eventually not ring from 1 (refusal)
This program is free software. You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation - either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful but without any warranty, without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.
You may re-distribute this software provided you preserve this Mustard description. Bug reports should be sent to Ken Turner, who would also appreciate receiving any corrections and comments.
updated for use with Cress 3.4
added list macro for declaring arrays
enhanced struct_of macro for nested structures
adapted for use with Cress 3.6
added -l option for selecting the Lotos library (stir is supplied with Cress not Mustard; use mod-is for general specifications)
Lotos translation now allows succeed/refuse within interleave by treating OK; Stop as Exit
Lotos translation has had small layout changes
Lotos translation no longer adds Exit to a branch if it ends with Stop
Lotos translation no longer adds Exit at the end of a decide branch
Lotos test translation now has Exit functionality as appropriate
The inputs construct has been introduced to allow Read/Send to be properly implemented for GS and WS, and to allow read/send to be properly identified in diagnostics. inputs also allows unspecified values to be defined on reading.
For read/Read, any value may now be specified by prefixing the type with '?'. For example:
read(broker.car.arrange,?Schedule) means any schedule response
read(broker.car.arrange,Schedule(0,?Text,20000.,10,?Number)) means a schedule response with any dealer and rate
read(broker.car.arrange,Refusal,?Text) means a refusal fault with any message.
Associated with this, SomeXXX constants are now generated in Lotos to match these unspecified values.
Unspecified values for a top-level type can be given names as variables, e.g. read(broker.car.arrange,?schedule:Schedule). The values of such variables are accessed later in a test by prefixing their names with '!', e.g. !schedule.
For nested structure types, e.g. Schedule(Address(Street(...),...),...), fields are accessed as !schedule.address.street.
The Lotos test translation now allows repeated Read/Send for the same primitive.
An XML translation has been introduced for testing of BPEL processes.
The Mustard file suffix has changed from '.mst' to '.mstd' (to avoid confusion with Microsoft Installer files).
An ambiguous value to be sent or read can now be qualified by its type (e.g. <> : Analysis). This type information is translated as appropriate for the target language: removed for BPEL, Of Analysis for Lotos.
An overloaded operation name can now be qualified with a '/' and its underlying constructor name (e.g. proposalLender/proposal). This information is translated as appropriate for the target language: operation name for BPEL (e.g. proposalLender), constructor name for Lotos (e.g. proposal). Qualification is also allowed in the top-level value for a read or Read, e.g. ?proposalLender/proposal or ?prop:proposalLender/proposal.
The Lotos test translation now removes leading/trailing zeros from floating point numbers. A variable called offer can now be used despite the Mustard macro of the same name.
Support has been added for BPEL validation through Mint, along with the corresponding -j and -p options.
Support has been added for validation of grid/web service partners on their own.
For GS and WS, a read, Read, send or Send can now qualify an operation name with a '/' and the associated EPR (EndPoint Reference). This a variable value that is read early in the test, e.g. read(factory.job.allocate,?some_epr:Reference) followed by send(mapper.job.translate/some_epr,'Nurse).
The sndplay.exe utility for Windows is now provided in the bin directory. The success and failure sound files are now supplied in the supp directory.
Since Offer is used in a number of Cress examples, it risks being interpreted as the Mustard macro offer. This is now handled correctly.
The list macro now works correctly.
A value can now be parenthesised to prevent an already-translated value from being translated again (e.g. in the argument of send or read).
All internal macros have now been prefixed with 'must_'. Visible macros can now optionally be given a prefix using the '-q' option.
The names of constants have been capitalised in the code for clarity, and for consistency with Cress 4.3.
A DS (Device Services) domain has been added.
For DS/GS/WS, an operation name can be suffixed with ':' to mean that its sort should be disambiguated (with "Of Operation" in Lotos).
For Lotos, a check is now made that the first element of a list is a (type) identifier.
For Lotos, an endpoint port called in is changed to inp.
For Lotos, the SomeValues is generated even if the first feature does not have any tests.
The Mustard suffix is now '.mustard', the Mustard comment prefix is now '//', and the Mustard quotes are now '{{...}}' (for consistency with Clove).
Mustard has been aligned with Mint 1.2.
When translating Mustard for BPEL, the type for the list macro is no longer used.
The test macro on its own is translated literally as 'test'.
For BPEL, the default vocabulary is now ws.
For BPEL, the list macro now 'wraps' the type name around the value list in the case of GS.
For Lotos, if the test depth is exceeded the error 'Too shallow' is reported.
For Lotos, '#', '^' are now allowed in strings.
macro quotes are now guillemets (« ... ») to ease use of {...} in regular expressions
For Lotos, '^' is again disallowed in a string as it is reserved for a space. '=' has again been allowed.
since the comment prefix '//' may be required in a URL, '%%' may be used instead
the SS (Statistics Service) domain is now supported
a configuration file is now not permitted for validation
failure before creating a tool log file is now handled cleanly
shell 'here documents' now have 'EOF' starting at the beginning of a line
the -k option has been added to define a server authorisation key for BPEL
no attempt is now made to play a sound file unless the OS is 'CygWin'
diagram paths can now contain '.'
the operating system version is now obtained from $OSNAME
an empty string ' is now translated into Lotos as <> Of Text
when calling Mint, Mustard now sets the -r option for the root directory of the main service diagram