Reporting

Test reporting is critical for DevOps. With so many releases and permutations, you’re probably sitting on a mountain of data. And that makes it hard to focus on what’s important — plus it slows you down.

Test Maker comes with powerful built-in reports and enable the developer to add their own reporters.

Built-in reporters in Test Maker:

• Allure Reporter
• Cucumber Reporter
• Excel Reporter
• Json Reporter
• Microsoft Teams Reporter
• Terminal Reporter
• Text Reporter

Test Maker is detailed and offer surgical information about the whole process like:

• Screenshots
• Video Recording
• Features Executed
• Scenarios Executed
• Steps Executed
• Sub Steps Executed
• Client Information
• Duration of each Step and whole task
• Number of Features
• Number of Passed Features
• Number of Failed Features
• Number of Skipped Features
• Detailed Error log that shows:
• Stack Trace
• Original Source code of error
• Original Source code of error Line Number
• Original Source code of error Column Number
• Original Source code of error in Spec Line Number
• Original Source code of error in Spec Column Number

By default Test Maker has the Terminal Reporter enabled but developer can chose to enable multiple reporters at the same time.

The following command enables both Terminal and Allure reporters.

This is how the same thing can be done through the configuration file.

Name	Type	Description	Default Value
error	object	Options for the order and the level of your error reporting.
hud	object	A hud visible in the browser window displaying info related to your test execution.
nativeDialogsHint	object	Options for a native dialog hint to be displayed in your browser window when a native dialog has been trigered in your test.
notification	object	Options for notifications to appear on your desktop at different step of your test execution.
path	string	Options for the path of your generated reports.	"./dist/reports"
reporters	object/string/ TestMakerReporter	Options for the different reporters that you want to use either built-in reporters or custom.	["terminal"]
screenshots	object	Options for your screenchots.
videos	object	Options for your videos.

error

Name	Type	Description	Default Value
allCodeFrames	boolean	Determines whether all code frames should be included in the error reporting.	true
codeFrameOrder	"topToBottom" | "bottomToTop" | undefined	Specifies the order in which code frames should be displayed.	"topToBottom"
stackTraceOrder	"topToBottom" | "bottomToTop" | undefined	Specifies the order in which stack traces should be displayed.	"topToBottom"
stackFilter	(params: { errorInfo?: ErrorInfo; stackFrames: CodeStackFrame[]; }) => Promise<CodeStackFrame[]> | undefined	A function to filter the stack frames based on certain conditions.	undefined

Details:

• allCodeFrames: Determines whether all code frames should be included in the error reporting. By default, it's set to true.
• codeFrameOrder: Specifies the order in which code frames should be displayed, either from top to bottom, bottom to top, or left as undefined.
• stackTraceOrder: Specifies the order in which stack traces should be displayed, either from top to bottom, bottom to top, or left as undefined.
• stackFilter: A function that can be used to filter the stack frames based on certain conditions. This function takes an object parameter with optional errorInfo and required stackFrames, and returns a Promise of filtered CodeStackFrame array. By default, it's undefined.

hud

Name	Type	Description	Default Value
enabled	boolean	Enable the presence of a test detailing hud during test execution .	false
opacity	number	Determine the opacity degree of the hud.	1

Test Maker HUD

nativeDialogsHint

Name	Type	Description	Default Value
alert	boolean	If true, Test Maker simulates an alert dialog by showing you a pop-up inside the browser window when an alert dialog is triggered.	false
closeTimeOut	number		4000
confirm	boolean	If true, Test Maker simulates a confirm dialog by showing you a pop-up inside the browser window when an alert dialog is triggered.	false
onbeforeunload	boolean	If true, Test Maker simulates an onbeforeunload dialog by showing you a pop-up inside the browser window when an alert dialog is triggered.	false
position	string	Define the location on the page of the hint pop-up.	top
prompt	boolean	If true, Test Maker simulates an onbeforeunload dialog by showing you a pop-up inside the browser window when a prompt dialog is triggered.	false

notification

Name	Type	Description	Default Value
onFeatureStart	boolean	If true, triggers a desktop notification upon reaching the start of a feature.	false
onScenarioDone	boolean	If true, triggers a desktop notification upon reaching the end of a scenario.	false
onScenarioStart	boolean	If true, triggers a desktop notification upon reaching the start of a scenario.	false
onStepDone	boolean	If true, triggers a desktop notification upon reaching the end of a step.	false
onStepStart	boolean	If true, triggers a desktop notification upon reaching the start of a step.	false
onTaskDone	boolean	If true, triggers a desktop notification upon reaching the end of a task.	false
onTaskStart	boolean	If true, triggers a desktop notification upon reaching the start of a task.	false
onFeatureDone	boolean	If true, triggers a desktop notification upon reaching the end of a feature.	false

reporters

Name	Type	Description	Default Value
name	string	Specify the name of a reporter.
options	undefined	Define the options of a reporter.

AllureReporterOptions

Name	Type	Description	Default Value
scoreBase	string	Specify the base for scoring (e.g., 'feature', 'scenario').	'feature'
dateFormat	string	Set the date format for displaying dates in the report.	'DD/MM/YYYY'
resultsFolder	string	Set the folder path for storing individual test results.	undefined
reportFolder	string	Set the folder path for storing the generated Allure report.	undefined
packResultsFiles	string	Specify the path for packing individual result files into a ZIP archive.	undefined
packReportFiles	string	Specify the path for packing the generated report files into a ZIP archive.	undefined
generateReport	boolean	Enable or disable the generation of the Allure report.	true
keepHistory	boolean	Enable or disable keeping a history of test results.	true
historyFolder	string	Set the folder path for storing historical test results.	'Allure'
open	boolean	Automatically open the generated report in the default web browser.	true
portable	boolean or { pack: boolean; platforms: ('windows' | 'mac' | 'linux')[]; }	Create a portable version of the report that can be run on different platforms.	undefined

Basic Usage

[
 {
                ...reporters.allure,
                ...{
                    options: <AllureReporterOptions>{
                        scoreBase: 'feature',
                        dateFormat: 'DD/MM/YYYY',
                        resultsFolder: './dist/allure-results',
                        reportFolder: './dist/allure-report',
                        packResultsFiles: './dist/test_results.zip',
                        packReportFiles: './dist/test_reports.zip',
                        generateReport: true,
                        keepHistory: true,
                        historyFolder: 'Allure',
                        open: true,
                        portable: {
                            pack: true,
                            platforms: ['windows', 'mac', 'linux'],
                        },
                    },
                },
            },
]

TerminalReporterOptions

Name	Type	Description	Default Value
stats	{ table?: boolean; features?: boolean; scenarios?: boolean; steps?: boolean; actions?: boolean; }	Specify which statistics to display in the terminal.	{ table: true, features: true, scenarios: true, steps: true, actions: true }
ignoreLogLevel	boolean	Ignore the log level when displaying log messages in the terminal.	false
completionCounter	boolean	Display a counter for completed actions in the terminal.	false
scoreBase	string	Specify the base for scoring (e.g., 'feature', 'scenario').	'feature'
dateFormat	string	Set the date format for displaying dates in the terminal.	'[On] dddd DD-MMM-YYYY [At] HH:mm:ss:SSS [GMT]Z'
bypass	({ target, phase, info }: TerminalReporterOptionsBypass) => Promise<boolean>	A function to customize whether to bypass specific reporting events.	undefined
feature	{ start?: boolean; done?: boolean; }	Specify whether to display messages for feature events.	{ start: true, done: true }
scenario	{ start?: boolean; done?: boolean; }	Specify whether to display messages for scenario events.	{ start: true, done: true }
step	{ start?: boolean; done?: boolean; }	Specify whether to display messages for step events.	{ start: true, done: true }
subStep	{ start?: boolean; done?: boolean; }	Specify whether to display messages for sub-step events.	{ start: false, done: true }
hook	{ start?: boolean; done?: boolean; }	Specify whether to display messages for hook events.	{ start: false, done: true }
controllerAction	{ start?: boolean; done?: boolean; source?: boolean; }	Specify whether to display messages for controller action events.	{ start: true, done: true, source: false }
selectorAction	{ start?: boolean; done?: boolean; source?: boolean; retries?: boolean; }	Specify whether to display messages for selector action events.	{ start: true, done: true, source: false, retries: false }
assertionAction	{ start?: boolean; done?: boolean; source?: boolean; retries?: boolean; }	Specify whether to display messages for assertion action events.	{ start: false, done: true, source: false, retries: false }

Basic Usage

[
 {
                ...reporters.terminal,
                ...{
                    options: <TerminalReporterOptions>{
                        stats: {
                            table: true,
                            features: true,
                            scenarios: true,
                            steps: true,
                            actions: true,
                        },
                        ignoreLogLevel: false,
                        completionCounter: false,
                        scoreBase: 'feature',
                        dateFormat: '[On] dddd DD-MMM-YYYY [At] HH:mm:ss:SSS [GMT]Z',
                        bypass: ({ target, phase, info }) => Promise.resolve(false),
                        feature: { start: true, done: true },
                        scenario: { start: true, done: true },
                        step: { start: true, done: true },
                        subStep: { start: false, done: true },
                        hook: { start: false, done: true },
                        controllerAction: { start: true, done: true, source: false },
                        selectorAction: { start: true, done: true, source: false, retries: false },
                        assertionAction: { start: false, done: true, source: false, retries: false },
                    },
                },
            },
]

TextReporterOptions

TextReporterOptions inherits from TerminalReporterOptions.

Basic Usage

[
 {
                ...reporters.text,
                ...{
                    options: <TextReporterOptions>{
                        stats: {
                            table: true,
                            features: true,
                            scenarios: true,
                            steps: true,
                            actions: true,
                        },
                        ignoreLogLevel: false,
                        completionCounter: false,
                        scoreBase: 'feature',
                        dateFormat: '[On] dddd DD-MMM-YYYY [At] HH:mm:ss:SSS [GMT]Z',
                        bypass: ({ target, phase, info }) => Promise.resolve(false),
                        feature: { start: true, done: true },
                        scenario: { start: true, done: true },
                        step: { start: true, done: true },
                        subStep: { start: false, done: true },
                        hook: { start: false, done: true },
                        controllerAction: { start: true, done: true, source: false },
                        selectorAction: { start: true, done: true, source: false, retries: false },
                        assertionAction: { start: false, done: true, source: false, retries: false },
                    },
                },
            },
]

TestMakerReporter

Basic Usage

reporting:{
        reporters:[
            reporters.allure,
            reporters.cucumber,
            reporters.excel,
            reporters.json,
            // reporters.microsoftTeams, require options to define microsoft team group.
            reporters.terminal,
            reporters.text
        ]
    }

screenshots

Name	Type	Description	Default Value
enabled	boolean	If true, enable the taking of screenshots.	true
enabledIn	object	Object containing options that set when a screenshot has to be taken.
failedOnly	boolean	If true, screenshots will be taken and saved in case of a failure in the test execution.	true
fullPage	boolean	If true, the screenshots will display the entirety of the website page.	false
path	string	The path where you want your screenshots to be located.	"./dist/reports/assets/screenshots"
pathPattern	string		""

enabledIn

Name	Type	Description	Default Value
AfterEachFeature	boolean	If true, a screenshot is taken after the end of a feature.	true
AfterEachScenario	boolean	If true, a screenshot is taken after the end of a scenario.	true
AfterEachStep	boolean	If true, a screenshot is taken after the end of a step.	true
AfterFeature	boolean	If true, a screenshot is taken after the end of a feature or after the end if of a feature hook if defined.	true
AfterScenario	boolean	If true, a screenshot is taken at the end of a scenario or after the end if of a scenario hook if defined.	false
BeforeEachFeature	boolean	If true, a screenshot is taken just before the start of a feature.	false
BeforeEachScenario	boolean	If true, a screenshot is taken just before the start of a scenario.	false
BeforeEachStep	boolean	If true, a screenshot is taken just before the start of a step.	false
BeforeFeature	boolean	If true, a screenshot is taken just before the start of a feature or of a feature hook if defined.	false
BeforeScenario	boolean	If true, a screenshot is taken just before the start of a scenario or of a scenario hook if defined.	false
Feature	boolean	If true, a screenshot is taken at the end of a feature.	false
Scenario	boolean	If true, a screenshot is taken at the end of a scenario.	false
Step	boolean	If true, a screenshot is taken at the end of a step.	false

video

Name	Type	Description	Default Value
enabled	boolean	If true, enable the recording of videos.	false
failedOnly	boolean	If true, videos will be recorded in case of a failure in the test execution.	true
ffmpeg	object
path	string	The path where you want your videos to be located.	"./dist/reports/assets/videos"
pathPattern	string
singleFile	boolean		false
size	object	Object defining the video format.

ffmpeg

Name	Type	Description	Default Value
encodingOptions	object	Define the encoding option of the recorded videos.
path	string		""

encodingOptions

size

Name	Type	Description	Default Value
height	number	Represents the height parameter of the recorded videos	768
width	number	Represents the width parameter of the recorded videos	1024