Evidence metrics in the Regulator Performance Framework

Performance indicator evidence metrics

ARPANSA publishes reports on the outcomes of annual self-assessments and any external reviews of its performance. These reports identify the extent to which ARPANSA is achieving its performance against the indicators in the Framework and highlight areas for improvement. ARPANSA uses the following metrics to measure our performance in self-assessments from 1 July 2017. We developed these indicators based on the framework and our experience with the previous indicators used from 2015 to 2017.

No. Summary Evidence metric
Key Performance Indicator 1: 
Regulators do not unnecessarily impede the efficient operation of regulated entities
1.1 Schedule adherence

Inspections are conducted in accordance with established inspection schedule.

Eight inspection areas have been identified for licensed facilities or sources to be undertaken at least once during a three year facility and six year source inspection cycle. The inspection schedule is updated annually and communicated to licence holders. Adherence to the schedule promotes trust, predictability and efficiency. It supports ARPANSA’s efforts to deliver quality regulatory services with due consideration of the health and safety of people and the environment, without being disruptive to business.

ARPANSA will measure the percentage of inspections conducted in accordance with the risk-informed long term inspection schedule [Quantitative].

1.2 Assessment timeliness

Applications are assessed within agreed time frames.

ARPANSA reviews and takes decisions in relation to applications for new licences, amendment of licences, and for other special approvals under the ARPANS Regulations. The complexity of applications varies widely. Using a risk-informed approach and in consultation with licence applicants or licence holders, regulatory staff will prioritise resources and establish a date for completion of the application assessment. This assists the licence holder in planning, avoiding impediments to business.

ARPANSA will measure the percentage of applications which are assessed within this agreed timeframe [Quantitative].

Key Performance Indicator 2: 
Communication with regulated entities is clear, targeted and effective
2.1 Information sharing

Information sharing meetings are held with licence holders

Meetings are held with licence holders to exchange information on regulatory matters such as upcoming legislative changes, new or amended regulatory guides, licence applications, or licensing and compliance issues. Examples of such established forums are the Defence-ARPANSA Liaison Forum (DALF) and ARPANSA Licence Holder Forums. Regular information exchange on regulatory matters reduces uncertainties about regulatory expectations, ultimately improving compliance rates and removing unnecessary regulatory burden.

ARPANSA will measure the number of meetings held in a year [Quantitative], and evaluate the quality, including any feedback received from these meetings, to determine if the communication is effective [Qualitative].

Key Performance Indicator 3: 
Actions undertaken by regulators are proportionate to the regulatory risk being managed
3.1 Risk-based schedule

Inspection Schedule is risk informed and reviewed annually.

ARPANSA applies a graded, risk informed, approach to its inspection program. This measure indicates ARPANSA’s ability to apply its regulatory resources efficiently and proportionately where needed.

For source licences a six year baseline inspection program based on the source (hazard) category is used. For efficiency, the inspection program also takes account of the geographical location of the sources so that regions with multiple sources may be inspected together.

Facility licences are reviewed annually and after any significant regulatory events such as non-compliance, incidents and accidents. The structured review takes account of the inherent risks presented by the facility, the level of safety controls and the performance of the operator. The outcome of this review is used to determine the overall proportion of regulatory resources applied to each facility.

This balanced approach recognises good performance with reduced regulatory effort and allows regulatory resources to be applied where they have most safety benefit.

ARPANSA will measure the conformance to this scheduling and risk review processes [Qualitative].

3.2 Graded approach

A graded approach is applied to compliance monitoring and enforcement actions.

ARPANSA applies a graded, risk informed, approach to compliance monitoring and non-compliance in accordance with published policies. When potential non-compliance is identified, the regulatory response is commensurate with its significance and ARPANSA will use the minimum response necessary to achieve the desired result, which, in most cases will be a return to compliance. Good safety performance characterised by a good safety (and security) culture will be taken into account when determining the response. The baseline inspection schedule is supplemented by augmented inspections where there are performance concerns.

ARPANSA will measure conformance with the policies for inspection outcomes and non-compliance, and the general proportion of effort applied to areas of greatest safety risk.

Key Performance Indicator 4: 
Compliance and monitoring approaches are streamlined and coordinated
4.1 Light-touch regulation

Actions are initiated within three months of the identification of an area for improvement.

When an Area For Improvement (AFI) is identified as a result of an inspection or other monitoring, there is an expectation that the licence holder will take corrective action in a timely fashion. The objective of identifying areas for improvement is to reduce regulatory burden by improving safety through a light touch, without the use of enforcement actions. Typically an AFI represents a situation that could lead to a non‑compliance or in which safety practice could be improved. As such, where licence holders voluntarily implement corrective action following the finding of an area for improvement it demonstrates good safety culture, and the effectiveness of non-enforcement actions and ARPANSA promotion of best practice.

ARPANSA will measure the percentage of time an action is initiated within 3 months of an area for improvement being formally identified to a licence holder [Quantitative].

4.2 Collaboration

Information is shared with collaborating regulatory agencies.

ARPANSA licence holders are also regulated by other regulatory agencies. ARPANSA will collaborate with other regulators, where appropriate, by the sharing of information or undertaking joint activities. The objective of collaboration is to co-ordinate work in common areas of interest so as to avoid duplication and unnecessary disruption to the licence holder, and in so doing reduce regulatory burden [Qualitative].

Key Performance Indicator 5: 
Regulators are open and transparent in their dealings with regulated entities
5.1 Transparency

ARPANSA’s risk framework, the basis for regulatory decisions, and the outcomes of compliance monitoring are published on the web.

To provide confidence to regulated entities and the wider community, ARPANSA publishes a range of reports and information on how it goes about its business. This information includes, whenever possible, the prompt publication of inspection reports and the ‘statement of reasons’ for any significant licensing decisions. Information relating to ARPANSA’s approach to risk and many of the processes used to manage the regulatory business are also published. These communication practices are part of ARPANSA’s no-surprises, evidence based approach to regulation that provide trust from licenced entities and the wider community.

Appropriate feedback from a range of interested parties requires transparency in the regulatory decision framework and decision making. ARPANSA will use its website as the primary mechanism to improve transparency [Qualitative].

5.2 Consultation

Stakeholders, including the public, are consulted on the development of codes and guidance publications.

ARPANSA publishes guides, codes and standards on a range of regulatory topics which set out expectations for ARPANSA’s licence holder with respect to safety of sources and facilities. This includes national documents such as the Regulatory Protection Series and local documents such as Regulatory guides. These documents typically reflect international best practice. ARPANSA will consult with licence holders for feedback on the development or significant amendment of guides and codes so as to improve transparency in regulation and support continuous improvement [Quantitative].

Key Performance Indicator 6: 
Regulators actively contribute to the continuous improvement of regulatory frameworks
6.1 Customer satisfaction

Feedback from licence holders is encouraged and feedback received is positive, constructive and drives improvement.

Soon after the completion of an inspection, licence application, change request (Regulation 51) or construction request (Regulation 54), the Office of the CEO will administer a survey independent of the Regulatory Services Branch, to ask for feedback on the service provided. The options for response from the licence holder range from ‘strongly agree’ to ‘strongly disagree’.

A survey score is used to trend ARPANSA’s performance [Quantitative].

The survey format provides the opportunity to add specific comments on the service provided. Feedback is an opportunity to identify improvements and enhance consistency of good practices. ARPANSA will analyse results to help gauge how effective regulatory staff is in putting the six KPIs into practice [Qualitative].

6.2 Continuous improvement

Improvements identified through internal or external reviews, self-assessment or feedback, are implemented effectively.

Areas for improvement in the regulatory framework can be identified via various routes such as annual self-assessments. Additional opportunities include internal procedure and policy reviews as part of the Regulatory Services Branch Quality Management System, staff suggestions, external audits of ARPANSA including international peer review missions, and stakeholder feedback from surveys and licence holder forums.

The number of regulatory improvements identified and implemented will measure ARPANSA’s actions to continuously improve the regulatory framework [Quantitative].

6.3 Framework development

Promote the use of international best practice across Australia.

ARPANSA’s regulatory activities should meet national and international standards of good practice. To achieve this, ARPANSA will cooperate with national and international bodies in the development of best practice radiation regulation including in the development of international standards and recommendations [Qualitative].