Previous evidence metrics in the Regulator Performance Framework

Performance indicator evidence metrics used for self-assessment 2015-17

ARPANSA is required to publish a report on the outcomes of each annual self-assessment and any external reviews of its performance. ARPANSA used the following performance indicator evidence metrics to undertake its self-assessment against the Regulator Performance Framework in the 2015-16 and 2016-17 financial years.

ARPANSA’s evidence metrics for the six KPIs in the Regulator Performance Framework
  List of evidence metrics Most Applicable KPI
 1.

Percentage of facility inspections conducted in accordance with the established inspection schedule (adherence to inspection schedule)
ARPANSA will measure the percentage of facility inspections conducted in accordance with an established long term inspection schedule.  Eight inspection areas have been identified for licensed facilities or sources to be undertaken at least once during a three-year inspection cycle. A long-term risk-informed inspection schedule is maintained for licensed facilities   and updated regularly to reflect the compliance maturity of the licence holder.  Adherence to the schedule promotes trust, predictability and efficiency, and allows the licensees to plan their work, and supports ARPANSA’s ability deliver quality regulatory services with due consideration of health of people and the environment, without being disruptive to business.

1
(Regulators do not unnecessarily impede the efficient operation of regulated entities)

2. 

Percentage of applications assessed within the time that was agreed with the licence holder (timeliness of application assessment)
ARPANSA is required to assess licence applications as well as applications to amend licences and other special approvals under the ARPANS Regulations. The complexity of applications varies widely.  Using a risk-informed approach and in consultation with licence applicants or licensees, regulatory staff will prioritise resources and establish a ‘need by’ date for completion of the application assessment, thereby assisting the licence holder with business planning and avoiding impediment to business. An application of a satisfactory standard that is approved by the agreed date is deemed to be timely.

1
(Regulators do not unnecessarily impede the efficient operation of regulated entities).

 3.

Percentage of licensee inspection feedback that is favourable (customer satisfaction)
ARPANSA issues an inspection report within 10 working days after the inspection closeout ‘exit’ meeting. Soon afterwards, the Office of the CEO will administer a survey independent of the Regulatory Services Branch, and ask for feedback on the inspection.  The options for response from the licence holder range from “strongly agree,” to “strongly disagree”. Negative feedback is an opportunity to improve systems of interaction. If the average response is neutral or better, then the feedback will be taken to be favourable. This will serve as an indicator of how effective regulatory staff are in putting the six KPIs into practice.

2
(Communication with regulated entities is clear, targeted, and effective)

 4.

The number and quality of information sharing meetings with licence holders and feedback following the meetings  (effective communication)
Meetings are held with licence holders to exchange information on regulatory matters such as upcoming legislative changes, licence applications or licensing and compliance issues. Examples of such established fora are the Defence-ARPANSA Liaison Forum (DALF) and ARPANSA Licence Holders’ Forums.  Regular information exchange meetings on regulatory matters improves licence holder understanding of regulatory expectations, ultimately improving compliance standards and reducing regulatory burden.  The number and quality of meetings in a year and the feedback from these meetings will indicate if communication is effective.

2
(Communication with regulated entities is clear, targeted, and effective)

 5.

Ratio of performance deficiencies to non-compliances during inspections (graded approach)
Potential non-compliances may arise when licence holders do not meet licence conditions or requirements of the ARPANS Act or Regulations, or any special condition issued with the licence.  In contrast, performance deficiencies may occur when a licence holder does not follow accepted international best practice or does not meet self-imposed standards.  By distinguishing between performance deficiencies and potential non-compliances, regulatory staff will adopt a risk-informed graded approach to compliance, hence reducing regulatory burden. ARPANSA will implement formal enforcement measures only when a breach has been determined by CEO ARPANSA and the licensee’s remedial actions (if any) are deemed unsatisfactory-.

3
(Actions undertaken by regulators are proportionate to the regulatory risk being managed)

 6.

Ratio of site visits (monitoring) to inspections at licensed facilities (performance monitoring)
ARPANSA inspectors monitor performance of licence holders on a regular basis outside of the inspection process through site visits.  Frequent site visits will be undertaken to observe licence holder operations. Unlike inspections, no detailed advance planning is required for site visits, and observations are shared verbally with the licence holder.  Frequent site visits improve regulatory oversight of licence holder operations, increase the visibility of the regulator/inspectors among licensees, and are expected to contribute to minimising the incidence of potential non-compliances through enhanced communication.

3
(Actions undertaken by regulators are proportionate to the regulatory risk being managed)

 7.

Percentage of inspections that take account of the risk profile of licence holders (risk informed regulation)
ARPANSA’s risk ranking methodology published on the ARPANSA website, both informs the licence holder and enables regulatory staff to gauge the risk of an undertaking based on the inherent risk of the facility or source, recent compliance history, quality of the licence holders’ procedures, and history of incidents and accidents. The risk ranking of each licence is reviewed annually and following inspections and incidents.  Establishing a transparent and planned inspection programme based on the risk ranking of the licensee’s operation ensures that ARPANSA streamlines its compliance monitoring programme as necessary to take account of any changes in licence holder risk profile, and reduces regulatory burden.

4
(Compliance and monitoring approaches are streamlined and coordinated)

 8.

Timeliness of implementation of corrective actions following the identification of a performance deficiency (light touch regulation)
When a performance deficiency is identified, either as a result of an inspection or other monitoring, there is an expectation that the licence holder will take corrective action in a timely fashion to address the performance deficiency. As the objective of identifying performance deficiencies is to reduce regulatory burden by improving compliance without formal enforcement action, the time taken for a licence holder to implement corrective actions following the finding of a performance deficiency, accompanied by communication of the timeliness expectations, is a measure of the transparency and effectiveness of the inspection/compliance monitoring programme.

4
(Compliance and monitoring approaches are streamlined and coordinated)

 9.

Percentage of branch personnel time that is devoted to direct regulatory activities (core business efficiency)
Direct regulatory activities are those which can be attributed to a particular licence holder. Activities include inspections, site visits, compliance monitoring, application assessments, enforcement activities and development of specific guidance.  As regulation is core business for ARPANSA, increased fraction of  time spent in direct regulatory activities is likely to improve ARPANSA’s understanding of licence holders operations, resulting in better compliance outcomes and transparency in its dealings with its licence holders. Carefully recording direct regulatory activities reduces regulatory burden in a transparent manner, as it is the legislated basis upon which the licence fees are based.

5
(Regulators are open and transparent in their dealings with regulated entities)

 10.

Number of instances in which licence holders are consulted on the development of guides, codes, and standards (transparent development of standards)
ARPANSA publishes guides, codes and standards on a range of regulatory topics which set out expectations for the licence holder’s dealings with respect to safety with sources and facilities.  These guides, codes and standards typically adopt international best practice and hence their adoption in Australia reduces unnecessary regulatory burden in comparison to international competitors. Consultation with licence holders for feedback on draft guides, codes and standards improves transparency in regulation and supports continuous improvement.

5
(Regulators are open and transparent in their dealings with regulated entities)

 11.

Number of improvements to how ARPANSA conducts its regulatory role (continuous improvement)
Areas for improvement in the regulatory framework can be identified via various routes such as annual self-assessments. Additional opportunities include internal procedure and policy reviews as part of the Regulatory Services Branch Quality Management System, external audits of ARPANSA including international peer review missions, stakeholder feedback from surveys and licence holder forums.  The number of regulatory improvements identified and implemented will measure ARPANSA’s actions to continuously improve the regulatory framework.

6
(Regulators actively contribute to the continuous improvement of regulatory frameworks)

 12.

Percentage of facility inspections in which expertise external to the branch was utilised (judicious use of regulatory expertise)
ARPANSA may utilise external experts in its inspection programme. Such persons may have experience in a particular field and/or regulation of particular sources or facilities. Judicious use of certain expertise promotes efficiency and improves regulatory results.  The use of external experts provides a valuable source of independent advice in specialised areas which can actively contribute to improvements in the regulatory framework. In addition, the use of external personnel when inspecting licences issued to other ARPANSA branches avoids conflict-of-interest.

6
(Regulators actively contribute to the continuous improvement of regulatory frameworks)