Below are notes from our ongoing Technical Forum meetings and calls. We do our best to document all comments so that they can be fully addressed during our development process. If you find discrepancies in our notes, please let us know. You can listen to recordings of our group calls as well using this link.
QA Technical Forum Meeting - February 27th, 2015 The ICP Technical team continued discussions regarding the ICP Quality Assurance (QA) Credential on February 27th, 2015. The Team is working to have a version 1.0 of the QA Provider Credential, as well as the associated QA Specification and QA Checklist prepared by the end of March. With this goal in mind, the discussions focused on what the QA process should involve. The following are highlights from the conversation with the Technical Forum:
On traditional projects currently, reviewers are brought on at different stages of a project, often late in a project's development, and there is much effort simply placed on finding the information necessary for the QA efforts. This information is also not usually in a standardized format, regarding calculations, reports, etc.
While the ICP does not require specific formats for information, it does standardize, through references to existing guidelines and standards, the information that needs to be collected, and that proper documentation needs to be provided.
Reasonableness - when a reviewer looks at the assumptions used in a project, what can be defined as reasonable? There was consensus that there are not many resources currently available to assist with these efforts. These resources may be developed in the future, but are not currently available. The best resource that the market currently can offer is the experience of the QA provider. While this will vary, ICP makes an attempt to bring some criteria to bear on this resource, through its QA Provider Credential Requirements (refer to notes from the meeting on 2/9/15).
Who pays for the QA efforts? - This can be a program, or the building owner, or the financier, or the developer. The fact is that these costs are already incurred by projects, and the review may even happen multiple times on a project by different parties involved. ICP works to standardize these efforts, so that it is a one-person / firm exercise, with direction regarding the expectations and the process to employ.
What happens if a QA provider and a PD cannot agree on what is reasonable? - The building owner or the financier has the final decision regarding whether a project can/should move forward. The QA provider and PD need to take a collaborative approach to the review efforts - review / comment / resolve. If a resolution cannot be managed, the project can still move forward. Documentation will be provided as a result of these QA efforts that will establish issues that were identified, and whether or not they were resolved. This will allow stakeholders to reflect on projects that did or did not perform, with documentation illustrating what was identified during the QA process, and how it was addressed or not addressed.
Fundamentally, the ICP QA process focuses on ensuring that a project complies with the ICP protocols. This is a binary process, and can be accomplished by filling out the ICP checklist. And to a certain extent, this is what financiers want to see - did the project follow the ICP process or not?
So should the QA provider be addressing the "assumptions" piece? Essentially, should the "assumptions" piece be dropped from the QA provider's role? This is an important component of the QA provider's value, and though not well defined by the industry with regards to what is reasonable, it is an important aspect of the review process. ICP can only base reasonableness on the experience and credentials of the QA providers, until the marketplace matures and better guidelines are available to provide more direction on this front.
QA Technical Forum Meeting - February 9th, 2015 The ICP Technical team continued discussions regarding the ICP Quality Assurance Credential on February 9th, 2015. The discussions involved finalizing the requirements of a credential ICP Quality Assurance Provider, review of the QA checklist, and discussion about the term "Quality Assurance" and if it is appropriate. The following are highlights from the conversation with the Technical Forum:
It was agreed by the group that the credential requirements are appropriate. The agreed-upon requirements are listed at the end of these notes.
Use of the PE - the use of a signature or stamp is interchangeable. And, this signature or stamp on an IREE project reflects the following: "The seal is not, and should not be considered, a certification mark or warranty of correctness. According to the Supreme Court (Edgeworth Construction Ltd. v. N. D. Lea & Associates Ltd.), the “seal attests that a qualified engineer prepared the document. It is not a guarantee of accuracy”. Instead, it should be considered a “mark of reliance”, an indication that others can rely on the fact that the opinions, judgments, or designs in the sealed documents were provided by a professional engineer held to high standards of knowledge, skill and ethical conduct."
Essentially, the PE's signature ensures that the project was performed by or under the supervision of the PE, that it conforms to the ICP, and that all attempts have been made to prepare the project using appropriate best-practices and methodologies, and that the assumptions and results appear reasonable. "The seal is not, and should not be considered, a certification mark or warranty of correctness”.
Checklist - it was agreed upon that the checklist draft is accurate and useful. The general sentiment is that it is thorough, and even "impressive," but not overbearing. That is, project developers typically perform all of these activities (or should do all of this, and this is agreed).
Need to incorporate a statement at the bottom of the checklist that attests to what it means for the PE to sign off on the project.
All participants agreed to the term "Quality Assurance Provider." As opposed to Quality Management or Quality Control, or something else. There are other terms for this service used in the industry, like Technical Reviewer," but in general, this term seems appropriate and applicable
Can be an individual, an independent firm, or a program
Must be a PE (if an individual) or have a PE on staff to oversee and approve all review efforts (if a firm)
Minimum of five years of relevant energy efficiency project development experience, and three years quality assurance review experience, documented in the form of a CV outlining relevant project experiences
Must provide three references demonstrating relevant project development and quality assurance review experience
Must attend the ICP Credentialed Project Developer training
Must complete the ICP Credentialed Quality Assurance Provider training
Primary responsibilities include:
Ensure that the project was developed according to the ICP Energy Performance Protocols leveraging the ICP Project Development Specification and QA Checklist
Check that methodologies, assumptions and results follow best practices and are reasonable
QA Technical Forum Meeting - January 27th, 2015 The ICP Technical Team hosted a Technical Forum call (you can catch it HERE), to discuss what it means to be a Quality Assurance (QA) Provider, what the ICP should require, as well as what requirements the ICP QA Provider Credential should include. The ICP Team will be hosting subsequent discussions on Feb 9th and Feb 27th at 11 am EST to continue these discussions, and develop "version 1.0" of this ICP QA Provider Credential, tentatively set for release and incorporation into the ICP Credentialing system by the end of March.
The following are highlights from the discussions:
The ICP QA Provider must ensure that a project was developed according to the ICP Energy Performance Protocols leveraging the ICP Project Development Specification and the QA Checklist. And, the QA Provider must check that methodologies, assumptions and results follow best practices and are reasonable.
The risk associated with a project or a measure drives the level of review detail necessary. The QA effort is all about risk mitigation.
The QA provider experience is key. If the credentialed project developer (PD) follows ICP, it will reduce review time. Standardization plays a big part in the efficiency and effectiveness of the review process.
Projects are complex - the review process is difficult to define discretely. Reviewer's experience is paramount in the process.
Many investors already require a PE; for the QA provider, ICP wants to establish a higher bar than PD. The PE license is important for QA. However, being a PE does not mean they know about energy efficiency (EE) projects. And, PE’s are more experienced but more expensive. Some large firms have lots of experience but not necessarily PE's on staff. But consensus is that the review process must be performed with oversight from a PE, however the work itself can be done by other staff, ideally with specific certifications pertaining to the areas of the project they are reviewing (CEM, BEMP, CMVP, etc.).
Make it clear that this review process should involve the PE overseeing the work, with others performing the actual review (those that have specialized skills for different EE project components)
Review associated with programs in NY, MA, NJ, CT - None of these programs require that the review is performed by a PE, but they require that it is performed with oversight by a PE (or registered architect)
Should the stamp be required? Some investors do require this. Without this requirement, many investors may push back - there is little risk associated with doing a review on the reviewer's part.
Is "signing off" by the PE enough, and how does this really differ from a PE stamp? - The QA provider is not saying that the project will perform.
Require five years of QA experience rather than PD experience? Maybe 5 years relevant EE PD experience, and 3 years QA experience?
Should ICP require certifications for the different components of an EE project review? Energy modeling, M&V, Cx, etc? This may be too restrictive, as some certifications, such as energy modeling for example, are still not well established in the industry.
Should there be difference requirements for targeted projects versus larger, more comprehensive projects?
PE STAMP:
For the public, the seal constitutes the distinctive mark of the professional engineer. It must be used to identify all work prepared by, or under the direct supervision of, a professional engineer as part of professional engineering services rendered to the public. It assures the document’s recipient that the work meets the standards of professionalism expected of competent, experienced individuals who take personal responsibility for their judgments and decisions. The seal is important because it is a visible commitment to the standards of the profession and signifies to the public that a particular P.Eng. accepted professional responsibility for the document. Affixing the seal to a document is a statement by a professional engineer to others that they can, with a high degree of confidence, depend upon the contents of the document for the furtherance of their projects. Since the outcome of a project depends on factors beyond the control of an engineer, however, a successful outcome cannot be guaranteed by an engineer. The seal is not, and should not be considered, a certification mark or warranty of correctness. According to the Supreme Court (Edgeworth Construction Ltd. v. N. D. Lea & Associates Ltd.), the “seal attests that a qualified engineer prepared the document. It is not a guarantee of accuracy”. Instead, it should be considered a “mark of reliance”, an indication that others can rely on the fact that the opinions, judgments, or designs in the sealed documents were provided by a professional engineer held to high standards of knowledge, skill and ethical conduct.
Suggestion of how to define the QA Provider requirements:
Can be an individual, an independent firm, or a program
Must be a PE (if an individual) or have a PE on staff to oversee and approve all review efforts (if a firm)
Minimum of five years of relevant energy efficiency project development experience, and three years quality assurance review experience, documented in the form of a CV outlining relevant project experiences
Must attend the ICP Credentialed Project Developer training
Must complete the ICP Credentialed Quality Assurance Provider training
Primary responsibilities include:
Ensure that the project was developed according to the ICP Energy Performance Protocols leveraging the ICP Project Development Specification and QA Checklist
Check that methodologies, assumptions and results follow best practices and are reasonable
ICP Quality Assurance Specification - Technical Meeting Recap, July 2015 Uncertainty and risk are always a consideration for any energy efficiency project, and certain areas of project development can introduce more uncertainty and risk than others. The Investor Confidence Project (ICP) is intended to provide a framework for energy efficiency (EE) project development to reduce risk and increase the reliability and consistency of energy savings. The ICP system is designed to provide a comprehensive framework of required elements that is flexible enough to accommodate a wide range of methods and resources that are specific to individual projects. The ICP Quality Assurance (QA) Specification identifies areas of an energy efficiency project that require greater levels of scrutiny and QA, as well as methods, roles, responsibilities and resources associated with the QA process. The ICP QA Specifications comprise a standardized methodology and evaluation criteria that ensures projects following the ICP protocols conform to the specifications of the protocols. The ICP team has been soliciting input from industry-experts on an ongoing basis to develop the QA Specification, due for release in late August. A final Technical Forum conference call was conducted on July 29th, to determine revisions that will be considered for the final draft of the document. The following is a list of revisions and topics that resulted from this discussion to be considered:
Differences between the protocols - Discuss the major differences between the protocols, such as: when should energy modeling be used as opposed to spreadsheet or other energy savings calculation methods; when are M&V Options A, B or C most appropriate; etc.
Flow chart - Consider a flow chart or similar method to describe when project development and QA tasks should be performed, and how these tie into critical financing decision milestones.
Whole building baseline - It was concluded that a whole building baseline should be developed for all projects, even those only involving a single or a handful of energy conservation measures (ECMs).
Baseline – Describe the process of developing a baseline upper and lower bound. This involves use of the energy use equation to calculate energy use for the baseline period, and comparison to historical utility bills. Using the error, standard deviation, and confidence/precision levels, create a minimum and maximum baseline – savings should fall within this range to be considered successful).
Data sufficiency - Application of the BEPA Standard to sub-metered loads.
Investment criteria - Reinforce the idea that all inputs necessary for the financial evaluation of a potential EE project should be developed by the project development team, such as implementation costs, estimated savings, available incentives, and effective useful life. Input from the investor should be provided regarding appropriate financial metrics and values to use for escalation rates, interest rates, discount rates, cost of capital, lease terms, and other appropriate financial inputs.
Simple payback is an accepted metric for some projects.
This protocol is intended for investors of all types. Many programs specifically define the financial metrics and inputs to use for analysis.
Better define the “project’s useful life” in the savings to investment ratio definition; this is defined as the weighted average of the expected useful life of all measures being considered.
Energy model calibration - Review and add additional details following ASHRAE Guideline 14 regarding calibration best practices.
Recommendation of monthly accuracy of the calibrated model to historical utility bills within +/-15% is appropriate.
Consider greater accuracy in annual calibration, such as +/-5%.
Risk and uncertainty in the model calibration should be documented and quantified.
Model should be calibrated to monthly data, and hourly data (if available).
Energy model resources - A comprehensive list of resources is already provided. Consider adding a resource for available energy modeling engines and front-ends, such as the DOE energy modeling tools library. Consider the paper by Dru Crawley, which compares ten specific energy modeling tools.
ECM modeling - Currently describes an iterative approach, and ECM bundled packages. Seek input on other potential approaches to ensure that interactive effects are being properly represented.
Savings calculations - Reference the uncertainty analysis methods described in the “Uncertainty and Risk” section of the specification in the Savings Calculation section. Uncertainty and risk best practices and quantification methods should be applied to the energy savings estimates.
Operational Performance Verification (OPV) - Term used to describe the commissioning process applied specifically to the ECMs (differentiates it from a whole-building commissioning effort).
Does the term “OPV” capture the intent, requiring complete commissioning of the process as it is applied to the project?
Would terms such as “targeted commissioning” or “project commissioning” better convey the idea being promoted?
Requires further discussion.
M&V - Will add short definitions of Options A, B and C, as well as when these are best applied based on the project or ECM type.
Uncertainty and risk - Will add references to this section in other sections of the specification as appropriate. Will also add mention of Latin Hypercube sampling in addition to Monte Carlo method.
Technical Forum Call - June 24th, 2014 Thanks to everyone who participated in today’s working group call! We had a great discussion regarding specifics of the QA Specification, which is summarized in the call notes below. A recording of the call will be posted soon on our Technical Forum section of the ICP website. Please, continue to review the QA Specification on GoogleDocs and provide us with your thoughts and comments. We are hoping to have a complete draft of the document prepared for the next call in late July, at which point we will discuss any final thoughts or comments, with the idea of releasing the document in mid to late August. - ICP Team
Energy data - ensure that we address filling data gaps (UT3 for example)
End use energy usage - suggest using ASHRAE Level II End Use Analysis workbook, to build up the end use energy usage from collected building information (lighting, nameplate data, etc)
Weather - mention using weather not just for baseline development, but also for calibration of the energy model, energy savings calculations, and M&V efforts
References - we don’t want to completely recreate guidelines and protocols that are already developed; but need to ensure we include enough critical information from these documents, particularly for documents that are not free, so the reader understands what is required. The resources can be referenced to provide greater detail, but the critical components should be included in the QA Specification
Retrofit isolation baseline - make sure it only applies to the system within the defined boundary. Applies to Option A and B approaches, and projects such as lighting retrofits, boiler retrofits, PV, etc. Refer to the CCC Guidelines for Verifying Savings from Commissioning Existing Buildings
Load profiles - refer to ECAM for guidance with this section
Baseline energy model development - should this be moved to the baseline section, or leave it in the savings calculation section? Is it considered part of the baseline development, or is it a “tool” used as part of the Savings Calculation?
ECM description - consider mentioning that a “persistence plan” should be included (this is already covered in the OM&M section however)
Financing / Investment Criteria - EUL is already included in this section (along with many other components)
Mention that the financial metrics should consider uncertainty if possible, as well as persistence of energy savings
Cost estimates - not a lot of great resources; RS Means, but what else? Mention that project should utilize third party contractor estimates whenever possible. Particularly if the building owner already has a relationship with specific contractors.
ECM modeling - discuss keywords used, assumptions, and documentation of sources. As part of QA, cross reference assumptions with construction documents
Operational Performance Verification - group still agrees it is appropriate to not use the term commissioning (Cx); but can still reference / use Cx resources to develop this section. Utilize construction checklists as well
Systems Manual - will reference ASHRAE’s new Process and Procedures for Development of a Systems Manual (the document should be approved next week)
Did not get to the OM&M and M&V sections of the QA Spec. Will discuss specific sections of these two in the Technical Forum, to get input from the Working Group.
Technical Forum Call - May 27, 2014
QA Specification overall structure
QA timing / sequence - EE project development tasks and subtasks / related QA activities
Five major sections: Baselining; Savings Calculations; Design, Construction and Verification; Operations, Maintenance and Monitoring; Measurement and Verification
Major sections broken down into Best Practices, QA Methods, and Resources
Owner’s representative and third party involvement
Uncertainty and risk
Documentation package
QA specification “sits behind” the QA checklists; there are six checklists available to support the QA process, for each of the Commercial and Multifamily protocols
The QA spec addresses all of the components of the protocols - each of the required elements/procedures/documents for Commercial and Multifamily
These are grouped by common elements (common to large, standard and targeted), or elements specific to each of the three types of protocols
Baseline - does our methodology apply to “existing” buildings; does the retrofit baseline development address what Ellen is talking about
RMI shared a tool under development, intended to be used to document energy modeling inputs and outputs. These type of report templates are intended to document what was done and the results, and to direct the modeler’s efforts, so they knows what important information to collect and report. RMI is developing instructions for its use, how to describe elements in the models, and develop programs that could automate collection of these data
RMI wants to road test this resource on actual projects
The QA Spec effort is looking for other resources like this one, that may be applicable to other aspects of the EE project development
The QA specification is shared on GoogleDocs. ICP is seeking comments and thoughts on the sections that have been developed or are under development. Use the “comment” feature in GoogleDocs to comment or share your thoughts. If logging in anonymously, please include your name in comments, so we can follow up with you directly when appropriate.
In addition to comments and thoughts on the QA specification itself, ICP is also looking for additional resources and tools we should be considering, or that are under development (such as ASHRAE Guideline 1.4 which is applicable to Systems Manual development)
Materials to consider
ECB Compliance workbook - for energy modeling, applies to new construction - documenting energy usage, but could be modified for use with the QA spec (Ron Herbst)
EEM Characterization Forms - for energy modeling, applies to new construction - documents weather data, utility rates, benchmarking, modeling general methodology, general model inputs, model results and errors, EEMs results and inputs as well as QC checks and “risks” commentary, QC checklist [[consider adding this as an appendix, referencing it in section 3.2.5 specific for LC]]
